Kafka Listeners

Probably, you’ve noticed that you are exposing two ports the 9092 and 29092. The reason for this is that we want to be able to access Kafka broker not only from outside the Docker Host (for example when kcat tool is used), but also from inside the Docker Host (for example when we were deploying Java services inside Docker).

When a client starts, it requests some metadata from the Kafka broker. One of these data is the endpoint of the current Kafka leader broker for a partition. This endpoint is then used to create a working connection to the cluster.

Then what’s happening when running the Kafka inside Docker? The Kafka broker returns the container hostname to the client (that is why in our tutorial the Kafka hostname is kafka).

Then what’s happening with the client? Well, it depends on the scenario:

  1. If the client is running inside the Docker Host, then it will be able to connect to the returned hostname by the broker (ie kafka).

  2. If the client is running outside of the Docker Host, then an error is thrown. The reason is that the client will get the hostname to connect from Kafka broker, which is the container hostname (ie kafka), which of course is not valid when client will try to reach the container host from outside the Docker Host.

hdhe

How to fix this problem?

To fix this problem, you need to expose two ports, one returning the hostname (ie kafka) and another one returning the localhost as the hostname.

hdhs

That’s why two ports are exposed in the current course.

You can run the next commands to visualize how broker host differs depending on the port:

  • kcat

  • kcat in Docker

kcat -b localhost:29092 -L
docker run -it --network=host edenhill/kcat:1.7.0 kcat -b localhost:29092 -L
Metadata for all topics (from broker 0: localhost:29092/0):
 1 brokers:
  broker 0 at localhost:29092 (controller)
 0 topics:
  • kcat

  • kcat in Docker

kcat -b localhost:9092 -L
docker run -it --network=host edenhill/kcat:1.7.0 kcat -b localhost:29092 -L
Metadata for all topics (from broker -1: localhost:9092/bootstrap):
 1 brokers:
  broker 0 at kafka:9092 (controller) (1)
 0 topics:
1 Notice that when the leader is queried using port 9092, then the hostname returned is kafka:9092.