Setup

Prerequisite CLI tools

The following CLI tools are required for running the exercises in this tutorial. Please have them installed and configured before you get started with any of the tutorial chapters.

Tool macOS Fedora windows

Git

Download

Download

Download

Docker

Docker for Mac

dnf install docker

Docker for Windows

VirtualBox

Download

Download

Download

Minikube v1.25.1

Download

Download

Download

kubectl v1.23.1

Download

Download

Download

stern

brew install stern

Download

Download

Apache Maven 3.6.3

Download

Download

Download

Kafkacat

Download or brew install kafkacat

Download

Download

Kstreamscat

Download

Download

Download

Java 11

Download

Download

Download

IMPORTANT: An alternative for running kcat is to use the following Docker command and image docker run -it --network=host edenhill/kcat:1.7.0 kcat

The following CLI tools are optional for running the exercises in this tutorial. Although they are used in the tutorial, you could use others without any problem.

Tool macOS Fedora windows

yq v2.4.1

Download

Download

Download

jq v1.6.0

Download

Download

Download

httpie

brew install httpie

dnf install httpie

https://httpie.org/doc#windows-etc

watch

brew install watch

dnf install procps-ng

kubectx and kubens

brew install kubectx

https://github.com/ahmetb/kubectx

hey

brew install hey

Download

Download

Download & Configure Tutorial Sources

Before we start setting up the environment, let’s clone the tutorial sources and set the TUTORIAL_HOME environment variable to point to the root directory of the tutorial:

export TUTORIAL_HOME="$(pwd)/kafka-tutorial"
cd $TUTORIAL_HOME

Setup Kafka

Assuming you’ve got Docker & Docker Compose installed.

cd $TUTORIAL_HOME/apps/it
docker-compose up --remove-orphans
...

kafka_1      | [2020-04-27 05:51:56,560] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
kafka_1      | [2020-04-27 05:51:56,561] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
kafka_1      | [2020-04-27 05:51:56,597] INFO [ExpirationReaper-0-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
kafka_1      | [2020-04-27 05:51:56,641] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka_1      | [2020-04-27 05:51:56,656] INFO [SocketServer brokerId=0] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
kafka_1      | [2020-04-27 05:51:56,666] INFO Kafka version: 2.4.0 (org.apache.kafka.common.utils.AppInfoParser)
kafka_1      | [2020-04-27 05:51:56,666] INFO Kafka commitId: 77a89fcf8d7fa018 (org.apache.kafka.common.utils.AppInfoParser)
kafka_1      | [2020-04-27 05:51:56,667] INFO Kafka startTimeMs: 1587966716657 (org.apache.kafka.common.utils.AppInfoParser)
kafka_1      | [2020-04-27 05:51:56,669] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Open a new terminal to run Kafka commands:

Verify that Zookeeper and Apache Kafka containers are up and running:

docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                                                                    NAMES
5152b0eabe99        strimzi/kafka:0.17.0-kafka-2.4.0   "sh -c 'bin/kafka-se…"   43 seconds ago      Up 42 seconds       0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp                         kafka
e3417f63ac61        strimzi/kafka:0.17.0-kafka-2.4.0   "sh -c 'bin/zookeepe…"   43 seconds ago      Up 43 seconds       0.0.0.0:2181->2181/tcp, 0.0.0.0:2888->2888/tcp, 0.0.0.0:3888->3888/tcp   it_zookeeper_1
Notice that Kafka container is exposing two ports to outside. The reason is that Kafka requires one port for accessing from host machine, and another one for accessing from Docker host machine. You can find a full explanation of this at Kafka Listeners.

Verify that the Kafka cluster is correctly started and configured by running the following command:

  • kcat

  • kcat in Docker

kcat -b localhost:9092 -L
docker run -it --network=host edenhill/kcat:1.7.0 kcat -b localhost:29092 -L
Metadata for all topics (from broker 0: localhost:9092/0):
 1 brokers:
  broker 0 at localhost:9092 (controller)
 0 topics: