Setup

Prerequisite CLI tools

The following CLI tools are required for running the exercises in this tutorial. Please have them installed and configured before you get started with any of the tutorial chapters.

Tool macOS Fedora windows

Tool	macOS	Fedora	windows
`Git`	Download	Download	Download
`Docker`	Docker for Mac	`dnf install docker`	Docker for Windows
`VirtualBox`	Download	Download	Download
Minikube v1.25.1	Download	Download	Download
`kubectl v1.23.1`	Download	Download	Download
stern	`brew install stern`	Download	Download
`Apache Maven 3.6.3`	Download	Download	Download
`kcat`	Download or `brew install kcat`	Download	Download IMPORTANT: An alternative for running `kcat` is to use the following Docker container image https://hub.docker.com/r/edenhill/kcat (documentation see https://github.com/edenhill/kcat#try-it-out-with-docker)
`Java 11`	Download	Download	Download

Git

Docker

dnf install docker

VirtualBox

Minikube v1.25.1

kubectl v1.23.1

brew install stern

Apache Maven 3.6.3

kcat

Download or brew install kcat

Download

IMPORTANT: An alternative for running kcat is to use the following Docker container image https://hub.docker.com/r/edenhill/kcat (documentation see https://github.com/edenhill/kcat#try-it-out-with-docker)

Java 11

Download

The following CLI tools are optional for running the exercises in this tutorial. Although they are used in the tutorial, you could use others without any problem.

Tool macOS Fedora windows

Tool	macOS	Fedora	windows
yq v2.4.1	Download	Download	Download
jq v1.6.0	Download	Download	Download
httpie	`brew install httpie`	`dnf install httpie`	https://httpie.org/doc#windows-etc
watch	`brew install watch`	`dnf install procps-ng`
kubectx and kubens	`brew install kubectx`	https://github.com/ahmetb/kubectx
hey	`brew install hey`	Download	Download

brew install httpie

dnf install httpie

https://httpie.org/doc#windows-etc

watch

brew install watch

dnf install procps-ng

kubectx and kubens

brew install kubectx

https://github.com/ahmetb/kubectx

hey

brew install hey

Download

Download & Configure Tutorial Sources

Before we start setting up the environment, let’s clone the tutorial sources and set the TUTORIAL_HOME environment variable to point to the root directory of the tutorial:

git clone https://github.com/redhat-developer-demos/kafka-tutorial.git kafka-tutorial

export TUTORIAL_HOME="$(pwd)/kafka-tutorial"

cd $TUTORIAL_HOME

Setup Kafka

Assuming you’ve got Docker & Docker Compose installed.

cd $TUTORIAL_HOME/apps/it
docker compose up --remove-orphans

...

kafka      | [2022-12-09 08:29:01,867] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
kafka      | [2022-12-09 08:29:01,878] INFO [MetadataCache brokerId=0] Updated cache from existing <empty> to latest FinalizedFeaturesAndEpoch(features=Map(), epoch=0). (kafka.server.metadata.ZkMetadataCache)
kafka      | [2022-12-09 08:29:01,893] INFO [ExpirationReaper-0-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
kafka      | [2022-12-09 08:29:01,929] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
kafka      | [2022-12-09 08:29:01,946] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Enabling request processing. (kafka.network.SocketServer)
kafka      | [2022-12-09 08:29:01,974] INFO Kafka version: 3.3.1 (org.apache.kafka.common.utils.AppInfoParser)
kafka      | [2022-12-09 08:29:01,974] INFO Kafka commitId: e23c59d00e687ff5 (org.apache.kafka.common.utils.AppInfoParser)
kafka      | [2022-12-09 08:29:01,974] INFO Kafka startTimeMs: 1670574541958 (org.apache.kafka.common.utils.AppInfoParser)
kafka      | [2022-12-09 08:29:01,975] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
kafka      | [2022-12-09 08:29:02,093] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Recorded new controller, from now on will use broker kafka:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)
kafka      | [2022-12-09 08:29:02,113] INFO [BrokerToControllerChannelManager broker=0 name=alterPartition]: Recorded new controller, from now on will use broker kafka:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)

Open a new terminal to run Kafka commands:

Verify that Zookeeper and Apache Kafka containers are up and running:

docker ps

CONTAINER ID   IMAGE                                      COMMAND                  CREATED         STATUS         PORTS                                              NAMES
0d4bc6ec2ddf   quay.io/strimzi/kafka:0.32.0-kafka-3.3.1   "sh -c 'bin/kafka-se…"   2 minutes ago   Up 2 minutes   0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp   kafka
d8fdbab971c6   quay.io/strimzi/kafka:0.32.0-kafka-3.3.1   "sh -c 'bin/zookeepe…"   2 minutes ago   Up 2 minutes   0.0.0.0:2181->2181/tcp                             zookeeper

Notice that the Kafka container is exposing two ports to outside. The reason is that Kafka requires one port for being accessed from your host machine, and another one for being accessed from the Docker host machine. You can find a full explanation of this at Kafka Listeners.

Verify that the Kafka cluster is correctly started and configured by running the following command:

kcat
kcat in Docker

kcat -b localhost:29092 -L

Metadata for all topics (from broker 0: localhost:29092/0):
 1 brokers:
  broker 0 at localhost:29092 (controller)
 0 topics:

docker run --rm -it --network=kafka-tutorial edenhill/kcat:1.7.1 -b kafka:9092 -L

Metadata for all topics (from broker 0: kafka:9092/0):
 1 brokers:
  broker 0 at kafka:9092 (controller)
 0 topics: