Deploying Apache Kafka Cluster

As part of the upcoming section of this chapter, we will be deploying a Knative Source, that will respond to Apache Kafka Topic messages(events). Before getting to other exercises, we need to first deploy Apache Kafka inside in your Kubernetes cluster.

The strimzi Kubernetes operator can be used to deploy the Apache Kafka Cluster in your Kubernetes cluster.

cd $TUTORIAL_HOME/eventing

Run the following command to create the kafka namespace

kubectl create namespace kafka

Deploy Apache Kafka into kafka namespace:

curl -L \
https://github.com/strimzi/strimzi-kafka-operator\
/releases/download/0.32.0/strimzi-cluster-operator-0.32.0.yaml \
  | sed 's/namespace:.*/namespace: kafka/' \
  | kubectl apply -n kafka -f -

Wait for the strimzi-cluster-operator to be running:

watch "kubectl get pods -n kafka"

The command should show the following output:

NAME                                        READY STATUS    AGE
strimzi-cluster-operator-85f596bfc7-7dgds   1/1   Running   1m2s

The strimzi operator would have installed several Apache Kafka related CRDs which can be used to create Apache Kafka core resources such as a topic, users, connectors etc., you can verify the CRDs that are available by querying api-resources:

kubectl api-resources --api-group='kafka.strimzi.io'

The command should show the following output:

NAME                 SHORTNAMES   APIVERSION                 NAMESPACED   KIND
kafkabridges         kb           kafka.strimzi.io/v1beta2   true         KafkaBridge
kafkaconnectors      kctr         kafka.strimzi.io/v1beta2   true         KafkaConnector
kafkaconnects        kc           kafka.strimzi.io/v1beta2   true         KafkaConnect
kafkamirrormaker2s   kmm2         kafka.strimzi.io/v1beta2   true         KafkaMirrorMaker2
kafkamirrormakers    kmm          kafka.strimzi.io/v1beta2   true         KafkaMirrorMaker
kafkarebalances      kr           kafka.strimzi.io/v1beta2   true         KafkaRebalance
kafkas               k            kafka.strimzi.io/v1beta2   true         Kafka
kafkatopics          kt           kafka.strimzi.io/v1beta2   true         KafkaTopic
kafkausers           ku           kafka.strimzi.io/v1beta2   true         KafkaUser

Now with the Apache Kafka operator running, you can deploy and verify a single node Apache Kafka cluster by running the command:

kubectl -n kafka apply -f kafka-broker-my-cluster.yaml

Watch the Kafka cluster deployment:

watch "kubectl get pods -n kafka"

Watch the kafka namespace for the cluster deployment:

my-cluster-entity-operator-6cb88758b7-hhxq7   3/3     Running   0              31s
my-cluster-kafka-0                            1/1     Running   0              54s
my-cluster-zookeeper-0                        1/1     Running   2 (104s ago)   118s
strimzi-cluster-operator-787d48b5bc-649jb     1/1     Running   2 (2m7s ago)   41m

The Kubernetes CRD resource $TUTORIAL_HOME/eventing/kafka-broker-my-cluster.yaml, will deploy a single Zookeeper, Kafka Broker and a Entity-Operator. The Entity-Operator is responsible for managing different custom resources such as KafkaTopic and KafkaUser.

Now that you have an Apache Kafka cluster deployed, you can create a Kafka Topic using the KafkaTopic CRD, the following listing shows how to create a Kafka Topic my-topic:

Create Kafka Topic my-topic

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 10 (1)
  replicas: 1

1	Partitions 10 allows for more concurrent scale-out of sink pods. In theory, up to 10 pods will scale-up if there are enough messages flowing through the Kafka topic.

You can choose to skip the manual pre-creation of a KafkaTopic but the automatically generated topics will have partitions set to 1 by default.

Create Kafka Topic

kubectl -n kafka create -f kafka-topic-my-topic.yaml

Verify the created topic:

kubectl -n kafka  get kafkatopics

The verify command should show the following output:

NAME       PARTITIONS   REPLICATION FACTOR
my-topic   10           1

Verify that your Kafka Topic is working correctly by connecting a simple producer, consumer and creating some test messages. The sample code repository includes a script for producing Kafka messages called kafka-producer.sh. Execute the script and type in "one", "two", "three". Hitting enter/return after each string:

Producer

$TUTORIAL_HOME/bin/kafka-producer.sh

On the terminal prompt try entering texts like:

>one
>two
>three

Consumer

You should also leverage the sample code repository’s kafka-consumer.sh script to see the message flow through the topic, open a new terminal and run:

$TUTORIAL_HOME/bin/kafka-consumer.sh

On the consumer terminal prompt you will receive texts like:

>one
>two
>three

You can use Ctrl-c to stop producer & consumer interaction and their associated pods.