Kafka Fundamentals

1. Module 1: Kafka Architecture

Planning your own distributed queue in pairs: write, read, keep data in parallel mode.

1. What's the format and average size of messages?

2. Can messages be repeatedly consumed?

3. Are messages consumed in the same order they were produced?

4. Does data need to be persisted?

5. What is data retention?

6. How many producers and consumers are we going to support?

2. Module 2: Kafka-topics, console-consumer, console-producer

1. Using internal Kafka-topics, console-consumer, console-producer

2. Create topic with 3 partitions & RF = 2

3. Send message, check the ISR

4. Organize message writing/reading with order message keeping

5. Organize message writing/reading without order message keeping and hash partitioning

6. Organize message writing/reading without skew data

7. Read messages from the start, end and offset

8. Read topic with 2 partitions / 2 consumers in one consumer group (and different consumer group)

9. Choose optimal number of consumers for reading topic with 4 partitions

10. Write messages with min latency

11. Write messages with max compression

3. Module 3: Web UI + Java, Scala, Python API + other languages (via Rest)

1. build simple consumer and producer

2. add one more consumer to consumer group

3. write consumer which reads 3 records from 1st partition

4. add writing to another topic

5. add transaction

Module 4: AVRO + Schema Registry

1. Add avro schema

2. compile java class

3. build avro consumer and producer with a specific record

4. add schema registry

5. add error topic with error topic and schema registry

6. build avro consumer and producer with a generic record

Module 5: SpringBoot + SpringCloud

Homework:

1. Write template for Spring App

2. Add Kafka Template with producer

3. Add Kafka Template with consumer

4. Add rest controller

5. Modify spring boot to work in async (parallel) mode

Module 6: Streaming Pipelines (Kafka Streams + KSQL + Kafka Connect vs Akka Streams vs Spark Streaming vs Flink)

Homework:

1. Choose the way to read data from a Kafka topic with 50 partitions

2. Try to use the checkpoint mechanism

3. Start the five executors and kill some of them

4. Check the backpressure

Module 7: Kafka Monitoring

Homework:

1. Build several metrics in Grafana