Kafka Fundamentals
Opis
wydawany jest certyfikat
Cele
- Understand Kafka architecture
- Understand the deployment and configuration of Kafka
- Use REST-based access to Kafka
- Create Kafka Java API clients
- Design multi-cluster architectures
- Use Kafka Connect tools
- Create Kafka Streams programs
Grupa docelowa
- Software Developers
- Software Architects
- Data Engineers
Warunki wstępne
- Development experience in Java (over 6 months)
Plan działania
1. Module 1: Kafka Architecture
Planning your own distributed queue in pairs: write, read, keep data in parallel mode.
1. What's the format and average size of messages?
2. Can messages be repeatedly consumed?
3. Are messages consumed in the same order they were produced?
4. Does data need to be persisted?
5. What is data retention?
6. How many producers and consumers are we going to support?
2. Module 2: Kafka-topics, console-consumer, console-producer
1. Using internal Kafka-topics, console-consumer, console-producer
2. Create topic with 3 partitions & RF = 2
3. Send message, check the ISR
4. Organize message writing/reading with order message keeping
5. Organize message writing/reading without order message keeping and hash partitioning
6. Organize message writing/reading without skew data
7. Read messages from the start, end and offset
8. Read topic with 2 partitions / 2 consumers in one consumer group (and different consumer group)
9. Choose optimal number of consumers for reading topic with 4 partitions
10. Write messages with min latency
11. Write messages with max compression
3. Module 3: Web UI + Java, Scala, Python API + other languages (via Rest)
1. build simple consumer and producer
2. add one more consumer to consumer group
3. write consumer which reads 3 records from 1st partition
4. add writing to another topic
5. add transaction
Module 4: AVRO + Schema Registry
1. Add avro schema
2. compile java class
3. build avro consumer and producer with a specific record
4. add schema registry
5. add error topic with error topic and schema registry
6. build avro consumer and producer with a generic record
Module 5: SpringBoot + SpringCloud
Homework:
1. Write template for Spring App
2. Add Kafka Template with producer
3. Add Kafka Template with consumer
4. Add rest controller
5. Modify spring boot to work in async (parallel) mode
Module 6: Streaming Pipelines (Kafka Streams + KSQL + Kafka Connect vs Akka Streams vs Spark Streaming vs Flink)
Homework:
1. Choose the way to read data from a Kafka topic with 50 partitions
2. Try to use the checkpoint mechanism
3. Start the five executors and kill some of them
4. Check the backpressure
Module 7: Kafka Monitoring
Homework:
1. Build several metrics in Grafana