Duration
2 days
Course Overview
Apache Kafka is a robust, open-source stream processing platform that enables real-time data feeding, processing, and analytics. It's designed to handle high volumes of data and support high-throughput applications. This course begins with an introduction to messaging systems, focusing on Apache Kafka's role in the modern data ecosystem and its workflow. We'll cover installation procedures, including configuring Kafka brokers and deploying Kafka in cloud environments. The crucial role of Zookeeper in managing cluster state and basic operations is also discussed.
Building on this foundation, we delve into Kafka producers and consumers—key components for writing and reading messages within Kafka. You'll learn how to construct producers, configure them for optimal performance, create consumers, and manage commits and offsets. Understanding Kafka's internals will provide insights into cluster membership, replication mechanisms, request processing, and physical storage considerations.
The course emphasizes reliable data delivery by exploring system reliability validation techniques. We also tackle building resilient data pipelines with Kafka Connect and stream-processing frameworks while addressing critical factors like timeliness, throughput, and security. Cross-cluster data mirroring techniques are explained to ensure robust multicluster architectures. Finally, administering Kafka topics, consumer groups, dynamic configurations, along with monitoring using Kafka APIs and metrics are covered to equip you with comprehensive administration skills.
Target Audience
- Students
- Entry Level
- Engineers
Methodologies
- Lecture
- Slides
- Case Studies
- Labs
- Group Discussion
What you'll learn
- Construct efficient Kafka producers and consumers for message handling
- Comprehend the role of Zookeeper in Apache Kafka's ecosystem
- Implement reliable data delivery systems using Apache Kafka
- Administer a Kafka environment effectively through topic management and performance monitoring
- Install and configure Apache Kafka brokers both locally and in cloud environments
- Understand the fundamental concepts of messaging systems and Apache Kafka
- Gain deep knowledge of Kafka internals such as cluster membership and replication
- Build scalable data pipelines utilizing Kafka Connect and stream-processing frameworks
Outlines
- Apache kafka – introduction
- What is messaging system
- Why Apache Kafka?
- Apache Kafka basic
- The Data Ecosystem
- Apache workflow
- Installing Kafka
- Apache kafka - Zookeeper role
- Kafka Producers : Writing Message to Kafka
- Kafka Consumers : Reading Data to Kafka
- Kafka Internals
- Reliable Data Delivery
- Building Data Pipeline
- Cross-Cluster Data Mirroring
- Administering Kafka
- Kafka API’s & Monitoring Kafka
- Stream processing
Programme scope, delivery arrangements and suitability are confirmed during enquiry.