EIEIOTECHNOVATION
← Training programmes

EIEIO / DATA SCIENCE

Apache Kafka

Course outline

2 days
Discuss this programme →

Duration

Apache Kafka

2 days

Overview

This comprehensive course on Apache Kafka provides a deep dive into the world of distributed messaging systems. Beginning with an introduction to messaging systems and Apache Kafka, the course covers a wide range of topics including installation, the role of Zookeeper, constructing and configuring producers and consumers, and understanding Kafka internals. The course also delves into reliable data delivery, building data pipelines, cross-cluster data mirroring, administering Kafka, monitoring Kafka through APIs, and stream processing. Each section is designed to provide a thorough understanding of different aspects of Apache Kafka.

What you'll learn

  • Get hands-on experience with administering Kafka including topic operations and dynamic configurations changes
  • Grasp the role of Zookeeper in managing cluster state in Kafka
  • Understand how to construct and configure Kafka producers and consumers
  • Master cross-cluster data mirroring techniques using Multi-node kafka
  • Understand the basics of messaging systems and their application in Apache Kafka
  • Learn how to install and configure Kafka brokers for optimal performance
  • Learn how to build reliable data pipelines using Kafka Connect versus Producer and Consumer
  • Gain insights into the internal workings of Apache Kafka including cluster membership and replication

Day 1

Kafka Introduction

  • What is messaging system
  • Why Apache Kafka?
  • Apache Kafka basic
  • The Data Ecosystem
  • Apache workflow

Installing Kafka

  • Installing kafka broker
  • Broker configuration
  • Kafka in the cloud

Apache kafka - Zookeeper role

  • Role of Zookeeper
  • Zookeeper Basic Operation

Recap and Q&A

Kafka Producers : Writing Message to Kafka

  • Constructing kafka producer
  • Sending a message to kafka
  • Configuring Producer

Kafka Consumers : Reading Data to Kafka

  • Kafka consumer concept
  • Creating kafka consumer
  • Commits and offsets
  • Standalone consumer

Kafka Internals

  • Cluster membership
  • The controller
  • Replication
  • Request processing
  • Physical storage

Reliable Data Delivery

  • Realibility guarantees
  • Using produsers in a reliable system
  • Using consumers in a reliable system
  • Validating system reliability

Recap and Q&A

Day 2

Building Data Pipeline

  • Considerations When Building Data Pipelines ○ Timeliness ○ Reliability ○ High and Varying Throughput ○ Data Formats ○ Transformations ○ Security ○ Failure Handling ○ Coupling and Agility
  • When to Use Kafka Connect Versus Producer and Consumer

Cross-Cluster Data Mirroring

  • Multicluster Architectures
  • Apache kafka mirror maker
  • Multi-node kafka
  • Multi-node cluster setup

Recap and Q&A

Administering Kafka

  • Topic operations
  • Consumer Group
  • Dynamic configurations changes
  • Consuming and producing

Kafka API’s & Monitoring Kafka

  • Kafka API’s
  • Metrics Basic
  • Kafka Broker metrics
  • Client monitoring
  • End-to-End monitoring

Stream processing

  • What is Stream processing?
  • Stream processing concepts
  • Stream processing design pattern
  • Kafka Stream
  • How to choose a Stream-Processing Framework

Recap and Q&A

Programme scope, delivery arrangements and suitability are confirmed during enquiry.