EIEIOTECHNOVATION
← Training programmes

EIEIO / DATA SCIENCE

APACHE CONFLUENT KAFKA

Course outline

5 days
Discuss this programme →

Duration

APACHE/CONFLUENT KAFKA

5 days

Prerequisites

Programming knowledge

SQL knowledge

This training is in a local windows environment for understanding basics and APIs and later on running Producers and Consumers on Kubernetes Kafka cluster environment.

Lab

16GB RAM with windows OS, MS Azure accounts (last one or two days based on client) for participants for Kubernetes cluster

Day 1

  • 1. Introduction to Big Data and Apache Kafka
  • Introduction to Big Data
  • Big Data Analytics
  • Need for Kafka
  • What is Kafka?
  • Kafka Features
  • Kafka Concepts
  • Kafka Architecture
  • Kafka Components
  • ZooKeeper
  • Where is Kafka Used?
  • Kafka Installation
  • Kafka Cluster
  • Types of Kafka Clusters
  • Configuring Single Node Single Broker Cluster

Hands on:

  • Kafka Installation
  • Implementing Single Node-Single Broker Cluster

Kafka Producer

  • Configuring Single Node Multi Broker Cluster
  • Constructing a Kafka Producer
  • Sending a Message to Kafka
  • Producing Keyed and Non-Keyed Messages
  • Sending a Message Synchronously & Asynchronously
  • Configuring Producers
  • Serializers
  • Serializing Using Apache Avro  Partitions Hands On:
  • Working with Single Node Multi Broker Cluster
  • Creating a Kafka Producer
  • Configuring a Kafka Producer
  • Sending a Message Synchronously & Asynchronously

Day 2

  • 2. Kafka Consumer
  • Consumers and Consumer Groups
  • Standalone Consumer
  • Consumer Groups and Partition Rebalance
  • Creating a Kafka Consumer
  • Subscribing to Topics
  • The Poll Loop
  • Configuring Consumers
  • Commits and Offsets
  • Rebalance Listeners
  • Consuming Records with Specific Offsets
  • Deserializers

Hands On:

  • Creating a Kafka Consumer
  • Configuring a Kafka Consumer
  • Working with Offsets
  • 3. Kafka Internals
  • Cluster Membership
  • The Controller
  • Replication
  • Request Processing
  • Physical Storage
  • Reliability
  • Broker Configuration
  • Using Producers in a Reliable System
  • Using Consumers in a Reliable System
  • Validating System Reliability
  • Performance Tuning in Kafka

Hands On:

  • Create topic with partition & replication factor 3 and execute it on multi-broker cluster
  • Show fault tolerance by shutting down 1 Broker and serving its partition from another broker

Day 3 and Day 4 part

  • 4. Kafka Cluster Architectures & Administering Kafka
  • Use Cases - Cross-Cluster Mirroring
  • Multi-Cluster Architectures
  • K8S with Confluent Kafka - with and without zookeeper
  • Building AWS MSK cluster and do kafka operations on MSK
  • Kafka Security on K8s
  • Security in Kafka – TLS & ACLs
  • Apache Kafka’s MirrorMaker/Confluent Replicator
  • Other Cross-Cluster Mirroring Solutions
  • Topic Operations
  • Consumer Groups
  • Dynamic Configuration Changes
  • Partition Management
  • Consuming and Producing
  • Unsafe Operations

Hands on:

  • Topic Operations
  • Consumer Group Operations
  • Partition Operations
  • Consumer and Producer Operations

Kafka Monitoring and Kafka Connect

  • Considerations When Building Data Pipelines
  • Metric Basics
  • Kafka Broker Metrics
  • Client Monitoring
  • Lag Monitoring
  • End-to-End Monitoring
  • Kafka Connect
  • When to Use Kafka Connect?
  • Kafka Connect Properties
  • Kafka Connect with Standalone and Distributed mode
  • File connector
  • HDFS connector
  • AWS S3
  • MYSQL
  • ActiveMQ
  • RabbitMQ

Hands on:

  • Kafka Connect

Day 4 part & day 5

  • 5. Kafka Stream Processing
  • Stream Processing
  • Stream-Processing Concepts
  • Stream-Processing Design Patterns
  • Kafka Streams by Example
  • Kafka Streams: Architecture Overview
  • KStreams and KSQL

Hands on:

  • Kafka Streams
  • KSQL CLI
  • Word Count Stream Processing
  • Various use cases using KSQL
  • 6. Integration of Kafka With Hadoop and Spark
  • Apache Hadoop Basics
  • Hadoop Configuration
  • Kafka Integration with Hadoop
  • Apache Spark Basics
  • Spark Configuration
  • Kafka Integration with Spark

Hands On:

  • Kafka integration with Hadoop
  • Kafka integration with Spark
  • 7. Integration of Kafka With Cassandra
  • Flume Basics
  • Integration of Kafka with Flume
  • Cassandra Basics such as and KeySpace and Table Creation
  • Integration of Kafka with Cassandra

Hands On:

  • Kafka demo with Flume
  • Kafka demo with Cassandra

PS: Training on Java API and Producer and Consumer with Scala API

  • 8. Course Deliverables
  • Workshop style coaching
  • Interactive approach
  • Course material
  • Hands on practice exercises for each topic
  • Quiz at the end of each major topic
  • Tips and techniques on Confluent Certified Developer for Apache Kafka (CCDAK)

Programme scope, delivery arrangements and suitability are confirmed during enquiry.