Duration
5 days
Course Structure
This custom course covers variety of topics from Kafka Development to Kafka Administration to Kafka with Big Data stack.
Intended Audience
The intended audience of the course is:
-Hadoop Developers
-Data Engineers
-Integration Engineers
-Architects
-Data Steward
Prerequisites and Lab Details
The intended audience should have a background on:
- Software Development
- Unix and Hadoop
The lab setup for this course would be:
- Windows 7 or Higher
- 16 GB RAM
- VMWare Player
Course Outline
Phase 1 and Phase 2: Kafka- A Developer’s Perspective (Day1 & 2)
Introduction
- The motivation for Apache Kafka
- Kafka Fundamentals
- Kafka’s Architecture
- Developing with Kafka
- More Advanced Kafka Development
- Schema Management in Kafka
- Kafka Connect for Data Movement
- Kafka Streams
Labs
- Using Kafka’s Command Line Tools
- Consuming from Multiple Partitions
- Writing a Basic Producer
- Writing a Basic Consumer
- Accessing Previous Data
- Using Kafka with Avro
- Running Kafka Connect in Standalone mode
- Using Kafka Connect with REST API
- Using JDBC Connector
- Writing Kafka Streams Application
Phase 3: Kafka from Big Data and Hadoop Perspective (Day 3)
Course Objectives
- What is Flume
- Flume Sources
- Flume Channel
- Flume Sinks
- Flume Agent
- Flume + Kafka Integration
- Introduction to Spark
- Spark Streaming
Labs
- Ingesting Data with Flume
- Using Kafka as a source or a sink for Flume
- Processing Data using Spark Streaming with Kafka as a Streaming Source
- Using FLAFKA
Phase 4 and Phase 5: Kafka- An Administrator’s Perspective (Day 4 & 5)
Course Objectives
- Providing Durability
- Optimizing Kafka Performance
- Kafka Security
- Managing a Kafka Cluster
- Design for High Availability
- Integrating Systems with Kafka Connect
- Important JMX Metrics
Labs
- Basic Operations and Metrics
- Investigating the Distributed Log
- Viewing Offsets and Modifying Partitions
- Performance Tuning
- Kafka Administrative Tools
Programme scope, delivery arrangements and suitability are confirmed during enquiry.