Duration

4 days

Course Outlines

1. Introduction to Big Data and Cloud Computing
Overview of Big Data concepts and technologies
Basics of cloud computing
Understanding distributed computing
2. Introduction to Spark
Understanding Spark architecture and components
Spark Core concepts (RDDs, DataFrames, Datasets)
Hands-on Exercise: Basic operations with Spark (creating RDDs and DataFrames)
3. Introduction to PySpark
Setting up PySpark environment
Basic operations with PySpark
4. PySpark SQL
Introduction to PySpark SQL
Working with DataFrames and Datasets
Performing basic SQL queries on structured data
Hands-on Exercise: Creating and querying DataFrames with PySpark SQL
5. Delta Lake
Introduction to Delta Lake
Delta Lake architecture and features
Time travel and ACID transactions
Schema evolution in Delta Lake
Hands-on Exercise: Creating and managing Delta tables, implementing schema evolution
6. Medallion Architecture
Understanding Medallion Architecture (Bronze, Silver, Gold)
Implementing Medallion Architecture with Delta Lake
Hands-on Exercise: Implementing a simple Medallion Architecture with Delta Lake using PySpark
7. Idempotency in Data Processing
Understanding idempotency
Implementing idempotent data pipelines
Hands-on Exercise: Building idempotent data processing tasks
8. Streaming Data with PySpark
Introduction to PySpark Streaming (overview)
Processing real-time data streams
Hands-on Exercise: Implementing a basic streaming data processing task (demonstration)

Programme scope, delivery arrangements and suitability are confirmed during enquiry.