EIEIOTECHNOVATION
← Training programmes

EIEIO / DATA SCIENCE

Apache Spark Development

Course outline

ApacheDev4 days
Discuss this programme →

Duration

Apache Spark Development

4 days

Overview

Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, Spark ML for machine learning, and Spark Streaming for live data stream processing. With Spark running on Apache Hadoop YARN, developers can create applications to derive actionable insights within a single, shared dataset in Hadoop.

This training course will teach you how to solve Big Data problems using the Apache Spark framework. The training will cover a wide range of Big Data use cases such as ETL, DWH and streaming. It will also demonstrate how Spark integrates with other well established Hadoop ecosystem products. You will learn the course curriculum through theory lectures, live demonstrations and lab exercises. This course will be taught in the Java programming language and we will be using Cloudera 7.1.4 using Spark 3.x.

Prerequisites

Following are the prerequisites for the course.

  • Programming knowledge in Java is required
  • Basic Knowledge of big data use-cases.
  • Basic knowledge of databases, OLAP/OLTP use cases, SQL
  • Knowledge of Java stack – JVM is helpful

Who Should Attend?

  • Application developers, Data Engineers, DevOps engineers, Architects, QAs, Technical Managers

Day 1

9:00am-10:30am

Introduction to Java

  • Brief overview of Java programming language
  • Benefits and features of Java

10:30am-10:45pm Morning Break

10:45pm-12:30pm

Basic Syntax in Java

  • Variables
  • Declaration and initialization
  • Primitive data types (int, double, boolean, etc.)
  • Reference data types (String, arrays, objects)
  • Control Structures
  • Conditionals (if-else, switch)
  • Loops (for, while, do-while)
  • Functions/Methods
  • Declaring and defining methods
  • Passing parameters and returning values

12:30pm-1:30pm Lunch

1:30pm-3:30pm

Building a Jar (Java Archive) File

  • Introduction to JAR files
  • Creating a JAR file using command-line tools
  • Including class files and resources in the JAR
  • Executing a JAR file

3:30pm-3:45pm Afternoon Break

3:45pm-5:00pm

Functional Programming in Java

  • Overview of functional programming concepts
  • Lambda Expressions
  • Syntax and examples
  • Functional interfaces and lambda expressions
  • Functional Interfaces
  • Predefined functional interfaces (Predicate, Function, Consumer, etc.)
  • Creating custom functional interfaces
  • Streams
  • Introduction to streams
  • Stream operations (map, filter, reduce, etc.)
  • Chaining operations with streams

Day 2

9:00am-10:30am

Introduction to Containers/Kubernetes

  • Working with Containers
  • Kubernetes as Cluster Manager
  • Understanding Pods/Deployments
  • Working with Spark Image
  • Starting Spark workloads on Kubernetes/OpenShift.

10:30am-10:45pm Morning Break

10:45pm-12:30pm

Introduction to Spark

  • What is Apache Spark: The evolution from Hadoop
  • Why Spark?
  • Advantages of Spark over Hadoop MapReduce
  • Logical Architecture of Spark
  • Programming Languages in Spark
  • Functional Programming with Spark

12:30pm-1:30pm Lunch

1:30pm-3:30pm

Working with Spark & S3

  • Data sources for Spark application
  • Introduction to S3
  • Spark with S3 as data source

3:30pm-3:45pm Afternoon Break

3:45pm-5:00pm

Resilient Distributed Dataset (RDD)

  • RDD Basics
  • Spark Context and Spark Session
  • RDD Operations: Read from file, transforming, and saving persistent
  • Leveraging in-memory processing
  • Pair RDD operations
  • Working with semi-structured data formats using regex, JSON, XML libraries
  • MapReduce Operations
  • Joins
  • Hands-on Exercises

Day 3

9:00am-10:30am

Spark Internals

  • Understanding Execution plans
  • Spark literature: Narrow, wide operations, shuffle, DAG, Shuffle, Stages, and Tasks
  • Job metrics
  • Fault Tolerance
  • Hands-on Exercises

10:30am-10:45pm Morning Break

10:45pm-12:30pm

Developing Spark Standalone Applications

  • Spark Applications vs. Spark Shell
  • Developing a Spark Application
  • Modifying Spark Key Configurations
  • Submitting Spark Application.
  • Hands-on Exercises

12:30pm-1:30pm Lunch

1:30pm-3:30pm

Advanced RDD

  • Caching
  • Distributed Persistence
  • Storage Levels
  • Broadcast using RDD
  • Accumulator using RDD
  • Hands-on Exercises

3:30pm-3:45pm Afternoon Break

3:45pm-5:00pm

Structured Data Processing in Spark

  • Spark SQL Overview
  • Spark SQL Documentation
  • Sub-modules of Spark SQL
  • Difference between RDD and DataFrame
  • Loading and processing data into DataFrame
  • DataFrame API
  • DataFrame Operations
  • Saving DataFrame to file systems
  • DataFrame internals that make it fast: Catalyst Optimizer and Tungsten
  • Hands-on Exercises

Semi-Structured Data Processing in Spark

  • Processing JSON data
  • Binary Data Processing
  • Why Binary Data Processing?
  • Comparison of various data formats
  • Working with Parquet
  • Working with Avro
  • Hands-on Exercises

Day 4

9:00am-10:30am

Advanced DataFrame

  • Hive Context vs Spark SQL Context
  • Integrating Spark SQL with Hive
  • Working with Hive Tables
  • Working with JDBC data source
  • Data formats: text format such as CSV, JSON, XML, binary formats such as Parquet, ORC
  • UDF in Spark DataFrame
  • Spark SQL as JDBC service and its benefits and limitations
  • Analytical queries in Spark
  • Hands-on Exercises

10:30am-10:45pm Morning Break

10:45pm-12:30pm

Using SQL in Spark

  • What is Catalog API?
  • Catalog API Demo
  • Table, View
  • Key considerations for Performance Tuning
  • Hands-on Exercises

12:30pm-1:30pm Lunch

1:30pm-3:30pm

Performance Tuning

  • Capturing job metrics using Spark History Server
  • Benefits of shared variables: Accumulator, Broadcast var
  • Types of Spark caching and their use cases
  • Spark SQL Optimization
  • Understanding Catalyst Optimizer
  • Exploring and Interpreting Spark SQL Plan
  • Partitions
  • Partition Pruning
  • Broadcast Join
  • Hands-on Exercises

Spark UI Partitions

  • Explain the concepts necessary to read it
  • Illustrate how this UI can be used for debugging or performance tuning
  • Partition
  • Illustrate the RDD/Dataframe partitioning and formulate recommendations through different scenarios
  • Coalesce vs. repartition
  • Repartition on column
  • Illustrate and formulate recommendations on File System/Table partitioning
  • Benefits of FS/table partitioning
  • How to do it and when?

Configuration Illustration and Recommendations

  • Driver Memory (including memory overhead) & number of Cores
  • Executor Memory (including memory overhead) & number of Cores
  • Quick tour on other relevant configuration parameters (shuffle partitions, parallelism, etc.)

3:30pm-3:45pm Afternoon Break

3:45pm-5:00pm

Code Refactoring & Structure

  • How to structure your repository
  • Unit Testing
  • Functional approach vs classic OOP

Illustrations of Interactions with Data Sinks

  • Delta Lake (CRUD operations)
  • S3 buckets
  • Hive Tables
  • Oracle

User Defined Functions

  • Hands-on exercises including classic Data Manipulations

Cache Management

  • Broadcasting variables/Dataframe

Programme scope, delivery arrangements and suitability are confirmed during enquiry.