EIEIOTECHNOVATION
← Training programmes

EIEIO / DATA SCIENCE

Apache Spark Training

Course outline

5 days
Discuss this programme →

Duration

5 days

What is Apache Spark?

Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, Spark ML for machine learning, GraphX for graph data processing, and Spark Streaming for live data stream processing. With Spark running on Apache Hadoop YARN, developers can create applications to derive actionable insights within a single, shared dataset in Hadoop.

Course Overview

This training course will teach you how to solve Big Data problems using Apache Spark framework. The training will cover a wide range of Big Data use cases such as ETL, DWH, data virtualization, streaming, graph data structure, machine learning. It will also demonstrate how Spark integrates with other well established products in Hadoop ecosystem. You will learn the course curriculum through theory lectures, live demonstrations and lab exercises. This course will be taught in Scala programming languages, but a few demos and lab exercises will be provided in Python and R.

Mode of Delivery

Classroom based instructor led program. For lab exercises students will use virtual machine which contains Cloudera distribution of Hadoop. The virtual machine image will be provided to the students.

Prerequisites

Following are the pre-requisites for the course.

  • Basic knowledge of databases, OLAP/OTLP use cases, SQL are required
  • Basic Knowledge of big data use-cases
  • Basic knowledge of Scala programming language will help you learn faster
  • Basic Knowledge of Java stack – JVM, Eclipse, Maven, SBT

Day 1 – Scala for Apache Spark

Module

Topics

Introduction to Scala Programming

  • What is Scala?
  • Why Scala for Spark?
  • Scala in other frameworks
  • Introduction to Scala REPL
  • Basic Scala operations
  • Variable Types in Scala
  • Control Structures in Scala
  • Functions and Procedures
  • Collections in Scala- Array, ArrayBuffer, Map, Tuples, Lists, and more.

Class in Scala

  • Getters and Setters
  • Primary and Auxiliary Constructor
  • Singletons
  • Companion Objects
  • Extending a Class
  • Overriding Methods
  • Traits as Interfaces
  • Layered Traits

Functional Programming

  • Abstractions
  • Concise expression
  • Immutable
  • Referential transparency
  • The problem – grouping continuous integers
  • Scala idioms

Day 2 – Introduction to Spark and Hadoop

Module

Topics

Introduction to Hadoop

  • What is Big Data and Big Data evolution
  • Characteristics of Big Data
  • What is Hadoop and its history
  • Key industry use cases in the context of Big Data
  • Storage, backup, archiving
  • ETL and ELT
  • Security – authentication, authorization, audit, encryption
  • Interactive Processing
  • Iterative processing for advanced analytics
  • OLAP and OLTP
  • Stream processing
  • Data ingestion
  • Search
  • Job Scheduling

Introduction to Spark

  • What is Apache Spark – the story of the evolution from Hadoop
  • Advantages of Spark over Hadoop Map Reduce
  • Lambda architecture for enterprise data and analytics services
  • Deployment modes – YARN, Standalone, Mesos
  • Developing on Spark using REPL, Zeppelin, IDE
  • Data sources for Spark application

Lab Exercise

  • Install and get started with VM
  • Launching spark REPL and Zeppelin

Spark Resilient Distributed Dataset (RDD)

  • RDD – operations – read from the file, transforming and saving persistent
  • Leveraging in memory processing
  • Pair RDD – operations
  • Working with semi structured data formats using regex, json, xml libraries

Lab Exercise

  • Explore data from data.sfgov.org using RDD
  • Explore Apache Server logs using regex
  • Join and aggregate data from grouplens.org

Day 3 – Spark Internal, Performance Tuning, Building Spark Application and Submitting Application to Cluster

Module

Topics

Spark Internals

  • Anatomy of Spark jobs on YARN andStandalone
  • RDD partitions
  • Spark literature: Narrow, wide operations, shuffle, DAG, Shuffle, Stages, and Tasks
  • Role of partitioners in Spark performance
  • Job metrics
  • Fault Tolerance

Performance Tuning

  • Factors that affect performance of spark application
  • Configuring memory and CPU for Spark drivers and executors in standalone and YARN mode
  • Controlling logging of Spark daemons and spark applications
  • Capturing job metrics using Spark History Server
  • Benefits of shared variables – accumulator, broadcast var
  • Types of spark caching and their use cases
  • Role of checking pointing of Spark RDD

Lab Exercises

  • Examine spark metrics and logs
  • Evaluate impact of different caching types on memory and processing time
  • Use shared variables

Building Application

  • Building Spark application using Eclipse IDE
  • Building Spark Application using SBT
  • Building Spark application using Jupyter Notebook (Optional)

Lab Exercise

  • Create a project using Eclipse and submit to cluster
  • Create a project using SBT and submit to cluster

Day 4: Spark Dataframe, Spark SQL and Hive

Module

Topics

Dataframe Basics

  • Introduction to Dataframe
  • Difference between RDD and Dataframe
  • Dataframe internals that makes it fast – Catalyst Optimizer and Tungsten Loading and processing data into dataframe
  • Saving dataframe to file systems

Lab Exercise

  • Process data.sfgov.org data using dataframe

Introduction to Apache Hive

  • Introduction to Hive architecture – roles of
  • Hive metastore
  • Hive JDBC service
  • Accessing Hive using CLI, Beeline, Hue
  • Hive 1.x vs 2.x
  • Basic data analysis using Hive
  • Scalar and complex data types in Hive
  • Analytical queries – window operation, pivot, roll up etc.
  • Benefits of table partitioning in Hive
  • Handling data formats using Hive serde
  • ETL patterns using Hive

Lab Exercises

  • Data transformation using Hive
  • Querying Apache web server logs using Hive
  • Create and load a partitioned table using Hive
  • Performing Join queries

Dataframe Advanced

  • Hive Context vs Spark SQL Context
  • Working with Hive Tables
  • Working with JDBC data source
  • Data formats – text format such csv, json, xml, binary formats such as parquet, orc
  • UDF in Spark Dataframe
  • Spark SQL as JDBC service and its benefits and limitations
  • Analytical queries in Spark – windows functions, pivot, rollup and cubes
  • Working with HBase (Optional)
  • An introduction to HBase use cases and its in role in Big Data ecosystem
  • Architecture overview of HBase
  • Working with HBase using Spark

Hands On

  • Persist spark tables using Hive
  • Processing Mysql data using Dataframe
  • Working with UDF
  • Working with file formats
  • Text formats – CSV, json, xml
  • Binary formats – ORC, Parquet, Avro
  • Integrating Spark with BI tools

Day 5: Spark Streaming, Graphx and Machine Learning

Module

Topics

Spark Streaming

  • Architecture of streaming application
  • Streaming Context – initialization, configuration, characteristics
  • Dstream – operations
  • Receiver characteristics
  • Window operation – batch internal, window length, sliding interval
  • Fault Tolerance using checkpointing and replication
  • Partition behavior of Dstream
  • Kafka Streaming

Lab Exercise

  • Trending analysis on live twitter stream using Spark stream
  • Saving live streams into HDFS and RDBMS
  • Saving live streams into HBase using Spark SQL (optional)

Streaming Advanced Receiver

  • Brief introduction to KAFKA receiver

Lab Exercise

  • Create a spark streaming application using KAFKA

Graphx

  • Industry use-cases for graph data structure
  • Graph DB literature: nodes, edges and properties
  • A few common algorithms supported by graph data structure, such as Pregel, Page Rank, Community Detection etc.

Introduction to Machine Learning

  • Descriptive and Inferential Statistics
  • Overview machine learning use cases
  • Identify machine learning that fits your need
  • Pipeline of machine learning operation
  • Introduction to Spark ML, Spark MLlib
  • Introduction to Spark R
  • Linear Regression

Lab Exercises

  • Create ML pipeline using Spark ML

Programme scope, delivery arrangements and suitability are confirmed during enquiry.