EIEIOTECHNOVATION
← Training programmes

EIEIO / DATA SCIENCE

Hadoop and Spark Training

Course outline

4 days
Discuss this programme →

Duration

Hadoop and Spark Training

4 days

Overview

Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, Spark ML for machine learning, and Spark Streaming for live data stream processing. With Spark running on Apache Hadoop YARN, developers can create applications to derive actionable insights within a single, shared dataset in Hadoop.

This training course will teach you how to solve Big Data problems using the Apache Spark framework. The training will cover a wide range of Big Data use cases such as ETL, DWH and streaming. It will also demonstrate how Spark integrates with other well established Hadoop ecosystem products. You will learn the course curriculum through theory lectures, live demonstrations and lab exercises. This course will be taught in the Python programming language and we will be using Cloudera 7.1.4 using Spark 3.x.

Prerequisites

Following are the prerequisites for the course.

  • Programming knowledge in Scala/Python is required
  • Basic Knowledge of big data use-cases.
  • Basic knowledge of databases, OLAP/OLTP use cases, SQL
  • Knowledge of Java stack – JVM is helpful

Who Should Attend?

  • Software Developer
  • Data Engineer
  • Data Scientist

Outline

Introduction to Hadoop Ecosystem

  • Big Data Overview
  • Key Roles in Big Data Project
  • Key Business Use cases
  • Hadoop Architecture
  • Typical Big Data Project Pipeline
  • Lambda Architecture
  • Kappa Architecture

Basic Concepts of HDFS

  • HDFS Overview
  • Why HDFS?
  • HDFS High-level architecture
  • HDFS Commands
  • Hadoop 2 vs Hadoop 3
  • Hands-on Exercise

YARN Framework and Architecture

  • Logical Architecture of MapReduce
  • Logical Architecture of YARN
  • High-level Architecture of YARN
  • Hands-on Exercise: Running Spark Application on YARN

Functional Programming in Scala

  • Functional Programming
  • Passing Function as arguments
  • Anonymous Functions
  • Why Scala?
  • Basic Syntax
  • User-Defined Functions within Scala
  • Hands-on Exercise

Spark Basics

  • What is Apache Spark- the story of the evolution from Hadoop
  • Why Spark?
  • Advantages of Spark over Hadoop Map Reduce
  • Logical Architecture of Spark
  • Programming Languages in Spark
  • Functional Programming with Spark
  • Lambda architecture for enterprise data and analytics services
  • Data sources for Spark application
  • Hands-on Exercise

Choosing Language in Spark

  • Which language to choose when?
  • Spark on Scala
  • Scala vs Python vs Java vs R

Resilient Distributed Dataset(RDD)

  • RDD - Basics
  • Spark Context and Spark Session
  • RDD – operations – read from the file, transforming and saving persistent
  • Leveraging in memory processing
  • Pair RDD – operations
  • Working with semi structured data formats using regex, json, xml libraries
  • MapReduce Operations
  • Joins
  • Hands-on Exercise

Spark Internals

  • Anatomy of Spark jobs on YARN, Standalone and Mesos
  • RDD partitions
  • Spark literature: Narrow, wide operations, shuffle, DAG, Shuffle, Stages, and Tasks
  • Job metrics
  • Fault Tolerance
  • Hands-on Exercise

Spark Physical Execution

Overview

  • Spark on YARN
  • YARN client vs YARN cluster modes
  • Deployment modes – YARN, Standalone, Mesos
  • The Spark Standalone Web UI
  • Hands-on Exercise

Developing Spark Standalone Applications

  • Spark Applications vs. Spark Shell
  • Developing a Spark Application
  • Modifying Spark Key Configurations
  • Hands-on Exercise

Structured Data Processing in Spark

  • Spark SQL Overview
  • Spark SQL Documentation
  • Sub-modules of Spark SQL
  • Difference between RDD and Dataframe
  • Loading and processing data into dataframe
  • Dataframe API
  • Dataframe Operations
  • Saving dataframe to file systems
  • Dataframe internals that makes it fast – Catalyst Optimizer and Tungsten
  • Hands-on Exercise

Semi-Structured Data Processing in Spark

  • Processing JSON data
  • Binary Data Processing
  • Why Binary Data Processing?
  • Comparison of various dataformats
  • Working with Parquet
  • Working with Avro
  • Hands-on Exercise

Programme scope, delivery arrangements and suitability are confirmed during enquiry.