Big Data & Spark

Duration

3 days

OVERVIEW

Big data analysis is a hot and highly valuable skill – and this course will focus the hottest technology in big data Apache Spark. This 3-day course is for data engineers, analysts, architects; software engineers; IT operations; and technical managers interested in a thorough, hands-on overview of Apache Spark. It covers the core APIs for using Spark, fundamental mechanisms and basic internals of the framework, SQL and other high-level data access tools, as well as Spark’s streaming capabilities and machine learning APIs. The class is a mixture of lecture and hands-on labs.

Each topic includes lecture content along with hands-on use of Spark through an elegant web-based notebook environment. Notebooks allow attendees to code jobs, data analysis queries, and visualizations using their own Spark cluster, accessed through a web browser. Students may keep the notebooks and continue to use them with the free Databricks Community Edition offering; all examples are guaranteed to run in that environment. Alternatively, each notebook can be exported as source code and run within any Spark environment.

Participant will understand basics of Big Data and Hadoop. Participant will learn how Spark enables in-memory data processing and runs much faster than Hadoop MapReduce. Participant will also learn about RDDs, different APIs, which Spark offers such as Spark Streaming, MLlib, Clustering, and Spark SQL. The program will also encompass the fundamental concepts like data capturing using Flume, data loading using Sqoop, Kafka cluster, Kafka API.

OBJECTIVES

After taking this class, participants will be able to

• Describe Spark’s fundamental mechanics.
• Use the core Spark APIs to operate on data.
• Articulate and implement typical use cases for Spark.
• Build data pipelines and query large data sets using Spark SQL and DataFrames.
• Analyze Spark jobs using the administration UIs and logs inside Databricks.
• Create Structured Streaming and Machine Learning jobs.
• Understand the basics of Spark internals.

PREREQUISITES

• Some familiarity with Apache Spark is helpful.
• Basic usage of Spark DataFrames is helpful.
• Basic programming experience in an object oriented or functional language is required. The class will be taught in a mixture of Python and Scala.

TARGET AUDIENCE

Data engineers and data analysts, with some experience in big data processing, who want to learn how to use Apache Spark to streamline their big data processing, how to build production Spark jobs, and how to understand and debug running Spark applications.

Day 1

• Introduction to Scala for Apache Spark

1.1. What is Scala

1.2. Why Scala for Spark

1.3. Scala in other frameworks

1.4. Introduction to Scala REPL

1.5. Basic Scala operations

1.6. Variable Types in Scala

1.7. Control Structures in Scala

1.8. Foreach loop, Functions and Procedures

1.9. Collections in Scala-Array

1.10. ArrayBuffer, Map, Tuples, Lists and more

• OOPS and Functional Programming in Scala
• Class in Scala
• Getters and Setters
• Custom Getters and Setters
• Properties with only Getters
• Auxiliary Constructor and Primary Constructor
• Singletons
• Extending a Class
• Overriding Methods
• Traits as interfaces and Layered Traits

2.10 Functional Programming

2.11 Higher Order Functions

2.12 Anonymous Functions

• Introduction to Big Data and Hadoop
• What is Big Data?
• Big Data Customer Scenarios
• Limitations and Solutions of Existing Data Analytics Architecture
• How Hadoop solves the Big Data problem
• What is Hadoop
• Hadoop’s key characteristics
• Hadoop Ecosystem and HDFS
• Hadoop Core Components
• Rack Awareness and Block Replication

3.10 Hadoop cluster and its Architecture

3.11 Data loading using Sqoop

• Apache Spark Framework
• Big Data Analytics with Batch & Real-Time processing
• Why Spark is Needed
• What is Spark
• How Spark Differs from its Competitors
• Spark at eBay
• Spark’s Place in Hadoop Ecosystem
• Spark Components & its Architecture
• Running Programs on Scala IDE & Spark Shell
• Spark Web UI

4.10 Configuring Spark Properties

Day 2

• Playing with RDDs
• Challenges in Existing Computing Methods
• Probable Solutions & How RDD Solves the Problem
• What is RDD, its functions, transforms and actions
• Data Loading and Saving through RDDs
• Key-Value Pair RDDs and other Pair RDDs
• RDD Lineage
• RDD Persistence
• WordCount Program using RDD Concepts
• RDD Partitioning & How it helps Archive Parallelization
• DataFrames and Spark SQL
• Need for Spark SQL
• What is Spark SQL
• Spark SQL Architecture
• SQL Context in Spark SQL
• Data Frames & Datasets
• Interoperating with RDDs
• JSON and Parquet File Formats
• Loading Data through Different Sources
• Machine Learning using Spark MLlib
• What is Machine Learning
• Where is Machine Learning Used
• Different types of Machine Learning Techniques
• Face Detection: USE CASE
• Understanding MLlib
• Features of MLlib and MLlib Tools
• Various ML algorithms supported by MLlib
• K-Means clustering & How it works with MLlib

Day 3

• Understanding Apache Kafka and Kafka Cluster
• Need for Kafka
• What is Kafka
• Core concepts of Kafka
• Kafka Architecture
• Where is Kafka used
• Understanding the Components of Kafka Cluster
• Configuring Kafka Cluster
• Producer and Consumer
• Capturing Data with Apache Flume and Integration with Kafka
• Need for Apache Flume
• What is Apache Flume
• Basic Flume Architecture
• Flume Sources
• Flume Sinks
• Flume Channels
• Flume Configuration
• Integrating Apache Flume and Apache Kafka
• Apache Spark Streaming

10.1 Why Streaming is Necessary

10.2 What is Spark Streaming

10.3 Spark Streaming Features

10.4 Spark Streaming Workflow

10.5 Streaming Context & DStreams

10.6 Transformations on DStreams

10.7 WordCount program using Spark Streaming

10.8 Slice, Window and ReduceByWindow Operators

10.9 Stateful Operators

10.10 Perform Twitter Sentimental Analysis Using Spark Streaming

Programme scope, delivery arrangements and suitability are confirmed during enquiry.