EIEIOTECHNOVATION
← Training programmes

EIEIO / DATA SCIENCE

Big Data with Hadoop and Spark

Course outline

3 days
Discuss this programme →

Duration

3 day

Overview

This foundational course provides an introductory overview of Apache Hadoop – HDFS and MapReduce and Spark. A basic understanding of Big Data from business and technology perspectives is provided, along with an overview of common benefits, challenges, and adoption issues. The course content is divided into a series of modular sections, each of which is accompanied by one or more hands-on exercises.

Prerequisite

Basic familiarity with databases and data management

Day 1

Fundamental Terminology and Concepts (30 min)

  • Veracity, variability, visualization, and value (and the 3 V’s)
  • HDFS and MapReduce in Hadoop
  • Unstructured, semi-structured, and structured data

Brief History of Big Data (30 min)

  • Google and MapReduce
  • Web 2.0
  • Hadoop vendors

Business Drivers for Big Data (30 min)

  • KYC (Know Your Customer)
  • Sales & Marketing
  • Financial forecasting Characteristics of Big Data (30 min) Benefits of Adopting Big Data (30 min)

Challenges and Limitations of Big Data (30 min)

HDFS and Distributed Storage (90 min)

  • Installing Hadoop and types of installations
  • HDFS and Data Ingestion using Sqoop or Flume

Hands-on Exercises (90 min)

  • Installing Hadoop
  • Working with Cloudera CDH
  • Basic Data Ingestion with Sqoop and Flume

Day 2

Introduction to MapReduce (90 min)

  • Map and Reduce
  • Partitioning mappers and reducers

MapReduce Architecture (90 min)

  • Working with MapReduce
  • Distributed Cacheing
  • Input and output formatters

MapReduce Examples (90 min)

  • Sample programs

Hands-on Exercises (90 min)

  • Writing MapReduce Jobs
  • Writing Partitioners
  • Writing Input and Output Formatters

Day 3

SparkSQL, DataFrames, and Datasets (90 min)

  • SparkSQL
  • Executing SQL commands on a dataframe
  • Using Dataframes instead of RDD’s

Spark MLLib (90 min)

  • Using MLLib to produce movie recommendations
  • Analyzing ALS Recommendation results
  • Using Dataframes with MLLib

Spark Streaming and GraphX (90 min)

  • Streaming data and NRT processing
  • VertexRDD and EdgeRDD

Hands-on Exercises (90 min)

  • Sample streaming
  • Sample GraphX script

Programme scope, delivery arrangements and suitability are confirmed during enquiry.