Duration
4 days
Overview
This course focuses on writing and executing MapReduce programs in a Hadoop environment.
Audience
This course is intended for System administrators, developers, and DevOps engineers creating Big Data solutions using Hadoop.
Prerequisite Knowledge/Skills
None
Course Objectives
Upon successful completion of this course, participants should be able to:
- Know the key aspects of writing and submitting MapReduce jobs in various forms
- Know the lifecycle of a job, the means to write MapReduce programs in Java (or a preferred language), writing unit tests, optimizing data transmission and storage, and using other products in the Hadoop ecosystem that can expedite developing and orchestrating MapReduce jobs
Course Outline
The content of this course is designed to support the course objectives.
- 1. Hadoop Overview
o 1.1. Big Data Introduction
o 1.2. History
o 1.3. Comparison to Relational Databases
o 1.4. Hadoop Ecosystem
- 2. Hadoop File System (HDFS)
o 2.1. Architecture/Concepts
o 2.2. Access
o 2.3. Namenodes
o 2.4. Filesystem Shell
o 2.5. Accessing HDFS with Java
o 2.6. Reading/Writing/Browsing file system
- 3. HBASE
o 3.1. Overview
o 3.2. Architecture
o 3.3. Data Model
o 3.4. Installation and Shell
o 3.5. Access via Java API
o 3.6. Administration access via Java
o 3.7. Scan API
o 3.8. Filters
o 3.9. Storage Model
o 3.10. Table Design
- 4. Map Reduce on YARN
o 4.1. Introduction
o 4.2. Processing Model
o 4.3. Command line tools
o 4.4. MapReduce framework
o 4.5. Submitting MapReduce Jobs
o 4.6. Writing MapReduce jobs in Java
o 4.7. MapReduce Theory
o 4.8. Distributive Cache
o 4.9. Speculative Execution
o 4.10. YARN Components
o 4.11. Counters
o 4.12. Details of MapReduce Job Execution
- 5. Hadoop Streaming
o 5.1. Implementing a streaming job
o 5.2. Counters in streaming jobs
o 5.3. Contrast with Java Jobs
- 6. MapReduce Workflows
o 6.1. Problem decomposition into MapReduce Jobs
o 6.2. Coding workflows
o 6.3. Using the JobControl Class
- 7. Oozie
o 7.1. Oozie Installation
o 7.2. Writing Oozie workflows
o 7.3. Deploying and running Oozie jobs
- 8. Pig
o 8.1. Installation
o 8.2. Pig Latin
o 8.3. Writing Pig Scripts
o 8.4. User Defined functions
o 8.5. Data set joins
- 9. Hive
o 9.1. Installation
o 9.2. Table creating and deletion
o 9.3. Partitioning
o 9.4. Loading data into Hive
o 9.5. Joins
o 9.6. Bucketing
Course Delivery Options
This course is currently available in the following formats:
MR-1CN-OSHDE: Instructor Led – Includes hands-on lab exercises that reinforce the concepts covered in lectures.
MR-1LN-OSHDE: Online ILT – Live course delivered via the Internet where participants attend virtual classroom interacting with instructors and other participants. A headset with microphone is REQUIRED to speak with the instructor and the rest of the class. Text communication is also available through the virtual classroom.
Programme scope, delivery arrangements and suitability are confirmed during enquiry.