Duration
5 days
Overview
In a 5 day course students will setup entire environment from ground 0 by deploying Cloudera on Google cloud and later use same environment for development purposes
Intended Audience
The intended audience of the course is:
- Hadoop Developers
- Data Engineers
- Integration Engineers
- Hadoop Administrators
- Data Steward
Pre-requisite and Lab Details
The intended audience should have a background on:
- Software Development
- Unix and Hadoop
The lab setup for this course would be:
- Windows 7 or Higher
- 4 GB RAM
- Google Cloud Free Account
Course Outline
Building Big Data Platform for Enterprises
The Case for Apache Hadoop
- Why Hadoop?
- Fundamental Concepts
- Core Hadoop Components
Hadoop Cluster Installation
- Rationale for a Cluster Management Solution
- Cloudera Manager Features
- Lab: Cloudera Manager Installation
- Lab: Hadoop (CDH) Installation
The Hadoop Distributed File System (HDFS)
- HDFS Features
- Writing and Reading Files
- NameNode Memory Considerations
- Overview of HDFS Security
- Lab: Web UIs for HDFS
- Lab: Using the Hadoop File Shell
MapReduce and Spark on YARN
- The Role of Computational Frameworks
- YARN: The Cluster Resource Manager
- MapReduce Concepts
- Apache Spark Concepts
- Running Computational Frameworks on YARN
- Exploring YARN Applications Through the Web UIs, and the Shell
- YARN Application Logs
- Lab: Running MapReduce and Spark Jobs
Hadoop Configuration and Daemon Logs
- Cloudera Manager Constructs for Managing Configurations
- Locating Configurations and Applying Configuration Changes
- Managing Role Instances and Adding Services
- Configuring the HDFS Service
- Configuring Hadoop Daemon Logs
- Configuring the YARN Service
Getting Data Into HDFS
- Ingesting Data From External Sources With Flume
- Ingesting Data From Relational Databases With Sqoop
- REST Interfaces
- Best Practices for Importing Data
- Lab: Ingesting Data using Flume and Sqoop
Planning Your Hadoop Cluster
- General Planning Considerations
- Choosing the Right Hardware
- Virtualization Options*
- Network Considerations
- Configuring Nodes
Installing and Configuring Hive, Impala, and Pig
- Hive
- Impala
- Pig
- Lab: Hive, Pig & Impala Hands-On
Hadoop Clients Including Hue
- What Are Hadoop Clients?
- Installing and Configuring Hadoop Clients
- Installing and Configuring Hue
- Hue Authentication and Authorization
- Lab: Hue Hands-On
Advanced Cluster Configuration
- Advanced Configuration Parameters
- Configuring Hadoop Ports
- Configuring HDFS for Rack Awareness
- Configuring HDFS High Availability
- Lab: Advanced Cluster configurations Hands-On
Hadoop Security
- Why Hadoop Security Is Important
- Hadoop’s Security System Concepts
- What Kerberos Is and how it Works
- Securing a Hadoop Cluster With Kerberos
- Other Security Concepts
Managing Resources
- Configuring cgroups with Static Service Pools
- The Fair Scheduler
- Configuring Dynamic Resource Pools
- YARN Memory and CPU Settings
- Impala Query Scheduling
Cluster Maintenance
- Checking HDFS Status
- Copying Data Between Clusters
- Adding and Removing Cluster Nodes
- Rebalancing the Cluster
- Directory Snapshots
- Cluster Upgrading
Cluster Monitoring and Troubleshooting
- Cloudera Manager Monitoring Features
- Monitoring Hadoop Clusters
- Troubleshooting Hadoop Clusters
- Common Misconfigurations
Python:
- Installing Python
- Data Processing
- Python Object and Data Structure Basics
- Introduction to Python Data Types
- Numbers
- Variable Assignments
- Introduction to Strings
- Indexing and Slicing with Strings
- Lists in Python
- Dictionaries in Python
- Tuples with Python
- Sets In Python
- Boolean in Python
- I/O with Basic Files in Python
- Labs: Python Hands-On
R:
- Installing R
- R Packages
- Labs: R Hands-On
MySQL:
- Installing MySQL/ PostgreSQL
- Demo: Big Data Platform for databases and tables.
Programme scope, delivery arrangements and suitability are confirmed during enquiry.