Introduction to Machine Learning Using R and Apache Spark

Prerequisite

Basic knowledge of statistics, mathematics and computer programming are required to take full advantage of this course.

Hands on programming skill in programming languages such as Java, python, R etc

Course Outline :: Introduction to Machine Learning Using R and Apache Spark::

Day 1

Topic Description

Introduction to Machine

Learning

Machine Learning vs Statistics vs Data Science vs Data

Engineering vs Data Analysis

Real world scenarios to understand the perspective

History, Pioneers and Modern Trends

Types of Machine Learning

◦ Supervised Learning

◦ Unsupervised Learning

◦ Reinforced learning

Introduction to R  What is R and its brief history and position in the industry

Inspecting Data using R

◦ Examine data types – quantitative, qualitative, continuous, discrete, ordinal, nominal etc.

◦ Probability distribution

Common Data Pre-processing Tasks

◦ Feature Transformation

◦ Encode Categorical features

◦ Handle Missing Data

◦ Handle Outliers

◦ Remove Problematic Features

Lab Exercises  Explore built in datasets in R

Linear Regression  Assumptions of Linear Regression

Simple Linear Regression

Multiple Linear Regression

Cross validation

Tuning and hyper parameter selection

Problems with Linear Regression – multi-collinearity

Regularization – Ridge, Lasso, Elastic Net for robustness

Lab Exercises  Install R, R-studio and R packages

Explore a few dataset from UCI ML library using R

Apply Linear Regression to predict the demand of Electricity

Power Demand

Learning Goals of Day 1

Spot the use cases of machine learning

Explore and transform data using R

Course Outline :: Introduction to Machine Learning Using R and Apache Spark::

Learn to predict numeric data (for example, revenue, price etc)

Learn to evaluate a regression model

Learn to validate and tune a regression model for best performance

Day 2

Topic Description

Classification  What are classification problems and types of classification algorithms

Decision Tree Classification

◦ Intuition of tree models

◦ CART regression trees

◦ Tree pruning

◦ Missing Data

Classification using Random Forrest

Logistic Regression

◦ Generalized linear models

◦ Interpreting coefficients in logistic regression

◦ Performance metrics – confusion matrix

◦ AUC value, ROC plot

Support Vector Classifiers

Compare algorithms

Lab Exercise  Evaluate of credit status using customer profile

Segmentation of insurance customers

Learning Goals of Day 2

Learn to predict categorical data types, for example – customer segmentation, spam detection etc

Evaluate and tune classification models

Course Outline :: Introduction to Machine Learning Using R and Apache Spark::

Day 3

Topic Description

Clustering  What is clustering and use cases

Proximity Matrices – find dissimilarity between two observation o Choice of attributes o Units of measure of attributes o Importance of scaling

Determining number of clusters

K-means clustering

Practical Issues with K-means clustering

Measure performance of clustering

Hierarchical clustering

Practical Issues in clustering

Lab Exercises  Clustering movies into genres based on movie attributes

Apply clustering on images for clinical diagnostics

Feature

Engineering

Principle Component Analysis

Independent Component Analysis

Rescaling

Lab Exercise  PCA on IRIS dataset UCI ML libraries

Learning Goals for Day 3

Apply clustering algorithms to gain insights about latent groups inside a dataset

Identify properties of the features that impact model performance

Transform data structure to facilitate model performance and computation speed

Course Outline :: Introduction to Machine Learning Using R and Apache Spark::

Day 4

Topic Description

Text

Analytics

Text Analysis Steps

Term Frequency – Inverse Term Frequency

POS tagging, lemmatization, stemming

Collecting raw text

Representing text

Categorize Documents by type

Determining sentiments

Gaining insights

Lab Exercise  Working with Twitter data to analyze sentiments

Introduction

to Spark

Apache Spark – a swiss army knife for data processing at scale

Apache Spark architectural overview

Data Abstraction for massively parallel processing – RDD, DF, DS

Building and submitting Spark application

Highly available, high throughput data sources

Efficient data formats for large scale data processing

Lab Exercise  Working with stocks market data for S&P 500 companies

Learning Goals of Day 4

Learn technique to analyze textual data and prepare it for further analysis such as classification, regression, clustering etc.

Extract information from text in the form of word cloud, sentiment

Day 5

Topic Description

Spark ML  Requirement of large scale machine learning and its practical challenges with large scale machine learning

Integrate R with Spark

Spark ML packages for regression, classification, clustering, recommender system

Lab

Exercise

Pre-process data from stack-overflow dataset

Find community from stack-overflow dataset using tag analysis

Learning Goals for Day 5

Learn how to build machine learning pipeline on massive data using Spark framework

Apply machine learning algorithms implemented in Spark ML

Combine the power of R and Spark for solving real life advance analytics problems

Programme scope, delivery arrangements and suitability are confirmed during enquiry.