Apache Spark 3

Duration

Apache Spark 3

4 days

Overview

One of the most valuable technology skills is the ability to analyze huge data sets, and this course is specifically designed to bring you up to speed on one of the best technologies for this task, Apache Spark! The top technology companies like Google, Facebook, Netflix, Airbnb, Amazon, NASA, and more are all using Spark to solve their big data problems! Spark can perform up to 100x faster than Hadoop MapReduce, which has caused an explosion in demand for this skill!.

This course is designed to cover the complete skillset of Apache Spark, from RDDs, Spark SQL, Dataframes, and Spark

Streaming, to Machine Learning with Spark ML, Advanced Analytics, data visualization, Spark Koalas, and Databricks.

Prerequisites

o Knowledge on Machine Learning o Python Programming Language

Outline

Spark Fundamentals

Apache Spark execution

Apache Spark ecosystem and documentation

PySpark: operation, cluster administration and architecture

Installing Apache Spark locally

Download Spark, Java and Anaconda

Setting environment variables

Running Spark in Prompt and Jupyter Notebook

Fixing common problems

Spark Fundamentals and RDDs

PySpark Cheat Sheet

RDD Fundamentals

Initialize PySpark with SparkSession and the SparkContext

Transformations in RDDs like map, filter, flatMap and distinct

Transformations in RDDs like reduceByKey, groupByKey or sortByKey

Course Outline :: Apache Spark 3::

RDD actions such as count, first, collect or take

Practical exercise: Basic Features and RDDs

Spark DataFrames and Apache Spark SQL

PySpark Cheatsheet: SQL

Fundamentals and advantages of DataFrames

Characteristics of DataFrames and data sources

Creating DataFrames in PySpark

Operations with PySpark DataFrames

Different types of joins in DataFrames

SQL queries in PySpark

Advanced features for loading and exporting data in PySpark

Practical Exercise: Spark DataFrames and Apache Spark SQL

Advanced features in Apache Spark

Funciones avanzadas y optimización del rendimiento

BroadCast Join and caching

User Defined Functions (UDF) and advanced SQL functions

Handling and imputation of missing values

Partitioning and catalog of APIs

Advanced Analytics with Apache Spark

Introduction to advanced analytics with Spark

Data loading and data schema modification

Inspect data in PySpark

Column transformation in PySpark

Advanced missing data imputation in PySpark

Data selection with PySpark and PySpark SQL

Data visualization and graph generation in PySpark

Persist data with PySpark

Practical Exercise: Advanced Analytics with Apache Spark

Koalas: The Apache Spark Pandas API

Spark Koalas Fundamentals

Course Outline :: Apache Spark 3::

Feature Engineering with Koalas

Creating DataFrames with Koalas

Data manipulation and DataFrames with Koalas

Working with missing data in Koalas

Data visualization and graph generation with Koalas

Importing and exporting data with Koalas

Hands-on exercise with Koalas

Machine Learning with Apache Spark

Fundamentals of Machine Learning with Spark

Spark Machine Learning Components

Stages of developing a Machine Learning model

Import data and exploratory data analysis (EDA)

Machine Learning with Spark hands-on exercise

Data preprocessing with PySpark

Lab Exercise 2 Machine Learning with Spark

Training the machine learning model in PySpark

Evaluation of the Machine Learning model

Lab Exercise 3 Machine Learning with Spark

Spark Streaming

Practical example of counting words with Spark Streaming

Spark Streaming Configurations: Output Modes and Operation Types

Time Window Operations in Spark Streaming

Spark Streaming Capabilities

Lab: Real-time bank fraud detection

Spark Streaming Exercise

Introduction to Databricks

Databricks Terminology and Databricks Community

Delta Lake

Create a free Databricks account

Apache Spark on Databricks

Course Outline :: Apache Spark 3::

Introduction to the Databricks environment

Getting started with Databricks

Databricks Platform

Importing notebooks, language configuration and markdown

Databricks File Dystem (DBFS)

Create, manipulate and visualize tables

Databricks widgets

Spark DataFrame API

Creating and saving DataFrames in Databricks

Data transformation and visualization in Databricks

Use case: Population data analytics

Machine Learning in Databricks

Import and exploratory analysis of the data

Variable preprocessing with PySpark and Databricks

Definition of the Machine Learning model and development of the Pipeline

Model evaluation with PySpark and Databricks

Hyperparameter tuning and registration in MLFlow

Predictions with new data and visualization of the results

Programme scope, delivery arrangements and suitability are confirmed during enquiry.