Duration
5 days
OVERVIEW
Web Mining refers to the two-step process, which are Web Crawling and Web Scrapping. This 3 day program focuses on various data retrieving techniques for various types of data from the internet, including data from the basic websites (extracting text from html pages), structured web tables, and other web data sources. The program focuses on the necessary processing steps involved in extracting useful content from many different types of web data. The program also covers some basic introduction to the machine learning techniques applied on collected data.
After successfully completing this course, participants will have a solid understanding of how Python programming language is used towards web mining. Participants will learn Python packages such as urlib2 and requests do perform web crawling and downloading web pages. The HTML source codes are then parsed using web scrapping library called Beautiful Soup. The scrapped data is then analysed using Pandas library. Participants will also learn another powerful python library called Scrapy. Using this library participants will be able develop their own asynchronous spiders. The participants will also learn on identify techniques for retrieving different types of data from the web, processing unstructured web text data, retrieving more structured web data from tables, via APIs, or from various sources such as Twitter, Facebook.
Practical exercises during the course prepare students to take the knowledge gained and apply it to their own web mining challenges. The lab challenges also include sentiment analysis of text documents from web reviews or blogs, and use of third-party APIs and web services to enrich data science projects. The class exercises and labs are hands-on, so students will internalize the topics covered, which will provide a jumpstart to the real world application of these techniques.
PROGRAM OBJECTIVES
- Creating a web crawler / spider using Scrapy
- Crawling a single or multiple websites and scrape data
- Deploying Spider to ScrapingHub
- Logging into Websites with Scrapy
- Exporting data extracted by Scrapy into CSV, Excel, XML, or JSON files
- Storing data extracted by Scrapy into MySQL and MongoDB databases
- Understanding the basic calls to server (GET, POST Methods)
- Web Scraping with Python Beautiful Soup and Requests
- Several real-life web scraping projects, including Craigslist, LinkedIn
WHO SHOULD ATTEND
- Programmers
- Data Scientists
PREREQUISITE
There are no prerequisites for this course but some programming background is preferred.
Day 1
- An Overview of Python
- What is Python?
- Interpreted languages
- Advantages and disadvantages
- Downloading and installing
- Structure of a Python script
- Using the interpreter interactively
- Running standalone scripts under Unix and Windows
- Getting Started
- Using variables
- String types: normal, raw and Unicode
- String operators and expressions
- Math operators and expressions
- Writing to the screen
- Command line parameters
- Reading from the keyboard
- Flow Control
- Indenting is significant
- The if and elif statements
- while loops
- Using lists
- Using the for statement
- The range() function
- Defining Functions and handling Files
- Syntax of function definition
- Formal parameters
- Global versus local variables
- Passing parameters and returning values
- Text file I/O overview
- Opening a text file
- Reading text files
- Raw (binary) data
- Using the pickle module
- Writing to a text file
- Errors and Exception Handling
- Dealing with syntax errors
- Exceptions
- Handling exceptions with try/except
- Cleaning up with finally
- Using Modules
- What is a module?
- The import statement
- Function aliases
- Packages
- Regular Expressions
- RE Objects and Pattern matching
- Parsing data
- Sub Expressions
- Complex substitutions
- RE tips and tricks
Day 2
- Scrapy
- Installing Scrapy
- Command Line tool
- Configuration settings
- Creating a project
- First Spider
- Extracting Data
- Storing the scrapped Data
- Using spider arguments
- Scrapy Shell
Introduction
- Configuring the shell
- Launch the shell
- Available shortcuts
- Available Scrapy objects
- Invoking the shell from spiders to inspect responses
- Scrapy Project
- Default structure of a Scrapy project
- Creating project using scrapy tool
- Controlling project using scrapy tool
- Scrapy tool commands – startproject
- Scrapy tool commands – genspider, runspider
- Scrapy tool commands – crawl, check, bench
- Scrapy tool commands – list, edit, fetch, view, shell
- Scrapy tool commands – parse, settings
- Spiders
- scrapy.Spider
- Spider arguments
- CrawlSpider
- XMLFeedSpider
- CSVFeedSpider
- SitemapSpider
- Selectors
- Constructing Selectors
- Using Selectors
- Nesting Selectors
- Using selectors with regular expressions
- Working with relative XPaths
- Selector Objects and SelectorList Objects
- Items
- Declaring Items
- Item Fields
- Creating Items
- Getting field values and Setting field values
- Accessing all populated values
- Extending items
- Item Objects and Field Objects
- Item Loaders
- Using Item loaders to populate items
- Input and Output processors
- Declaring Item Loaders
- Declaring Input and Output Processors
- Item Loader Context
- Item Loader Objects
- Nested Loaders
- Reusing and extending item loaders
- Available built-in processors
- Item Pipeline
- Price validation
- Dropping items with no prices
- Write items to a JSON file
- Write items to MongoDB
- Activating an Item Pipeline component
- Feed Exports
- Serialization formats – JSON
- Serialization formats – JSON Lines
- Serialization formats – CSV
- Serialization formats – XML
- Serialization formats – Pickle
- Serialization formats – Marshal
- Storages
- Storage URI Parameters
- Storage back ends – Local file system
- Storage back ends – FTP
- Storage back ends – S3
- Standard Output
Day 3
- Requests and Responses
- Passing additional data to callback functions
- Using errbacks to catch exceptions in request processing
- Bind Address
- Download timeout, latency, fail on data loss, maximum retry times.
- Form Request objects
- Using Form Request to send data via HTTP POST
- Using Form Request to simulate user login
- Response Objects
- TextResponse
- HtmlResponse
- XmlResponse
- Link Extractors
- Built in Link Extractors
- LxmlLinkExtractor
- Logging
- Log Levels
- How to log messages
- Logging from spiders
- Logging configurations
- Logging settings
- Command-line options
- Scrapy.utils.log Module
- Beautiful soup
- Installing Beautiful Soup
- Installing a parser
- Making the soup
- Kinds of objects
- Navigating the tree
- Going down
- Going up
- Going sideways
- Going back and forth
- Managing the tree
- Searching the tree
- Append the tree
- Insert inside the tree
- Clear
- Extract, decompose, replace with,
- wrap and unwrap
- Output
- Pretty-printing
- Non-pretty printing
- Output formatters
- Get Text
- Output Encoding
- Unicode
Day 4
- Data distribution
- Center
- Spread
- Shape – Symmetry, Number of peaks, Skewness, Uniform
- Unusual Features – Gaps, Outliers
- Measures of central tendency - Mean, Median, Mode, Midrange
- Measures of spread - Range, Variation, Standard deviation, Interquartile range
- Measures of shape - Empirical rule, Chebyshev's rule, Skewness, Kurtosis
- Measures of relative position – Quartiles, Percentiles, Midquartile
- Python Data distribution
- Introduction to Series
- Introduction to Pandas
- DataFrames
- Read From CSV
- Methods: head, shape, info, mean, median mode
- Histogram
- Methods: min, max, range, sqrt
- Methods: sorted, std, hist, correlation, heatmap
- Methods: skew, kurt, cov, quantile
- Python for Data Analysis – NumPy
Introduction
- Ndarray Object
- Data Types
- Array Attributes
- Array Creation Routines
- Array from existing data
- Numerical ranges
- Array Indexing and Slicing
- Advanced Indexing
- Iterating over Array
- Array Manipulation
- Arithmetic Operators
- Binary Operators
- String Functions
- Mathematical Functions
- Statistical Functions
Day 5
- Python for Data Analysis – SciPy
Introduction
- Basic functions
- Special functions
- Integration
- Optimization
- Interpolation
- Fourier transforms
- Signal Processing
- Linear Algebra
- Sparse Eigenvalue Problems with ARPACK
- Compressed Sparse Graph Routines
- Spatial data structures and algorithms
- Statistics
- Multidimensional image processing
- File IO
- Python for Data Analysis – Pandas
- Introduction to Pandas
- Series
- DataFrames
- Missing Data
- Group By
- Merging Joining and Concatenating
- Operations
- Data Input and Output
- Python for Data Visualization
- Matplotlib
- Seaborn
- Distribution Plots
- Categorical Plots
- Matrix Plots
- Grids
- Regression Plots
- Pandas Built-in Data Visualization
- Plotly and Cufflinks
- Geographical Plotting
- Choropleth Maps
Programme scope, delivery arrangements and suitability are confirmed during enquiry.