Web Scraping 101

Here you will learn the concept of web scraping and some of the techniques used in the process, get an introduction to the robots.txt file, which sets the guidelines for web scraping bots, and walk through a hands-on example with Python and the httpx library.

Web scraping is a form of data mining (which will be covered in another article) focused on extracting data from web pages. The process involves converting the collected data into a structured form, so it can be used in analyses, reports, database integrations and more. It is used in many contexts, such as monitoring prices of specific items, searching for documents, gauging reactions on social networks and others.

Some scraping techniques:

Manual copying

• Manual extraction by copying and pasting information (we’ve all been there, right?).
• Suitable for one-off information and tasks.

HTTP requests

• Sends requests directly to the server.
• Receives HTML or JSON as the response.
• In Python, libraries like urllib and httpx are used to automate the process.

Semantic recognition

• Analyzes the structure and meaning of the page content.
• Natural language processing techniques are also used.
• Used to extract information from text or content with implicit meaning.

Computer vision

• Used to extract data from images or screenshots.
• Applicable in scenarios where the data is not available as text, such as charts, captchas or flowcharts.
• Uses character recognition tools to convert images into text.

Through APIs

• Accesses the data through APIs provided by web services.
• Avoids the need for HTML parsing.
• Reduces the risk of being blocked by sites, since it is the official way to access the data.

What is robots.txt

The robots.txt file is a text file websites use to tell search engines and other crawlers which parts of the site may or may not be crawled or indexed. It lives at the root of a website’s domain (for example, https://medium.com/robots.txt).

How to read a robots.txt file

• User-agent: defines which robots the rule applies to. The asterisk ‘*’ refers to all robots.
• Disallow: specifies the directories the defined robot should not crawl.
• Allow: permits crawling of directories, even inside a “Disallow” rule.

User-agent: *
Disallow: /private/
Disallow: /temp/

User-agent: c3p0
Disallow: /no-c3p0/

In the example above, no robot should crawl the private and temp directories, while the c3p0 indexer should not access the no-c3p0 directory.

Hands-on with Python

Let’s scrape a classic of the genre, the Books to Scrape site, available here: https://books.toscrape.com/

Let’s start by installing the required libraries:

pip install httpx beautifulsoup4

Importing the libraries:

import httpx
from bs4 import BeautifulSoup
import json

Making the request, creating a BeautifulSoup object and finding all the HTML elements that represent books:

# Page URL
url = "https://books.toscrape.com/"
# Making the HTTP request
response = httpx.get(url)
# Creating a BeautifulSoup object to parse the page's HTML content
soup = BeautifulSoup(response.text, "html.parser")
# Finding all the HTML elements that represent books on the page
books = soup.find_all("article", class_="product_pod")

Storing the information for every book on the page, iterating over each element found:

# List to store the book information
books_info = []
# Iterating over each book element found
for book in books:
    # Extracting the book title
    title = book.h3.a["title"]
    # Extracting the book price
    price = book.find("p", class_="price_color").get_text().strip()
    # Extracting the book rating (the CSS class indicates the rating)
    rating = book.p["class"][1]
    # Extracting the book availability
    stock = book.find("p", class_="instock availability").get_text().strip()

    # Creating a dictionary with the book information
    book_info = {
        "title": title,
        "price": price,
        "rating": rating,
        "availability": stock
    }
    # Adding the dictionary to the list of book information
    books_info.append(book_info)

    # Printing each book's information (optional)
    print(f"Title: {title}")
    print(f"Price: {price}")
    print(f"Rating: {rating}")
    print(f"Available: {stock}")
    print()

And now saving the book information to a JSON file:

with open("books_info.json", 'w', encoding='utf-8') as json_file:
    json.dump(books_info, json_file, ensure_ascii=False, indent=4)

This code is a good starting point for anyone who wants to practice web scraping techniques. An interesting next step would be adapting the code to extract information from every page on the site.

If you have any questions or suggestions, feel free to reach out. You can find the full code for this example at: https://github.com/francocontigo/artigos/blob/main/WebScraping101/scraping.py

For other contact information, GitHub.

#python#web-scraping