In the previous Python tutorial for digital marketers 1, we discussed what a digital marketer can benefit from Python superpower, why she or he needs it, and how to install and set up the latest Python version for Mac OS. As you might be aware, one of the most essential Python benefits to digital marketers is to scrape web data and update the data automatically. 

So in this Python Tutorial, I’ll talk about how to set up an environment to write python scripts for the purpose of scraping objective website data. This article doesn’t go into details regarding Python methods introduction, code writing, and feeding the data to a spreadsheet or database. I’ll release other articles and videos to walk through. But the purpose of this article is to let you understand the big picture of what components are necessary and how it works.

By the end of this Python Tutorial, you can master the installation of beautifulsoup4, requests, lxml, html5lib, and sublime text, and how to scrape web data by them.

BeautifulSoup4

Beautiful Soup is a library that makes it easy to scrape information from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree.

Installing Beautifulsoup4 is not complex, below are the steps

1. Go to Pypi.org and download the latest version beautifulsoup4-4.9.3

beautifulsoup4

2. Open Mac terminal, and input 

cd Desktop

(Note: Desktop means is the beautifulsoup4 file location you save)

cd beautifulsoup4-4.9.3

sudo python3 ./setup.py install

3. Check if beautifulsoup4 is installed successfully

Input: pip3 install beautifulsoup4. If the return value is a requirement already satisfied, that means the installation is done.

beautifulsoup4

Once it’s installed, we need to make sure we have parsers to parse the HTML. Parsers are essential to scrape the data and get the correct return result. Basically, it’s because the objective HTML page information matters. If the target page structures are built in a perfect form, there is no difference between the parsers, but if the target page structures have mistakes, different parsers can fill in the missing information differently and ensure the return result is correct.

In BeautifulSoup4 documentation, there is a section that explains the difference among parsers, but basically, they suggest installing and use the lxml parser and html5lib parser. So here I show how to install in Mac terminal:

Pip3 install lxml

Pip 3 install html5lib

Requests

Requests is a Python library used to easily make HTTP or HTTPS requests. Basically, its primary purpose is to call the objective data and show on your screen by running a Python script, which is functioning as you type in a URL on a browser to open the page. Generally, Requests has two main use cases, making requests to an API and getting raw HTML content from websites (i.e., scraping).

To install Requests is pretty easy, below are the steps

beautifulsoup4

  1. Open the Mac Terminal.
  2. Input: pip3 install requests (Note: please remember to use pip3 if you haven’t created the alias between your Mac Python version and the latest Python3 version which I use here as an example, otherwise it might cause to install on a wrong folder path).
  3. Wait and see Requests are successfully installed, which include the date and related version information.

Sublime Text Editor

Sublime Text is a shareware cross-platform source code editor with a Python application programming interface (API) for free. It natively supports many programming languages and markup languages, and functions can be added by users with plugins, typically community-built and maintained under free-software licenses

There are lots of available free editors such as atome, etc. You can use another similar software if you already have one. I’ll take sublime text as an example to walk you through how to use it to create scripts and scrape web data.

1. Check the build system and update the latest Python

In Sublime text, if you go to tools and build systems, you can find many programming language options are available, including Python. However, the default Python version might not be updated. As you can see from the below screencap, we select Python and input a single line code, and it shows Python 2.7, instead of the latest Python3

Import sys

print(sys.version)

Photo 1

Photo 2

2. Add a new Python3 build system

Adding a build system and the script shows a line of code:

“Shell_cmd”: “make”

Replace it with the codes below and save. The latest Python3 version is created and you can check by inputting import sys, print(sys.version)

“cmd”: [“python3”, “-i”, “-u”, “$file”],    

“file_regex”: “^[ ]File \”(…?)\”, line ([0-9]*)”,    

“selector”: “source.python”

Photo 1

Photo 2

python 3.9

Web Scraping Case:

(www.easy2digital.com/topics/ecommerce/)

Things are ready now, and we can test a web scraping in Sublime.

First of all, we need BeautifulSoup and requests, so let’s start by inputting

from bs4 import BeautifulSoup

import requests

And then below is a variable that requests to get HTML source text data of my website eCommerce article section

source = requests.get(‘http://www.easy2digital.com/topics/ecommerce/’).text

Then, we can parse this source code information into BeautifulSoup and print it out.

soup = BeautifulSoup(source,’lxml’)

print(soup.prettify())

Last but not least, input command B to run the coding and as you can see, all source code of the page are generated. This data is still not helpful because we need to create lines of code to specifically scrape the division data we need and also implement multi-page web scraping.

python tutorial for marketers

Being said that, Web scraping environment by a sublime text editor is already working, and the thing is what we aim to scrape and write the codes based on the objective in a sublime text editor. 

I hope you enjoy reading Python Tutorial for Digital Marketers 2: Web Scraping with BeautifulSoup, Requests, Sublime Text. If you did, please support us by doing one of the things listed below, because it always helps out our channel.

By Louis Lu

Growth Hacker & Digital Marketer, with a proven record of over 11 years experience in 20+ Asian markets, and 25,000+ connections in Linkedin

One thought on “Python Tutorial for Digital Marketers 2: Web Scraping with BeautifulSoup, Requests, Sublime Text”
  1. In this tutorial, we notice that there is a collection we should pay attention to, where we thought it just talked about how to install…! This collection is the easiest and simplest tutorial to learn and apply to the work so far.

Leave a Reply

Your email address will not be published. Required fields are marked *