In previous chapters, we discussed how to scrape website HTML information and Shopify product information via JSON API. Actually, on most websites and platforms, there is more than one page to show articles, products, and so on. Basically, we call it pagination. For example, page 1, or previous page, or next page, and the previous codings and dataset only can scrape the single URL page.
In this article, I would walk you through how to scrape the web pagination and Shopify products using Easy2Digital API. It’s for capturing all target datasets in bulk. By the end of this article, you can master Pandas library and some new methods. Also, you can customize the script based on your business needs.
Table of Contents: Build a Shopify Bot to Scrape Store Product Data at Scale Using Easy2Digital APIs
- Import modules
- Identify the Website Pagination URL Structure
- Write lines of code to scrape the target dataset
- Append the dataset
- Save the dataset in excel format using DataFrame and to_csv methods
- Easy2Digital API – Shopify Product Scraper
- Free Easy2Digital API Free Token and Full Python Script of Shopify Scraper
- SHOPIFY Latest Trending API Endpoint Recommendation
Import Web Scraping Modules
We would use bs4, requests, and Pandas library in this script. As we would take Shopify as the other example as well, so we need to import JSON
Pandas is a fast, powerful, flexible, and easy-to-use open-source data analysis and manipulation tool. It’s built on top of the Python programming language. It is pretty useful to restructure the dataset and save it in CSV format.
Identify the website pagination URL structure
I take Easy2Digital’s blog folder as the 1st example. As you can see from the blog path, the number following up after page/ is the location of the backward pagination page. Thus, we can create a variable that can be after the page and change, the loop to scrape accordingly
Here are the codings where we set the pagination as ‘x’, and we use the ‘for’ looping, range function, and str function.
Range function actually creates a sequence of numbers from 0 to N, and prints each item in the sequence. In this case, we can set a number like 20, and this number is already more than my blog pagination pages. I recommend
The str() function of Python returns the string version of the object. It ensures the return is a string.
Last but not least, we need to create a variable with an empty value at the moment, which is used for generating the whole scraping dataset at the end.
If we have to scrape via the platform API like Shopify, below are the codings by taking another website for example – Wasserstein Home
In the Shopify frontend product API, the JSON structure is like this, where each page has at most 250 pieces of product data. The page parameter represents the pagination value
So it’s quite similar to website HTML pagination, but just needs to scrape via the platform API
Write lines of code to scrape target datasets
Now we already have scraped the block data and it’s time to find what data we need.
Below is the Easy2Digital blog example for your reference. For more details, please check out another article, because we have talked about it previously.
Chapter 4: Create a Website Bot to Scrape Specific Website Data Using BeautifulSoup
Chapter 8: Build a Shopify Scraper to Fetch Competitor Webshop Product Data Using Easy2Digital APIs
Append the Web Scraping dataset
Previously in the CSV module and Google, we talked about how to append the scraped dataset. Here we are using the Pandas library, which is more convenient to manipulate the data in row and column
First thing first, we create a variable to define the scraped dataset name. Then, we can append the function and the data can be organized into a separate column with the unique head name defined in element_info
Then we use the len() function, in order to show how many pieces you can scrape, and the number helps you understand if the dataset size makes sense or not.
Save the dataset in excel format using DataFrame and to_csv methods
Those who are familiar with R know the data frame as a way to store data in rectangular grids that can easily be overviewed. Each row of these grids corresponds to measurements or values of an instance. And each column is a vector containing data for a specific variable. This means that a data frame’s rows do not need to contain, but can contain, the same type of values: they can be numeric, character, logical, etc.
DataFrames in Python are very similar, they come with the Pandas library, and they are defined as two-dimensional labeled data structures with columns of potentially different types. In general, you could say that the Pandas DataFrame consists of three main components: the data, the index, and the columns.
We use the data frame function and to_csv function, that is along with the Pandas library, below are the final script of the Shopify product pagination scraper and the generated excel file
Use Easy2Digital API – Shopify Product Scraper
If you find the script might be complicated and also requires you to update scripts and fix bugs on and off, you can leverage Easy2Digital Shopify Product Scraper API. Here is the token endpoint as follows:
https://www.buyfromlo.com?token=&ysiteURL=&protocal=
By using this API endpoint, you just need to add the Easy2Digital token, the target shop domain brand name, and the type of top-level domain or subdomain (www, us, HK, etc) you aim to scrape. The scraped result is the same as the one shown above.
For more details regarding Marketing APIs, please check out this page.
Full Python Script of Shopify Product Feed Data Scraper
If you would like to have a free API token and the full version of the Python Script of Shopify Bot, please subscribe to our newsletter by adding the message Python Tutorial 10. We would send you asap to your mailbox.
So easy, right? I hope you enjoy reading Chapter 10 – Build a Shopify Bot to Scrape Store Product Data in Bulk Using Easy2Digital APIs. If you did, please support us by doing one of the things listed below, because it always helps out our channel.
- Support and donate to our channel through PayPal (paypal.me/Easy2digital)
- Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
- Follow and like my page Easy2Digital Facebook page
- Share the article on your social network with the hashtag #easy2digital
- Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2023)
- You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount codes
- Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)
Chapter 11: Google SERP Bot to Scrape SERP Data Using Google Search and Easy2Digital APIs
Shopify API Endpoint Recommendation
Shopify Product Scraper API
Price: US$5
Scrape products information from any webshop built by using Shopify platform. Product scraping is up to 500 SKUs. 15+ product data metrics are included in the scraped result that include pricing, date_created, product name, etc. And Support to scrape ether top domain or subdomain shopify webshop domain
More API options from the Shopify collection.