Chapter 8: Build a Shopify Scraper to Fetch Competitor Webshop Product Data Using Easy2Digital APIs

Previously, I talked about how to store all fetched data in the Google sheet in your Google drive, which you are not limited to your local computer or laptop to open the data, or your partners can open it to work for projects anytime.

When talking about digital and eCommerce projects, the Shopify project must be one of the most popular topics which many eCommerce sellers like to discuss. If you don’t believe me, please just check out the Shopify stock pricing and ask your surroundings how many sellers are significantly benefiting from Shopify, thanks to easy-to-use and super agile features. The fragment eCommerce era has come, creating an online store is as easy as creating a blog.
So in this Python Tutorial, I’ll walk you through how to create a Python script and scrape any Shopify website product data just by changing a website URL. By the end of this piece, you can learn how to find Shopify storefront API and read Shopify JSON, then write the python codes for specific product information, such as title, price, compared price, etc, as long as Shopify product data includes in the JSON format.

Featured Video Play Icon

In the previous chapter, I talked about how to store all fetched data in the Google sheet in your Google drive.  You are not limited to your local computer or laptop to open the data. Or your partners can open it to work on projects anytime.

When talking about digital and eCommerce projects, the Shopify project must be one of the most popular topics which many eCommerce sellers like to discuss. If you don’t believe me, please just check out the Shopify stock pricing and ask your surroundings how many sellers are significantly benefiting from Shopify. It’s all because of the easy-to-use and super agile features. The fragmented eCommerce era has come, and creating an online store is as easy as creating a blog.

So in this Python Tutorial, I’ll walk you through how to create a Python script and scrape any Shopify website product data just by changing a website URL. By the end of this piece, you can learn how to find Shopify storefront API and read Shopify JSON, then write the python codes for specific product information, such as title, price, compared price, etc, as long as Shopify product data includes in the JSON format.

Table of Contents: Build a Shopify Scraper to Fetch Competitor Webshop Product Data Using Easy2Digital APIs

Shopify Product JSON Data

Basically, there are two types of Shopify products JSON API. One is storefront API: domainname.com/products.json which is open to any visitors, and the other is store admin API: domainname.myshopify.com/admin/products.json. For competitor monitoring, as you are not the store admin, so you can only leverage the open JSON API.

As you can see, the product information listed on the website is shown here, however, compared with the admin API, the open API is only showing all visible product data feed on the website and excluding inactive SKUs. Also, it might be quite difficult to see in this JSON format, so you could install a browser plugin to prettify the format. Here I use chrome and install a JSON-formatted extension. The storefront API JSON product data would be prettified immediately

Import JSON & Create A Variable with the JSON method

Firstly we would need to create modules that we use for scraping the product data. Here we would use requests and JSON modules

import JSON

Import requests

Then, we create a variable called whatever like “URL”, then copy and paste the storefront JSON API path

url = 'https://www.buyfromlo.com/products.json'

Then, we create a variable to visit the URL, and the other variable to visualize the data in JSON format

buyfromloweb = requests.get(url)

ProductData = buyfromloweb.json()

Specify Product Data & Create Python for loops

Storefront API provides quite a lot of data you can scrape, such as product title, price, compare price, URL, SKU, photos, etc. I think the title, price, URL, and photos are popular with sellers. So here I try to write a few lines of code, which create variables to represent different product data of title, variants, vendor, price, and created date, and use try/except for avoiding processing errors when bumping into some target data that is empty.

Also, we need to use for loops methods to loop the scraping until it comes to end. This approach is used in other Python tutorials as well, so I’m not going to deep dive and elaborate. However, the difference is actually under product level. It is because you can see the product data under variants like saying price, variants, etc. So we need to create another loop for variants within the product level.

Save the Scraped Data in Google Sheets via Easy2Digital API

After we test to run the script and find product data can show as we specify in the sublime text, we can add on the google sheet API modules and the codings.

In fact, it’s the same methodology as I talked about in Youtuber search results stored in Google Sheets. So if you are interested in the coding process, please check out this article.

Chapter 7: Manipulate Data in Google Sheets Using Easy2Digital APIs and Google Sheets Key

And finally, all data is in place in the assigned Google sheet after we run the Shopify JSON coding and Google sheet API codings in the sublime text. This piece of raw data can help you to build up your product-market pricing value, compared with competitors. And if you set up import XML based on your dashboard requirements, you can automatically refresh the competitor data when you open the file.

And if you like to scale up processing multi-target websites, you only need to replace the web JSON URL in the script.

google sheet api

Use Easy2Digital API – Shopify Product Scraper

If you find the script might be complicated and also requires you to update scripts and fix bugs on and off, you can leverage Easy2Digital Youtube Bot API. Here is the token endpoint as follows:

https://www.buyfromlo.com?token=&ysiteURL=&protocal=

By using this API endpoint, you just need to add the Easy2Digital token, the target shop domain brand name, and the type of top-level domain or subdomain (www, us, HK, etc) you aim to scrape. The scraped result is the same as the one shown above.

For more details regarding Marketing APIs, please check out this page.

Easy2Digital Marketing APIs Documentation

Free Easydigital API Token and Full Python Script of Shopify Competitor Product Data Scraper

If you would like to have a free Easy2Digital token and the full version of the Python Script of Shopify Competitor Product Data Scraper, please subscribe to our newsletter by adding the message Chapter 8. We would send you asap to your mailbox.

Contact us

So easy, right? I hope you enjoy reading Chapter 8: Build a Shopify Scraper to Fetch Competitor Webshop Product Data Using Easy2Digital APIs. If you did, please support us by doing one of the things listed below, because it always helps out our channel.

  • Support and donate to our channel through PayPal (paypal.me/Easy2digital)
  • Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
  • Follow and like my page Easy2Digital Facebook page
  • Share the article on your social network with the hashtag #easy2digital
  • Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
  • You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount codes
  • Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)

The next chapter is about how to pull Google trends data using Pytrends as listed below. Enjoy! 🙂

Python Tutorial 9: Big Picture Matters, How to Pull Data from Google Trends API via Pytrends

1 thought on “Chapter 8: Build a Shopify Scraper to Fetch Competitor Webshop Product Data Using Easy2Digital APIs

  1. This is very interesting, You are a very skilled blogger.
    I’ve joined your feed and look forward to seeking more of your magnificent post.
    Also, I’ve shared your website in my social networks!

Comments are closed.