September 21, 2021

Python Tutorial for Digital Marketers 8: One Script to Scrape Competitor Shopify Web Product Data

Previously, I talked about how to store all fetched data in the Google sheet in your Google drive, which you are not limited to your local computer or laptop to open the data, or your partners can open it to work for projects anytime.

When talking about digital and eCommerce projects, the Shopify project must be one of the most popular topics which many eCommerce sellers like to discuss. If you don’t believe me, please just check out the Shopify stock pricing and ask your surroundings how many sellers are significantly benefiting from Shopify, thanks to easy-to-use and super agile features. The fragment eCommerce era has come, creating an online store is as easy as creating a blog.
So in this Python Tutorial, I’ll walk you through how to create a Python script and scrape any Shopify website product data just by changing a website URL. By the end of this piece, you can learn how to find Shopify storefront API and read Shopify JSON, then write the python codes for specific product information, such as title, price, compared price, etc, as long as Shopify product data includes in the JSON format.

python tutorial

In the previous Python tutorial, I talked about how to store all fetched data in the Google sheet in your Google drive.  You are not limited to your local computer or laptop to open the data. Or your partners can open it to work for projects anytime.

When talking about digital and eCommerce projects, the Shopify project must be one of the most popular topics which many eCommerce sellers like to discuss. If you don’t believe me, please just check out the Shopify stock pricing and ask your surroundings how many sellers are significantly benefiting from Shopify. It’s all because of the easy-to-use and super agile features. The fragment eCommerce era has come, creating an online store is as easy as creating a blog.

shopify

So in this Python Tutorial, I’ll walk you through how to create a Python script and scrape any Shopify website product data just by changing a website URL. By the end of this piece, you can learn how to find Shopify storefront API and read Shopify JSON, then write the python codes for specific product information, such as title, price, compared price, etc, as long as Shopify product data includes in the JSON format.

Python Tutorial: Shopify Product JSON

Basically, there are two types of Shopify product JSON API. One is storefront API: domainname.com/products.json which is open to any visitors, the other store admin API: domainname.myshopify.com/admin/products.json. For competitor monitoring, as you are not the store admin, so you can only leverage the open JSON API.

This is my website example. www.buyfromlo.com/products.json.

json shopify

As you can see, the product information listed on the website is shown here, however, compared with the admin API, the open API is only showing all visible product data feed on the website and excluding inactive SKUs. Also, it might be quite difficult to see in this JSON format, so you could install a browser plugin to prettify the format. Here I use chrome and install a JSON formatted extension. The storefront API JSON product data would be prettified immediately

json formatter

shopify json

Import JSON & Create A Variable with JSON method

Firstly we would need to create modules that we use for scraping the product data. Here we would use requests and JSON modules

import JSON

Import requests

Then, we create a variable called whatever like “URL”, then copy and paste the storefront JSON API path

url = 'https://www.buyfromlo.com/products.json'

Then, we create a variable to visit the URL, and the other variable to visualize the data in JSON format

buyfromloweb = requests.get(url)

ProductData = buyfromloweb.json()

python module

Specify Product Data & Create Python for loops

Storefront API provides quite a lot of data you can scrape, such as product title, price, compare price, URL, SKU, photos, etc. I think title, price, URL, and photos are popular with sellers. So here I try to write a few lines of code, which create variables to represent different product data of title, variants, vendor, price, created date, and use try/except for avoiding processing errors when bumping into some target data that is empty.

python for loops

Also, we need to use for loops methods to loop the scraping until it comes to end. This approach is used in other Python tutorials as well, so I’m not going to deep dive and elaborate. However, the difference is actually under product level. It is because you can see the product data under variants like saying price, variants, etc. So we need to create another loop for variants within the product level.

Python Tutorial: Save the Scraped Data in Google sheet via Google Sheet API

After we test to run the script and find product data can show as we specify in the sublime text, we can add on the google sheet API modules and the codings.

In fact, it’s the same methodology as I talked about Youtuber search results stored in Google Sheet. So if you are interested in the coding process, please check out this article.

Python Tutorial for Digital Marketers 7: Save Web Scraping Data in Google Sheet via API

And finally, all data is in place in the assigned Google sheet after we run the Shopify JSON coding and Google sheet API codings in the sublime text. This piece of raw data can help you to build up your product-market pricing value, compared with competitors. And if you set up import XML based on your dashboard requirements, you can automatically refresh the competitor data when you open the file.

And if you like to scale up processing multi-target websites, you only need to replace the web JSON URL in the script.

google sheet api

Full Python Script of Shopify Competitor Product Data Scraper

If you would like to have the full version of the Python Script of Shopify Competitor Product Data Scraper, please subscribe to our newsletter by adding the message Python Tutorial 8. We would send you the script immediately to your mailbox.

Contact us

 

So easy, right? I hope you enjoy reading Python Tutorial for Digital Marketers 8 – One Script to Scrape Competitor Shopify Web Product Data. If you did, please support us by doing one of the things listed below, because it always helps out our channel.

  • Support my channel through PayPal (paypal.me/Easy2digital)
  • Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
  • Follow and like my page Easy2Digital Facebook page
  • Share the article to your social network with the hashtag #easy2digital
  • Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
  • You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount code on Buyfromlo products and digital software
  • Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)

The next chapter is about how to pull Google trends data using Pytrends as listed below. Enjoy! 🙂

Python Tutorial for Digital Marketers 9: Big Picture Matters, How to Pull Data from Google Trends API via Pytrends

1 thought on “Python Tutorial for Digital Marketers 8: One Script to Scrape Competitor Shopify Web Product Data

  1. This is very interesting, You are a very skilled blogger.
    I’ve joined your feed and look forward to seeking more of your magnificent post.
    Also, I’ve shared your website in my social networks!

Leave a Reply

Your email address will not be published. Required fields are marked *