In the previous Python Tutorial, I talked about how to store all fetched data in the Google sheet in your Google drive, which you are not limited to your local computer or laptop to open the data, or your partners can open it to work for projects anytime.
When talking about digital and eCommerce projects, the Shopify project must be one of the most popular topics which many eCommerce sellers like to discuss. If you don’t believe me, please just check out the Shopify stock pricing and ask your surroundings how many sellers are significantly benefiting from Shopify, thanks to easy-to-use and super agile features. The fragment eCommerce era has come, creating an online store is as easy as creating a blog.
So in this Python Tutorial, I’ll walk you through how to create a Python script and scrape any Shopify website product data just by changing a website URL. By the end of this piece, you can learn how to find Shopify storefront API and read Shopify JSON, then write the python codes for specific product information, such as title, price, compared price, etc, as long as Shopify product data includes in the JSON format.
- Storefront API & Shopify Product JSON
- Import Requests, JSON & create the variable with JSON method
- Specify product data and create Python for loops
- Save the scraped data in Google sheet via Google Sheet API
Shopify Product JSON
Basically, there are two types of Shopify product JSON API. One is storefront API: domainname.com/products.json which is open to any visitors, the other store admin API: domainname.myshopify.com/admin/products.json. For competitor monitoring, as you are not the store admin, so you can only leverage the open JSON API.
This is my website example. www.buyfromlo.com/products.json.
As you can see, the product information listed on the website is shown here, however, compared with the admin API, the open API is only showing all visible product data feed on the website and excluding inactive SKUs. Also, it might be quite difficult to see in this JSON format, so you could install a browser plugin to prettify the format. Here I use chrome and install a JSON formatter extension. The storefront API JSON product data would be prettified immediately
Import JSON & Create A Variable with JSON method
Firstly we would need to create modules that we use for scraping the product data. Here we would use requests and JSON modules
Then, we create a variable called whatever like ‘url’, then copy and paste the storefront JSON API path
Then, we create a variable to visit the URL, and the other variable to visualize the data in JSON format
buyfromloweb = requests.get(url)
ProductData = buyfromloweb.json()
Specify Product Data & Create Python for loops
Storefront API provides quite a lot of data you can scrape, such as product title, price, compare price, URL, SKU, photos, etc. I think title, price, URL, and photos are popular with sellers. So here I try to write a few lines of code, which create variables to represent different product data of title, variants, vendor, price, created date, and use try/except for avoiding processing errors when bumping into some target data that is empty.
Also, we need to use for loops methods to loop the scraping until it comes to end. This approach is used in other Python tutorials as well, so I’m not going to deep dive and elaborate. However, the difference is as you can see the product data under variants like saying price, variants, etc are actually under product level. So we need to create another loop for variants within the product level.
Save the Scraped Data in Google sheet via Google Sheet API
After we test to run the script and find product data can show as we specify in the sublime text, we can add on the google sheet API modules and the codings.
In fact, it’s the same methodology as I talked about Youtuber search results stored in Google Sheet. So if you are interested in the coding process, please check out this article.
And finally, all data is in place in the assigned Google sheet after we run the Shopify JSON coding and Google sheet API codings in the sublime text. This piece of raw data can help you to build up your product-market pricing value when comparing with competitors. And if you set up importXML based on your dashboard requirements, you can automatically refresh the competitor data when you open the file.
And if you like to scale up processing multi-target websites, you only need to replace the web JSON URL in the script.
So easy, right? I hope you enjoy reading Python Tutorial for Digital Marketers 8: One Script to Scrape Competitor Shopify Web Product Data. If you did, please support us by doing one of the things listed below, because it always helps out our channel.
- Support my channel through PayPal (paypal.me/Easy2digital)
- Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
- Follow and like my page Easy2Digital Facebook page
- Share the article to your social network with the hashtag #easy2digital
- Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
- You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount code on Buyfromlo products and digital software
- Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)
The next chapter is about how to pull Google trends data using Pytrends as listed below. Enjoy! 🙂