Chapter 21: Amazon Best Selling Product Scraper to Find Niche Products, Monitor Competitors, and Identify Potential Clients
Amazon, the eCommerce giant globally, is the important lighthouse of product trend and direction to millions of sellers and hundreds of brands worldwide. Most of them are keeping eyes on the popularity of product selling and momentum of trends on Amazon. No brainer to ask a question where to find the trend? Amazon best selling products and brands is the place you should often drop by. It makes more sense if you are asking how to implement more efficiently in an automatic format. If you do, this piece can be helpful to you.
Amazon best selling products are being updated hourly every day. The updates are based on sales performance and customer rating of the product. However, you might carefully look into who is selling the products. Basically, the best-selling merchants are almost hourly although the position might vary. Just some new sellers might stand out by selling new popular niche SKUs.
In a way, it’s not realistic and unnecessary to manually monitor every hour on all categories and SKUs. What’s more, the interface on the Amazon best-selling page is not a friendly way for you to understand the performance in a data format. So in this article, I would walk you through how to create an Amazon best-selling product scraper using Python. By the end of this piece, you can learn how to code and create the scraper for fetching your competitor’s up-to-date product performance and pricing, finding potential customers, and identifying potential products. And then you can set up Crontab to automate the process and refresh your dashboard.
Table of Contents: Amazon Best Selling Product Scraper
- Scrape Amazon Besting Selling Page URL by Category
- Fetch Amazon Best Selling Product Data by Category
- Full Script of Amazon Best Selling Product Scraper
Amazon Besting Selling Product Ranking Page by Category
Amazon has hundreds of categories and subcategories in Best selling product pages. Your brand and business might not be related to each category and product, but you must like to realise which page to check the information is the most proper. The information is valuable for you to explore new niche products, monitor competitor’s SKUs and spot potential clients.
Amazon ranks the best selling products by departments. Each department has categories, and under each category, it also has many subcategories, and sub subcategories as well. Take the amazon devices & accessories for the example as listed below. It looks like an onion, which can be splitted layer by layer.
https://www.amazon.com/Best-Sellers/zgbs/amazon-devices
https://www.amazon.com/Best-Sellers-Amazon-Device-Accessories/zgbs/amazon-devices/370783011
So first thing first, we can fetch all department’s URL by using the URL below
values_list = https://www.amazon.com/Best-Sellers/zgbs
All category page left hand side menu’s html coding is with the same naming. I assume you have already fetched all the department’s URL by using the URL above. Below is the full list of department’s URLs
If you like to scrape categories to understand each department, you need to create a loop to scrape using selenium, gspread and beautifulsoup.
First thing first, you need to use the simulation driver to open Amazon best selling page, as Amazon disallows BeautifulSoup direct access to page html.
amazonSERP = []
Then, you can create a table in Google Sheets, and paste all department URLs in a column. You can use gspread to read the URLs.
Last but not least, the category URLs in Amazon best selling product pages are sitting under id zg_browseRoot. It’s formatted by <li></li>. Thus, below are the looping scraper of codings
Fetch Amazon Best Selling Product Data by Category
Once you have a full list of category URLs on hand, you can use the specific category URL to set up the scraper.
First thing first, each category best selling product has two pages, one is top 50 and the other is 51 – 100. You can use the parameter ?pg= after the category URL.
What’s more, the block of each product is class = aok-inline-block zg-item. So basically we can use selenium and beautifulsoup to find all blocks of the best selling product dataset
Then, we need another loop to fetch the specific data from each product, such as title, product url, review, ratings, pricing, etc.
Last but not least, we can use pandas to append the data connected with the variables created above in the scraping codes. Then automatically it can update the data into the Google Sheets.
## append the data and upload to Google Sheets ##
element_info = {
"Market": Market,
"Channel": Channel,
"Tier of Cate": Tier,
"Name of Cate": category,
"Title": title,
"URL": url,
"Review": review,
"Stars": Stars,
# "Min-Price": min_price,
"Max-Price": price
}
amazonSERP.append(element_info)
df = pd.DataFrame(amazonSERP)
Full Script of Amazon Best Selling Product Scraper
If you would like to have the full version of the Python Script of Amazon Product Price Tracker, please subscribe to our newsletter by adding the message “Chapter 21”. We would send you the script immediately to your mailbox.
I hope you enjoy reading Chapter 21: Amazon Best Selling Product Scraper – Awesome Approaches to Find Niche Products, Monitor Competitors, and Identify Potential Clients. If you did, please support us by doing one of the things listed below, because it always helps out our channel.
- Support and donate to our channel through PayPal (paypal.me/Easy2digital)
- Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
- Follow and like my page Easy2Digital Facebook page
- Share the article on your social network with the hashtag #easy2digital
- Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
- You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount codes
- Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)
