Chapter 11: Google SERP Bot to Scrape SERP Data Using Google Search and Easy2Digital APIs

I believe we can’t live without search engine channels in life and work anymore. Depending on countries, Google, Yahoo, Naver, Baidu, and so on have been part of the body. Every coin has two sides, as marketers, we might be suffering from overusing search engines to research the market and competitor information. We’re feeling dizzy while watching the screen in front of the laptop for a long day at work.

In this chapter, I would talk about how to scrape all search result information by using Python, Pandas, Google custom search API, and CSE (custom search engine). By the end of this Python Tutorial, you just need to add keywords. Then you can find potential publishers, bloggers, competitors, and popular content, download the images, etc. Also, you are able to store the information with title, landing URL, and so on information into a local CSV file.

Table of Contents: Google SERP Bot to Scrape SERP Data Using Google Search and Easy2Digital APIs

• Google Custom Search API
• Custom Search Engine
• Build the Service Object and Use CSE Function
• For Loop function, List Function, and Arguments
• Limitation of Free Custom Google API
• Easy2Digital API – Google SERP Scraper
• Full Python Script of Google SERP Scraper

Google SERP Bot – Google Search API

The Google Custom Search Engine API is a RESTful API that allows developers to get web or image search results data in JSON or Atom format. With the API, developers can add web search, and site search capabilities to their website, blog, or collection of websites. Also, the API allows marketers to fetch keyword query result information and store it in a CSV file. It facilitates finding partners, such as bloggers, publishers, and competitors.

As mentioned before in the Youtube API article, we can search Google API in the Google API console library, and install it to the project. If you like to learn more about how to create a Google API project, please refer to the other article here:

Chapter 5 – Build a Youtube Bot to Scrape Trending Videos Using Youtube and Easy2Digital APIs

After you install the custom search API, you can find it in the credentials. And please copy the API key to the clipboard, which will be used in the python script.

Google SERP Bot – Custom Search Engine

CSE is short for a custom search engine, or it’s also called Programmable Search Engine. Basically, it lets you include a search engine on your website to help your visitors find the information they’re looking for. Because Programmable Search Engine is based on Google’s core search technology, you can be confident that your users are getting high-quality, relevant results. If you are running AdSense network ads on your website, you should be familiar with CSE, because the search bar ads are also from CSE.

For scraping search results via Python, basically, the data is from CSE API and we would need the ID. The SERP indexation from CSE and Google.com are the same, except for some of the result snippet formats and real-time rankings might be different.

Build the Service Object and Use the CSE function

As we do in the Youtube article, we import build() the function from the google API client. discovery. And also we import pandas to append the data and store it in a local file.

We plug in the custom search API key and create the build function. There are two things we need to pay attention to here. First, we need to follow the parameters of the build function and plug in the custom search value. Secondly, as we would call the data from CSE API, so we need to add CSE() function in this object

For Loop function, List Function, and Arguments

We need to create a loop because each SERP has 10 pieces of information, so we would add a loop code to scrape the other pagination information.

In custom search API JSON syntax and arguments, we need to list functions and at least use q, cx, and start arguments if we like to fetch SERP data. We need to plug in the CSE project’s unique ID in CX. In the end, execute() is necessary to activate the function.

If you try to print(result), you can see the SERP dataset in JSON format. The snippet and naming are almost the same as the youtube JSON dataset. You can select the dataset you like to scrape based on the given JSON dataset.

For pagination setup and dataset filter to call, you can refer to the other article here: 

Limitation of Free Custom Google API

The Google Custom Search API currently limits searches to 100 results per query, and 100 free queries per day. There is no daily query limit if you enable billing on your account (Google will bill $5 per additional 1000 queries, per day).

Easy2Digital API – Shopify Product Scraper

If you find the script might be complicated and also requires you to update scripts and fix bugs on and off, you can leverage Easy2Digital Google SERP scraper API. Here is the token endpoint as follows:

https://www.buyfromlo.com?token=&=&googleKey=&keyword=&totalPage=

By using this API endpoint, you just need to add the Easy2Digital token, Google search key, and the specific keyword and total SEPR pages you aim to scrape. The scraped result is the same as the one shown above.

For more details regarding Marketing APIs, please check out this page.

Easy2Digital Free Token and Full Python Script of Google SERP Scraper

If you would like to have a free Easy2Digital API Token and the full version of the Python Script of Google SERP scraper, please subscribe to our newsletter by adding the message Python Tutorial 11. We would send you the script immediately to your mailbox.

Contact us

So easy, right? I hope you enjoy reading Chapter 11: Google SERP Bot to Scrape SERP Data Using Google Search and Easy2Digital APIs. If you did, please support us by doing one of the things listed below, because it always helps out our channel.

• Support and donate to our channel through PayPal (paypal.me/Easy2digital)
• Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
• Follow and like my page Easy2Digital Facebook page
• Share the article on your social network with the hashtag #easy2digital
• Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
• You sign up for our weekly newsletter to receive Easy2Digital’s latest articles, videos, and discount codes
• Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)