Python Tutorial for Digital Marketers 11: Scrape SERP via Search API and Custom Search Engine

I believe we can’t live without search engine channels in life and work. Depending on countries, Google, Yahoo, Naver, Baidu, and so on have been part of the body. Every coin has two sides because marketers might be suffering from overusing search engines to research the market and competitor information. We’re feeling dizzy while watching the screen in front of the laptop for a long day at work.

In this article, I would introduce you to a way to scrape all search result information by using Python, Pandas, Google custom search API, and CSE (custom search engine). By the end of this article, you just need to add keywords, you can find potential publishers, bloggers, competitors, and popular content, download the images, etc, and store the information with title, landing URL, and so on information into a local CSV.file.

python tutorial

I believe we can’t live without search engine channels in life and work anymore. Depending on countries, Google, Yahoo, Naver, Baidu, and so on have been part of the body. Every coin has two sides, as marketers, we might be suffering from overusing search engines to research the market and competitor information. We’re feeling dizzy while watching the screen in front of the laptop for a long day at work.

In this Python Tutorial, I would talk about how to scrape all search result information by using Python, Pandas, Google custom search API, and CSE (custom search engine). By the end of this Python Tutorial, you just need to add keywords. Then you can find potential publishers, bloggers, competitors, and popular content, download the images, etc. Also you are able to store the information with title, landing URL, and so on information into a local CSV.file.

Python Tutorial – Google Custom Search API

The Google Custom Search Engine API is a RESTful API that allows developers to get web or image search results data in JSON or Atom format. With the API, developers can add web search, site search capabilities to their website, blog, or collection of websites. Also, the API allows marketers to fetch keyword query result information and store it into a CSV file. It facilitates finding partners, such as bloggers, publishers, competitors.

As mentioned before in the Youtube API article, we can search Google API in the Google API console library, and install it to the project. If you like to learn more about how to create a Google API project, please refer to the other article here:

Python Tutorial for Digital Marketers 5: Create Youtube API & Scrape Youtube Videos

python tutorial

After you install the custom search API, you can find it in the credentials. And please copy the API key to the clipboard, which will be used in the python script.

python tutorial

Python Tutorial – Custom Search Engine

CSE is short for a custom search engine, or it’s also called Programmable Search Engine. Basically, it lets you include a search engine on your website to help your visitors find the information they’re looking for. Because Programmable Search Engine is based on Google’s core search technology, you can be confident that your users are getting high-quality, relevant results. If you are running AdSense network ads on your website, you should be familiar with CSE, because the search bar ads are also from CSE.

python tutorial

For scraping search results via Python, basically, the data is from CSE API and we would need the ID. The SERP indexation from CSE and Google.com are the same, except for some of the result snippet formats and real-time rankings might be different.

python tutorial

Build the Service Object and Use CSE function

As we do in the Youtube article, we import build() function from the google API client. discovery. And also we import pandas to append the data and store in a local file.

Before that, we need an empty variable and we create a cdbsearch here

from googleapiclient.discovery import build

import pandas as pd

cbdsearch = []

Then, we plug in the custom search API key and create the build function. There are two things we need to pay attention to here. First, we need to follow the parameters of the build function and plug in the custom search value. Secondly, as we would call the data from CSE API, so we need to add CSE() function in this object

python tutorial

api_key = 'abcxxxabc'

googlesearchT = build('customsearch', 'v1', developerKey=api_key).cse()

For Loop function, List Function, and Arguments

We need to create a loop, because each SERP has 10 pieces of information, so we would add a loop code to scrape the other pagination information.

In custom search API JSON syntax and arguments, we need to list functions and at least to use q, cx, start arguments if you like to fetch SERP data. In CX, we need to plug in the CSE project unique ID. At the end, execute() is necessary to activate the function.

for i in range(0,5):

result = googlesearchT.list(q='cbd capsule hong kong',cx='abxbsdferesdf',start=str(i)+'1').execute()

If you try to print(result), you can see the SERP dataset in JSON format. The snippet and naming are almost the same with the youtube JSON dataset. You can select the dataset you like to scrape based on the given JSON dataset.

for item in result['items']:

title = (item['title'])

link = (item['link'])

element_info = {

'title': title,

'link': link

}

cbdsearch.append(element_info)

print(len(cbdsearch))

df = pd.DataFrame(cbdsearch)

print(df)

df.to_csv('cbdcphk.csv')

For pagination set up and dataset filter to call, you can refer to the other article here

Limitation of Free Custom Google API

The Google Custom Search API currently limits searches to 100 results per query, and 100 free queries per day. There is no daily query limit if you enable billing on your account (Google will bill $5 per additional 1000 queries, per day).

Full Python Script of Google SERP Scraper

If you would like to have the full version of the Python Script of Google SERP scraper, please subscribe to our newsletter by adding the message Python Tutorial 11. We would send you the script immediately to your mailbox.

Contact us

So easy, right? I hope you enjoy reading Python Tutorial for Digital Marketers 11: Scrape SERP via Search API and Custom Search Engine. If you did, please support us by doing one of the things listed below, because it always helps out our channel.

  • Support my channel through PayPal (paypal.me/Easy2digital)
  • Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
  • Follow and like my page Easy2Digital Facebook page
  • Share the article to your social network with the hashtag #easy2digital
  • Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
  • You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount code on Buyfromlo products and digital software
  • Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)

Leave a Reply

Your email address will not be published. Required fields are marked *