Python Tutorial 41: Scrape Google Keyword Volume, CPC, Keyword Ideas, Competition Using SERPMaster API

Keyword volume and CPC are very familiar terms as long as you are running digital marketing campaigns and doing performance prediction. I believe it’s a critical signal to deploy content, budgeting between organic and paid media ads, and also evaluate a business investment. It’s because it represents the low funnel opportunities in a way.

However, it’s a pity in terms of keyword volume and CPC research that there are 3 challenges you might have experienced – Free lunch is not free, people manually input and download, and 3rd party software is experienced. So this article is a method to resolve this issue.

serpmaster api

Keyword volume and CPC are very familiar terms as long as you are running digital marketing campaigns and doing performance prediction. I believe it’s a critical signal to deploy content, budgeting between organic and paid media ads, and also evaluate a business investment. It’s because it represents the low funnel opportunities in a way.

However, it’s a pity in terms of keyword volume and CPC research that there are 3 challenges you might have experienced – Free lunch is not free, people manually input and download, and 3rd party software is experienced. So this article is a method to resolve this issue.

In this Python Tutorial, I will take the SERPMaster API for example as an API with the free trial, to walk you through how to scrape Google keyword data. By the end of this python tutorial, you can learn why an API is super useful, how to code and make it using the SERPMaster API.

Alternative Free Options: Google ads keyword volume API and test account token

Good value for money paid option: keyword anywhere

Normal paid options: SEMRush, DataforSEO, Georanking

Python Libraries: Requests, JSON, pandas

Why you need the keyword volume API

There are so many platforms out there you can leverage to generate the Google keyword data. However it’s a question how we can have the best optimal solution to manage the data scalably in bulk and save dollars meanwhile. I try to consolidate three types of general approaches and you will find why a proper keyword volume API is the best optimal solution

Free Lunch Is Not Free

Google keyword planner is definitely the best free version when you are researching the opportunities and plan your organic and paid media strategy. In the data set from the keyword planner in Google ads, it includes avg monthly searches, CPC, competition, keyword ideas and performance forecast.

serpmaster api

However, Google obviously guides you to start a campaign to get the volume number because it only provides you a rough size from each keyword. In a way, 1M – 10M can’t tell you anything but just the size of the pool. So it’s not good for prediction when you plug these data into P&L, CLV or NPV calculators.

Fine dinner might be not your meal everyday

I take SEMRush for example. It charges US$449 monthly to activate the API access feature. To be fair, if your research volume is huge like a performance marketing agency, it might be good value for money. But for a brand or SMEs or individuals, it’s too expensive.

serpmaster api

It’s not bad to try the fine dinner taste one or two times, but as a businessman, we need to consider the software investment ROI at the end.

Download and Upload yourself is not the best optimal way to manage

Lots of SaaS platforms out there give you a feature to input keywords and output the keyword data such as volume, CPC, etc from Google. The sexiest part, compared to Google keyword planner, does normally has a more specific number rather than just a size pool from a keyword.

serpmaster api

However, you download and upload again and again manually, which is not a sustainable way to make it scalable and automatic. Yes, you can build up a vlookup formula in excel upfront, however, it stills involve lots of manual work and might create inaccuracy

What is SERPMaster and how to sign up a free account

SERPMaster is a scraper SaaS platform. More precisely, it provides organic, paid ads and product data from different channels, such as Google, Amazon. If you read my articles before, you will find out that the data from SERPMaster can integrate with many business analytics and marketing operation modules, such as P&L calculator, SEO keyword mapping, etc.

As elaborated the three challenges why API above, the main purpose of good keyword volume API can resolve the issue from many tools out there. And SERPMaster basically can help out lots of brands, SMEs and individual business owners like influencers. It provides many API endpoint module options.

serpmaster api

SERPMaster provides a free trial without requiring any credit card information. In terms of the account sign up, it’s very easy and straightforward using your email address. Then the SERPMaster team staff would reach out to you and send you the credential and token.

Google Keyword Data: Volume, CPC, Competition, Keyword Ideas

In SERPMaster API documentation, it provides the Python code script template for your reference. Basically it includes the headers, job_params and response.

In Google keyword volume modules, there is a guide to show you what mandatory parameters and optional parameters are given by SERPMaster. I usually use these 6 parameters as listed below

Mandatory:

  • q: input your keyword
  • scraper: select the module
  • access_token: input the API key
  • auth: input your username and password given by SERPMaster

Optional:

  • ksv_currency: select the CPC currency
  • ksv_ideas_limit: control the quantity output of keyword ideas

Token = “yourSERPMasterAPIKey”

headers = {'Content-Type': 'application/json'}

job_params = {

'q': keywords,

'access_token': token,

'scraper':'google_ksv',

'ksv_currency': 'USD',

'ksv_ideas_limit': int(10)

}

response = requests.post(

'https://rt.serpmaster.com',

headers=headers,

json=job_params,

auth=('youruserName', 'yourAccountPassword')

)

result = response.json()

Pandas Concat Method

In SERPMaster, seed keyword represents the keyword you input and ideas represent the keyword variants. They sit in two lists from the JSON data set. So you can create a separate loop for each list to output the keyword data.

At the end, you can leverage the pandas concat method to combine two data sets into one table. Then it can be either integrated with Google sheet or your dashboard and all data can be uploaded there automatically in an organized format.

Google Keyword Suggestion from Google SERP

Apart from the keyword ideas, keyword suggestion is another API module we can use in SERPMaster. The data is mainly from the Google search suggestion list when you input a keyword before searching.

Here is the Python code sample

From the JSON data, the keyword suggestions sit in the sublist from a list. So you can add replace before a loop to output the suggested keywords

content1 = content.replace('[','').replace(']','').replace('"','').split(',’)

Full Python Script of Google Keyword Volume API using SERPMaster

If you are interested in the full Python script of Scrape Google Keyword Volume, CPC, Keyword Ideas, Competition Using SERPMaster API, please subscribe to our newsletter by adding the message “Python Tutorial 41”. We would send you the script immediately to your mailbox.

I hope you enjoy reading Python Tutorial 41: Scrape Google Keyword Volume, CPC, Keyword Ideas, Competition Using SERPMaster API. If you did, please support us by doing one of the things listed below, because it always helps out our channel.

  • Support my channel through PayPal (paypal.me/Easy2digital)
  • Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
  • Follow and like my page Easy2Digital Facebook page
  • Share the article to your social network with the hashtag #easy2digital
  • Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
  • You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount code on Buyfromlo products and digital software
  • Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)

Leave a Reply

Your email address will not be published.