Keyword extraction using website URLs facilitates you to quickly learn about a new brand from scratch rather than reading through all information. There are many tools out there, nevertheless, it’s expensive to subscribe or the free tool is not user-friendly, notably not providing APIs for you to integrate with your business dashboard.

In this article, I would try using Easy2Digital APIs – brand info scraper and Google SERP scraper, to build a keyword extractor.

Ingredients on building a keyword extractor using Easy2Digital API

Table of Contents

• Collecting Brand relevant URLs using Easy2Digital APIs
• Scrape the content from the URLs using BeautifulSoup
• De-duplicate the extracted keywords
• Full Python Script of Keyword Extractor

Collecting Brand relevant URLs using Easy2Digital APIs

The first question is where to extract keywords that are related to the brand you are investigating. The answer is the brand’s official site and surrounding portal sites.

Here is the code sample of the API usage as follows:

Scrape the content from the URLs using BeautifulSoup

For non-react-based website page scraping, Beautifulsoup is the best option without a doubt. For better scraping as many raw texts as possible at this stage, I suggest scraping h1, h2, h3, and p content separately and using find_all() to make it. Here are the code samples as follows:

Extract the text’s keywords using stop-words

In the keyword extractor top argument, we can add a number to extract only the most frequent keywords, such as top 10, and top 5. Additionally, for avoiding getting the English stop words, we can add the list in the second argument – stopwords

There are some stopwords modules out there, here I recommend you to use scikit-learn because it has the longest stopword list compared to others based on my personal experiences. Please be sure to install the scikit-learn python package before importing this module into the script.

from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS

In the extract_keywords, we need to feed the scraped text from the websites we just did. Here I created a variable named full_text, which included the scraped text.

keywords = kw_extractor.extract_keywords(full_text)

De-duplicate or Word Cloudify the extracted keywords

The script can generate the extracted keywords like this if it’s working properly. It’s a JSON format, and we can create a loop to extract the keyword without the score value.

On the whole, we have two ways to use these keywords for intelligent insight. One is to generate a word cloud by using thousands of keywords for showing the keyword popularity by size. The other is to de-duplicate the keywords and rest only a list of unique keywords. Here I would show to de-duplicate the script.

final_keyword = list(dict.fromkeys(keywordResult))

Full Python Script of Keyword Extractor

• If you are interested in getting the full version of the script and vibe coding prompt mentioned in this article, please subscribe to our Youtube Channel and comment “keyword extractor” in the following video (Keyword Extractor Script and Vibe Coding Prompt)
• Support our channel through PayPal (paypal.me/Easy2digital)
• Follow our Facebook page
• Sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount code on Buyfromlo products and digital software