Python Tutorial 40 – Utilize Youtube Bots to Scrape Videos, Profiles and Contacts by Using JSON & Selenium
Automatic chain effect supplier is a nice phrase to describe the youtube data source. Most popular video content in SERP, active and famous Youtube channels, and the contact points such as email, website and social channels are in place as long as the youtubers add on. Basically people are able to grab content ideas, youtuber profile and automate outreach from every time of youtube bot scraping command.
In this Python Tutorial, I’m going to show you how to build a Youtube bot and leverage it to finish a series of commercial tasks. You can apply these actions to complete lots of business tasks, such as demand generation, partnership recruitment etc.
Python Modules and Libraries: Requests, JSON, Selenium, Pandas
- Youtube API Key and URL Endpoint
- Scrape the polarity video content and channel URL using keywords
- Fetch specific channel data in bulk
- Grab the contact points and automate the messaging
- Full Python script of Youtube Bot
Youtube API Key and URL Endpoint
Before starting coding, there are two ingredients we need. The 1st one is a key to connect with the Youtube data center. We need to create an account in Google api console and enable the youtube data v3 app. For more details, please check out this article I released earlier
Python Tutorial 5 – Youtube Scraper to Fetch Trending Videos Using Google API
Once the key is in place, we would use the youtube api endpoint this time to scrape the data. It’s different with the method I used to scrape top videos in SEPR and the video performance data.
There are two URL endpoints this time. One is the search API endpoint, and the other is for channel data fetch. Each endpoint has many parameters for you to filter and extract the specific data you like to grab. I’ll introduce and explain how to use the parameters to complete our task in a moment.
Search Data:
https://www.googleapis.com/youtube/v3/search?key=&maxResult&......
Channel Data:
https://www.googleapis.com/youtube/v3/channels?key=&maxResult&......
If you like to explore the other method I use to scrape the polarity videos in SERP using Googleapis and outh client modules in python, please check out this article
Scrape the polarity video content and channel URL using keywords
There are 6 parameters you will use in the URL endpoint to scrape the popular video content and get the creator channel page URL.
Api parameter– Insert the youtube api key you generate from the Google apis console credentialmaxResult– Each SERP has max.50 results to scrape. Normally you would input 50nextPageToken– The next page value from the Youtube JSONQuery– the keywordOrder– you can filter the SERP result by viewCount or video count. From my point of view, I will use the viewCount because it shows you the polarity and it’s easier to consolidate popular channel profilesregionCode– It allows you to generate video content from a specific country, such as Singapore, Japan, etc
When the parameters’ value are ready, below is a sample using requests
url2 = requests.get('https://www.googleapis.com/youtube/v3/search?key=' + api + '&maxResults=' + maxResult + '&part=snippet&q=' + query + '&type=video' + '&order=' + order + '®ionCode=' + regionCode + '&pageToken=' + resutl['nextPageToken'])
You might find that it just comes up with response 200 if you just print out the url2 variable above. It’s because you need to format the data from youtube data in a json format.
So we need to create a variable to transform the data into JSON
resutl2 = url2.json()
Grab the nextPageToken Value and Leverage Nested Loop for more than 50 SERPs
It might miss out some valuable content and channels if just referring to top 50 SERPs. So we need to leverage the nextPageToken parameter mentioned above. For the value of nextPageToken, Youtube API would include a dynamic value in each response. So first thing first, we need to grab the value from each SERP json data like the script below
I would recommend using 2 or 3 SERPs from each keyword, because the quality of channel profiles are getting lower after the 4th SERP.
Once the nextPageToken value is in place, we could create a nested loop, which would loop the pageToken first and then loop scraping the search snippet value, such as channel ID
Fetch specific channel data in bulk
Scraping the top and most polarity content is the first step where you can grab the great content as inspiration. However, if you like to network more top tier youtubers and content creators for commercial purposes, specific channel data is the next step.
Youtube channel URL structure is like this below.
https://www.youtube.com/channel/ + Channel ID (UCF2oW5-MO8dB6ul9WH9xi0A
)
In the SERP scraping section, you can grab a list of channel IDs from each top ranking video. So you can set up a timer to wait a while and automatically visit this channel one after one by combining the youtube channel folder and the channel ID.
For the channel api URL endpoint, there are also different parameters you can check and leverage based on your needs.
For finding the qualified profiles, usually I would use these parameters
Statistics– we can grab the follower amountbrandingSettings– Fetch the channel about page description, sometimes the owner leaves the email and phone number contact there.topicDetails– Here tells what is the main content topics from this channel page.
Once the choices are ready, below is the sample of the URL with optimal parameters
url1234 = requests.get('https://www.googleapis.com/youtube/v3/channels?key=' + api + '&part=statistics' + '&id=' + chID).json()
Grab the contact points and automate the messaging
Now you have a list of qualified channel profiles you find opportunities to collaborate. So the next step is to scrape the contact points and automate the messagings.
Except for the description part, generally Youtubers add their contact in the channel about page. The URL structure of about page adds the slash tag “about” after the channel ID page.
https://www.youtube.com/channel/UCF2oW5-MO8dB6ul9WH9xi0A/about
As you can see, there is one section called for business inquiries. Youtube sets up a very strict human verification for constraining the bot scraping action. Normally I would use Python to detect if the youtuber leaves an email in about page by using this code. Then I can manually get them once I find this channel is worthy of investing time.
The turnaround solution is the Youtuber’s social channels and website. We can integrate the scraped social channel address and website URL with the specific social bots I shared previously. So we can contact them in social messenger, like instagram message, TikTok, and through email, such as Gmass.
For more details, please check out this list of articles to build up a RPA
Social Bots
Email Automation
Full Python script of Youtube Bot
If you would like to have the full version of Youtube Bot for scraping content, profiles and contacts using JSON & Selenium, please subscribe to our newsletter by adding the message Python Tutorial 40. We would send you the script immediately to your mailbox.
I hope you enjoy reading Python Tutorial 40 – Youtube Bot for scraping content, profiles and contacts using JSON & Selenium. If you did, please support us by doing one of the things listed below, because it always helps out our channel.
- Support my channel through PayPal (paypal.me/Easy2digital)
- Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
- Follow and like my page Easy2Digital Facebook page
- Share the article to your social network with the hashtag #easy2digital
- Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
- You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount code on Buyfromlo products and digital software
- Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)
