Site icon EASY2DIGITAL

Chapter 5 – Build a Youtube Bot to Scrape Trending Videos Using Youtube and Easy2Digital APIs

Louis Lu

Previously we took the Ring.com website as an example in Python Tutorial. We walked you through how to specify Ring product price data to scrape and save the bulk of data in the local drive. Basically, the script code can scrape any general websites that are built and developed by CMS, such as WordPress, Shopify, etc. If you want to have a copy of the python script file, please contact us.

Being said that, we can’t use the script to scrape all types of websites. It’s because the data feed on some websites can be only accessible via API, such as Youtube, Facebook, Amazon, etc. Frankly, if you just want to scrape these sites’ data by URL, I suggest you leverage Google Sheet importxml instead of Python because it’s much easier (Time is money). On the other hand, automation and a bulk amount of scraping data work is your regular work, for example, if you are a social media marketer, you need to recruit influencers or KOLs, and the following series of articles can help relieve your stress and repeat manual work.

We would start with Youtube video scraping first, and by the end of this Python Tutorial, you can learn how to install the Youtube API key, what give methods you can use by Youtube API, and how to scrape youtube videos and video analytic data at scale.

Table of Contents: Build a Youtube Bot to Scrape Trending Videos Using Youtube and Easy2Digital APIs

Create a Youtube API Key and Install it on your Mac

The first thing first in Python Tutorial, you need to create a Google API console account and a new project. It’s very simple and free to sign up, what you need to do is just a few steps:

1) Search the Google API console, and sign up for a new account.

2) Then, on the right corner, you can see the “New Project” button. If you have already created it will appear in a separate window. The below picture will help you out.

3） You can click ‘enable APIs and services, and then search youtube. You can have some Youtube API options available and you can select Youtube Data API v3 if you are going to scrape video data.

4) In credential, you need to create a new API key, which will be used in a moment on your Python script. The API key is a unique string and number to identify the gateway you use to get the data feed, so please keep it confidential and don’t disclose it. Otherwise, you need to remove it and recreate it again.

5) We go to the installation and install the google APIs module on our computer (Mac or Windows).

As usual, we open the Mac terminal and pass in this command

Pip3 install google-api-python-client

After installation is done, you can pass in a command: pips list, and as you can see, all modules shown here include beautifulsoup, etc.

How to use Youtube API documentation to start coding Python Script

First of all, we need to create a Python3 script file on sublime text, import the google API module, and create a variable that is passed in the Youtube API key. And for any Google APIs-related documentation, you can either go to Google APIs – Python, or go to the Google APIs page on GitHub. 

from googleapiclient.discovery import build

api_key = 'xxxxxxxxxx…..’ 

(Note: Please keep your Youtube API Key number confidential and safe, otherwise you might be hacked easily and have to recreate a new one)

Then, we need to create a service object, which takes an API name and API version as arguments. Regarding Python object creation, I will not go into details in this article, but I’ll release another one to walk you through.

Build Function

For this, we can refer to the build (), and below is the argument parameter in this function.

build(serviceName, version, http=None, discoveryServiceUrl=DISCOVERY_URI, developerKey=None, model=None, requestBuilder=HttpRequest, credentials=None, cache_discovery=True, cache=None,client_options=None, adc_cert_path=None, adc_key_path=None, num_retries=1)

So we create a variable named in youtube and pass in the arguments to create an object

 youtube = build(‘youtube’, ‘v3’, developerKey=api_key)

Then, as we need to scrape the videos from a keyword query of search results, so firstly we need to check if API allows us to grab, them and what Youtube API method allows us to use. We can just search youtube API and go to the Youtube API reference page to check.

As you can see, Youtube provides a search method and list method for users to code and grab search result data. And within the method, there are quite a lot of parameters for you to adopt. In one of those, the part is a must in the line of coding. So in terms of video search result scraping,  we can try this line of code, and it aims to grab the video information from the keyword query “ring spotlight camera”. We can try different keywords as you like.

request = youtube.search().list(

        part='snippet',

        q='ring spotlight camera',

        maxResults='50',

)

response = request.execute()

print(response)

It’s working and max.50 videos by using maxresult can be grabbed and shown on Sublime Text.

Scrape Videos in Youtube Search Query Result

Now the data from the keyword query is accessible, we assume to grab 3 types of data and let us see how to find the data location.

The first thing first in Python Tutorial is we can see all the data of each video we aim to grab within the item [….]. And channel name & video title data are under the snippet path, and the video id is under the id path. So based on this, we can write lines of code.

'items': [{'kind': 'youtube#searchResult', 'etag': 'GGoUw-TsOsWwwPIBcKVUozGGrW8', 'id': {'kind': 'youtube#video', 'videoId': 'd7iJwF-5xGU'}, 'snippet': {'publishedAt': '2017-08-03T19:29:56Z', 'channelId': 'UCSDG3M0e2mGX9_qtHEtzj2Q', 'title': 'How to Install & Setup Ring Spotlight Cam | Easy to Connect', 'description': 'Learn how to easily setup and install your Ring Spotlight Cam & Battery. Spotlight Cam Battery comes with a 1080HD camera with two-way talk, two LED ...', 'thumbnails': {'default': {'url': 'https://i.ytimg.com/vi/d7iJwF-5xGU/default.jpg', 'width': 120, 'height': 90}, 'medium': {'url': 'https://i.ytimg.com/vi/d7iJwF-5xGU/mqdefault.jpg', 'width': 320, 'height': 180}, 'high': {'url': 'https://i.ytimg.com/vi/d7iJwF-5xGU/hqdefault.jpg', 'width': 480, 'height': 360}}, 'channelTitle': 'Ring', 'liveBroadcastContent': 'none', 'publishTime': '2017-08-03T19:29:56Z'}},{'kind': 'youtube#searchResult', 'etag': '3P6Ws8ggqau3I0Dew2nDsRcsr0Q', 'id': {'kind': 'youtube#video', 'videoId': 'U-06WEwtaSk'}, 'snippet': {'publishedAt': '2020-01-21T14:52:47Z', 'channelId': 'UCXn_loz0TlUKarhS6sUoKVw', 'title': 'Is the Ring Spotlight Camera Worth Buying?', 'description': 'In this video we review the Ring Spotlight Camera. We like the Spotlight and the Floodlight because it will shine its lights and start recording video whenever ...', 'thumbnails': {'default': {'url': 'https://i.ytimg.com/vi/U-06WEwtaSk/default.jpg', 'width': 120, 'height': 90}, 'medium': {'url': 'https://i.ytimg.com/vi/U-06WEwtaSk/mqdefault.jpg', 'width': 320, 'height': 180}, 'high': {'url': 'https://i.ytimg.com/vi/U-06WEwtaSk/hqdefault.jpg', 'width': 480, 'height': 360}}, 'channelTitle': 'HomeAutomationX', 'liveBroadcastContent': 'none', 'publishTime': '2020-01-21T14:52:47Z'}}, 

channel_title = (item['snippet']['channelTitle'])

print(channel_title)

video_title = (item['snippet']['title'])

print(video_title)

try:

    vid_ids = (item['id']['videoId'])

except Exception as e:

    vid_ids = None

print(vid_ids)

Some video IDs are missing from the data, the reason is the result also shows the channel page information or related videos, so for avoiding scraping errors in the process, we need to add try/except for the scraping video id above.

Looping to Scrape

For looping the scrape process, we need to add a line of code as a parent level before it

for item in response['items']:

Then all videos can be scraped as you can see from the result.

Now we can import the data into a CSV file

(Note: If you are interested in learning how to parse and pass data into a CSV, please check out this article)

Chapter 3: Utilise CSV Module to Write, Parse, Read CSV Files to Manage Scraped Data

Each youtube video has a unique ID, and the video URL structure basically is like this

https://www.youtube.com/watch?v= + Video ID

Either you can leverage the format method in the code, or you can leverage CONCATENATE() in the excel file.

Leverage Easy2Digital APIs

If you find the script might be complicated and also requires you to update scripts and fix bugs on and off, you can leverage Easy2Digital Youtube Bot API. Here is the token endpoint as follows:

https://www.buyfromlo.com?token=&youtubeKey=&keyword=

By using this API endpoint, you just need to add the Youtube key, Easy2Digital token, and the keywords related to the video content you aim to scrape. The scraped result is the same as the one shown above.

For more details regarding Marketing APIs, please check out this page.

Easy2Digital Marketing APIs Documentation

Easy2Digital API Free Token and Full Python Script of the Youtube Bot

If you would like to have a free Easy2Digital API token and the full version of the Python Script of Youtube Bot Scraper, please subscribe to our newsletter by adding the message Chapter 5. We would send you asap to your mailbox.

Contact us

So easy, right? I hope you enjoy reading Chapter 5: Build a Youtube Bot to Scrape Trending Videos Using Youtube and Easy2Digital APIs. If you did, please support us by doing one of the things listed below, because it always helps out our channel.

If you are interested in more Python Scraping via Youtube API, you can continue to read more in Chapter 6

Chapter 6: Leverage Easy2Digital APIs and Youtube Key to Scrape View, Comment, and Like Data of More than 50 Videos From Top Ranking Ones

FAQ:

Q1: What is the Youtube Video API?

A: The Youtube Video API is a service provided by Youtube that allows developers to access and integrate Youtube video functionality into their own applications or websites.

Q2: What can I do with the Youtube Video API?

A: With the Youtube Video API, you can perform various tasks such as searching for videos, retrieving video information, uploading videos, managing playlists, and more.

Q3: How can I get access to the Youtube Video API?

A: To access the Youtube Video API, you need to create a project in the Google Developers Console, enable the Youtube Data API, obtain an API key, and authenticate your requests using the key.

Q4: Is there a cost for using the Youtube Video API?

A: The Youtube Video API is free to use, but there are quota limits and usage restrictions. If you exceed the free quota limits, you may need to upgrade to a paid plan to continue using the API.

Q5: Can I monetize my application or website using the Youtube Video API?

A: Yes, you can monetize your application or website by integrating ads using the Youtube Video API. However, you need to comply with Youtube’s monetization policies and guidelines.

Q6: Are there any restrictions on the usage of the Youtube Video API?

A: Yes, there are certain restrictions on the usage of the Youtube Video API. You should review and comply with the Youtube API Services Terms of Service and Developer Policies to ensure your usage is in compliance.

Q7: Can I retrieve video statistics and analytics using the Youtube Video API?

A: Yes, the Youtube Video API provides access to video statistics and analytics data, allowing you to retrieve metrics such as views, likes, dislikes, comments, and more.

Q8: Can I upload videos to Youtube using the Youtube Video API?

A: Yes, you can upload videos to Youtube programmatically using the Youtube Video API. This allows you to automate the video uploading process and manage your channel’s content programmatically.

Q9: Is the Youtube Video API suitable for small businesses or startups?

A: Yes, the Youtube Video API can be beneficial for small businesses and startups as it enables them to leverage the Youtube platform and its video capabilities to enhance their applications or websites.

Q10: Where can I find documentation and resources for the Youtube Video API?

A: You can find detailed documentation, tutorials, code samples, and other resources for the Youtube Video API on the official Youtube Developers website.
Exit mobile version