Previously we took Ring.com website as an example in Python Tutorial and walked you through how to specify Ring product price data to scrape and save the bulk of data in local drive. Basically, the script code can scrape any general websites that are built and developed by CMS, such as WordPress, Shopify, etc. If you want to have a copy of the python script file, please contact us.

Being said that, not all websites can be indexable and crawlable, because the data feed on some websites can be only accessible via API, such as Youtube, Facebook, Amazon, etc. Frankly, if you just want to scrape these sites’ data by URL, I suggest you leverage Google Sheet importxml instead of Python, because it’s much easier (Time is money). On the other hand, automation and bulk amount of scraping data work is your regular work, for example, if you are a social media marketer, you need to recruit influencers or KOLs, the following a series of articles can help relieve your stress and repeating manual work.

We would start with Youtube video scraping first, and by the end of this Python Tutorial, you can learn how to install the Youtube API key, and what give methods you can use by Youtube API, how to scrape youtube videos, and video analytic data in bulk.

Create a Youtube API Key and Install Google API on your Mac

The first thing first in Python Tutorial, you need to create a Google API console account and a new project. It’s very simple and free to sign up, what you need to do is just a few steps:

1) Search Google API console, and sign up a new account.

Python Tutorial for Digital Marketers

2) Then, on the right corner, you can see the “New Project” button. If you have already created it will appear on a separate window. The below picture will help you out.

Python Tutorial for Digital Marketers

3) You can click ‘enable apis and services’, and then search youtube. You can have some Youtube API options available and you can select Youtube Data API v3 if you are going to scrape video data.

Python Tutorial for Digital Marketers

4) In credential, you need to create a new API key, which will be used in a moment on your Python script. The API key is a unique string and number to identify the gateway you use to get the data feed, so please keep it confidential and don’t disclose. Otherwise, you need to remove it and recreate it again.

Python Tutorial for Digital Marketers

5) We go to the installation and install the google apis module on our computer (Mac or Window).

As usual, we open the Mac terminal and pass in this command

Pip3 install google-api-python-client

After installation is done, you can pass in a command: pips list, and as you can see, all modules show here that include beautifulsoup, etc.

How to use Youtube API documentation to start coding Python Script

First of all, we need to create a Python3 script file on sublime text, and import the google api module and create a variable that is passed in the Youtube API key. And for any Google apis related documentation, you can either go to Google APIs – Python, or go to the Google APIs page in github. 

from googleapiclient.discovery import build

api_key = ‘xxxxxxxxxx…..’ 

(Note: Please keep your Youtube API Key number is confidential and safe, otherwise you might be hacked easily and have to recreate a new one’)

Then, we need to create a service object, which takes an API name and API version as arguments. Regarding Python object creation, I will not go into details in this article, but I’ll release another one to walk you through.

For this, we can refer to the build (), and below is the argument parameter in this function.

build(serviceName, version, http=None, discoveryServiceUrl=DISCOVERY_URI, developerKey=None, model=None, requestBuilder=HttpRequest, credentials=None, cache_discovery=True, cache=None,client_options=None, adc_cert_path=None, adc_key_path=None, num_retries=1)

So we create a variable named in youtube and pass in the arguments to create an object

 youtube = build(‘youtube’, ‘v3’, developerKey=api_key)

  • serviceName: We can pass in ‘youtube’ because the API is youtube
  • version: We can use the version installed on our Google API console, which should be v3
  • developer key: As we have created a api_key variable, so we can directly and simply pass in the variable ‘api_key’

Then, as we need to scrape the videos from a keyword query of search results, so firstly we need to check if API allows us to grab, and what Youtube API method allows us to use. We can just search youtube API and go to the Youtube API reference page to check.

As you can see, Youtube provides a search method and list method for users to code and grab search result data. And within the method, there are quite lots of parameters for you to adopt. In one of those, the part is a must in the line of coding. So in terms of video search result scraping,  we can try this line of code, and it aims to grab the video information from the keyword query “ring spotlight camera”. We can try different keywords as you like.

request = youtube.search().list(

        part=’snippet’,

        q=’ring spotlight camera’,

        maxResults=’50’,

)

response = request.execute()

print(response)

It’s working and max.50 videos by using maxresult can be grabbed and shown on Sublime Text.

Scrape Videos in Youtube Search Query Result

Now the data from the keyword query is accessible, we assume to grab 3 types of data and let us see how to find the data location.

  • Channel Name
  • Video Title
  • Video ID

First thing first in Python Tutorial is we can see all the data of each video we aim to grab is within the item [….]. And channel name & video title data are under snippet path, and video id is under id path. So based on this, we can write lines of code.

‘items’: [{‘kind’: ‘youtube#searchResult’, ‘etag’: ‘GGoUw-TsOsWwwPIBcKVUozGGrW8’, ‘id’: {‘kind’: ‘youtube#video’, videoId’: ‘d7iJwF-5xGU’}, ‘snippet’: {‘publishedAt’: ‘2017-08-03T19:29:56Z’, ‘channelId’: ‘UCSDG3M0e2mGX9_qtHEtzj2Q’, ‘title’: ‘How to Install & Setup Ring Spotlight Cam | Easy to Connect’, ‘description’: ‘Learn how to easily setup and install your Ring Spotlight Cam & Battery. Spotlight Cam Battery comes with a 1080HD camera with two-way talk, two LED …’, ‘thumbnails’: {‘default’: {‘url’: ‘https://i.ytimg.com/vi/d7iJwF-5xGU/default.jpg’, ‘width’: 120, ‘height’: 90}, ‘medium’: {‘url’: ‘https://i.ytimg.com/vi/d7iJwF-5xGU/mqdefault.jpg’, ‘width’: 320, ‘height’: 180}, ‘high’: {‘url’: ‘https://i.ytimg.com/vi/d7iJwF-5xGU/hqdefault.jpg’, ‘width’: 480, ‘height’: 360}}, ‘channelTitle’: ‘Ring’, ‘liveBroadcastContent’: ‘none’, ‘publishTime’: ‘2017-08-03T19:29:56Z’}},{‘kind’: ‘youtube#searchResult’, ‘etag’: ‘3P6Ws8ggqau3I0Dew2nDsRcsr0Q’, ‘id’: {‘kind’: ‘youtube#video’, ‘videoId’: ‘U-06WEwtaSk’}, ‘snippet’: {‘publishedAt’: ‘2020-01-21T14:52:47Z’, ‘channelId’: ‘UCXn_loz0TlUKarhS6sUoKVw’, ‘title’: ‘Is the Ring Spotlight Camera Worth Buying?’, ‘description’: ‘In this video we review the Ring Spotlight Camera. We like the Spotlight and the Floodlight because it will shine its lights and start recording video whenever …’, ‘thumbnails’: {‘default’: {‘url’: ‘https://i.ytimg.com/vi/U-06WEwtaSk/default.jpg’, ‘width’: 120, ‘height’: 90}, ‘medium’: {‘url’: ‘https://i.ytimg.com/vi/U-06WEwtaSk/mqdefault.jpg’, ‘width’: 320, ‘height’: 180}, ‘high’: {‘url’: ‘https://i.ytimg.com/vi/U-06WEwtaSk/hqdefault.jpg’, ‘width’: 480, ‘height’: 360}}, ‘channelTitle’: ‘HomeAutomationX’, ‘liveBroadcastContent’: ‘none’, ‘publishTime’: ‘2020-01-21T14:52:47Z’}}, 

channel_title = (item[‘snippet’][‘channelTitle’])

print(channel_title)

video_title = (item[‘snippet’][‘title’])

print(video_title)

try:

    vid_ids = (item[‘id’][‘videoId’])

except Exception as e:

    vid_ids = None

print(vid_ids)

Some video ID are missing from the data, the reason is the result also shows the channel page information, or related videos, so for avoiding scraping error in the process, we need to add try/except for scraping video id above.

For looping the scrape process, we need to add a line of code as a parent level before it

for item in response[‘items’]:

Then all videos can be scraped as you can see from the result.

Now we can import the data into a CSV file

(Note: If you are interested in learning how to parse and pass data into a CSV, please check out this article)

Python Tutorial for Digital Marketers 3: How to Write, Parse, Read CSV Files with Scraped Data

Python Tutorial for Digital Marketers

Each youtube video has a unique ID, and the video URL structure basically is like this

https://www.youtube.com/watch?v= + Video ID

Python Tutorial for Digital Marketers

Either you can leverage the format method in the code, or you can leverage CONCATENATE() in the excel file.

Python Tutorial for Digital Marketers

So easy, right? I hope you enjoy reading Python Tutorial for Digital Marketers 5: Create Youtube API & Scrape Youtube Videos. If you did, please support us by doing one of the things listed below, because it always helps out our channel.

  • Support my channel through PayPal (paypal.me/Easy2digital)
  • Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
  • Follow and like my page Easy2Digital Facebook page
  • Share the article to your social network with the hashtag #easy2digital
  • Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
  • You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount code on Buyfromlo products and digital software
  • Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)

If you are interested in more Python Scraping via Youtube API, you can continue to read more in Chapter 6

Python Tutorial for Digital Marketers 6: Scrape View, Comment, Like Data of More than 50 Videos from the Youtube Search List

By Louis Lu

Growth Hacker & Digital Marketer, with a proven record of over 11 years experience in 20+ Asian markets, and 25,000+ connections in Linkedin

2 thoughts on “Python Tutorial for Digital Marketers 5: Create Youtube API & Scrape Youtube Videos”
  1. Magnificent beat! how can I subscribe to your newsletter?
    I had been acquainted with this your broadcast provided bright
    clear idea

Leave a Reply

Your email address will not be published. Required fields are marked *