In this Python Tutorial, I will walk you through how to create a Twitter bot scraping top ranking topics’ data and influencer profiles. By the end of this Python Tutorial, you will find that it’s totally for free and you don’t need to rely on the paid softwares out there anymore. And most importantly, it’s easy to create and custom based on your business needs.
Twitter Bot Component: Twitter Developer Account, API key and token, Tweepy Module, Pandas, CSV
- Why Twitter Bot Is Valuable for Digital Marketer
- Create a Twitter Developer Account
- Create a new App and Get the API Key and Token
- Tweepy Module
- Twitter Connection, Tweet Search and Search Operators
- Scrapable Tweet and the Tweeter Data
- Rename the Data Header
- Full Python Script of Twitter Bot
Why Twitter Bot Is Valuable for Your Business?
As you might still remember, lots of studies focus on the skill of Donald Trump using Twitter for his politician career. Since his joining in 2009, Twitter has become a tremendously popular platform to instantly share views, retweets and comments with users’ real time sentiment.
According to the latest statistics from Statista, Twitter has 206 million daily active users (DAUs) and in this user base, below are the top 10 twitter users by country. Users love to reach through, discuss news, entertainment, professional area topics as well on Twitter.
Twitter’s tweet and retweet volume are hugely large and the average visit duration on the web version of Twitter worldwide was nearly 11 minutes, although the tweet character length is shorter than other social media platforms. In terms of speed and news reaction, Twitter represents the real-time, latest trend and instant sentiment of users in a way in front of events and incidents.
In other words, the top ranking content released by influencers, and sentiment data can give brands and merchants a hint of the specific topics of trend and true feeling from the user perspective. The question is how to scalably scrape these data from Twitter automatically? This is the value of Twitter Bot.
Twitter Bot – Create a Twitter Developer Account
First thing first, you need a Twitter developer account, and it’s for free but it might take some time to get approval by Twitter. The reason we need the account is because of getting the Twitter API key and token used in the Twitter Bot Python script.
For the Twitter developer API account, it has the free version and enterprise version. We don’t need the enterprise version in fact, although it has a monthly limit in terms of tweets pulled, which is 2 million. It will reset on the 1st day of every month.
In terms of the account application, basically you need to use an email address and submit the reason why you need an API. For me, it’s approved by saying it’s for software development.
Twitter Bot – Create A New App and Get the API Key and Token
Once you have the developer account, you need to first create a new App for the Twitter Bot project. There are 3 options and it’s no difference if it’s used for creating a Twitter Bot in a Python script. But I will suggest selecting the production which you can use for live products, such as the web application.
Then, you can get the API key and Token. Below are the 4 keys you need for a Twitter Bot. Please copy and paste them in one place first. You would need them in a moment.
- API Key
- API Key Secret
- Access Token
- Access Token Secret
Tweepy is an open source Python package that gives you a very convenient way to access the Twitter API with Python. It includes a set of classes and methods that represent Twitter’s models and API endpoints, and it transparently handles various implementation details. They are such as data encoding, data decoding, http requests, results pagination, streams, authentication and rate limits
This module can facilitate you to connect with Twitter API and build up a Twitter Bot working for scraping the popular tweets and influencer profile data. Although you can directly code and connect with Twitter API using requests in Python, it would take you much time honestly.
For Tweepy installation, it’s very easy. Take Mac for example, you can just open the terminal and input “pip3 install tweepy”, then it’s done.
As well as other Python script, you just need to input “import tweepy” at the beginning of the Twitter Bot script.
For the details regarding Tweepy API documentation, here is the link
Twitter API Connection, Tweet Search and Search Operators
First thing first, you need to connect the script with the Twitter API. Here is the code. I created the variables in advance, which are the apiKey, apiKeySecret, AccessToken, AccessTokenSecret. You can copy and paste the set of api keys you grabbed just now.
auth = tweepy.OAuthHandler(apiKey, apiKeySecret)
api = tweepy.API(auth, wait_on_rate_limit=True)
Now you have the connected api in the script, and below is the tweepy search tweet method you can use to define your object of the twitter bot. For more details, you can refer to the Tweepy API documentation to custom your bot for the tweet search and collecting data.
list(tweepy.Cursor(api.search_tweets(q, *, geocode, lang, locale, result_type, count, until, since_id, max_id, include_entities)¶.items())
From my point of view, there are three parameters you might use often.
q – query
Compared to other social bots, Twitter provides more deep-dive operator options to specify the search. For example, there is a match type mechanic, in which you can choose exact match or broad match as well as Google SEM ads, it’s very interesting.
Also, you can use hashtags as well as you use in Instagram bot. Or you could use mention either to find all tweets mentioning a specific Tweet account. It’s super helpful for you to either scrape the most popular content or most popular account.
This parameter can specify what type of search results you would prefer to receive. The default is “mixed” if you don’t assign a specific value in the method. Basically there are three types
- mixed : include both popular and real time results in the response
- recent : return only the most recent results in the response
- popular : return only the most popular results in the response
If the USA and Japan are both your target markets, this parameter would be very helpful to narrow down the search object. You can add the country code, or city name in this parameter value. As a result, the scraped data would be only from this country, or city, which includes the influencer profile data.
Scrapable Tweet and the Tweeter Data
There are two types of data you can scrape by using Tweepy. One is the Tweet data and the other is the tweet user data (influencer profiles)
Here is the variable with adding some useful parameters that you would need usually for scraping most popular content and most popular accounts.
tweets_list = [[tweet.created_at, tweet.id, tweet.text, tweet.retweet_count, tweet.favorite_count, tweet.user.screen_name, tweet.user.location, tweet.user.url, tweet.user.followers_count, tweet.user.favourites_count] for tweet in tweets]
Rename the Data Header
Tweepy API data is in a list format, instead of JSON format. By default, the column data header is named by number 0, 1, 2, 3, 4, 5. Frankly, it’s not convenient to find or lookup the data later on in the business analysis.
So you need to use Pandas to rename the column name if you use Tweep to connect Twitter and build the Twitter Bot. Basically you need to use the rename () method and column parameter.
Here is the code sample:
Full Python Script of Twitter Bot
If you are interested in the full script of Twitter Bot for Scraping Most Popular Topics and Influencer Profiles, please subscribe to our newsletter by adding the message “Python tutorial 34”. We would send you the script immediately to your mailbox.
I hope you enjoy reading Python Tutorial 34: Twitter Bot for Scraping Most Popular Topics and Influencer Profiles. If you did, please support us by doing one of the things listed below, because it always helps out our channel.
- Support my channel through PayPal (paypal.me/Easy2digital)
- Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
- Follow and like my page Easy2Digital Facebook page
- Share the article to your social network with the hashtag #easy2digital
- Buy products with Easy2Digital 10% OFF Discount code (Easy2DigitalNewBuyers2021)
- You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount code on Buyfromlo products and digital software
- Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)