HowToDataScience : Scraping Twitter Data

Key Takeaways

[Music] in this video we'll be learning and extract all the tweets that have a certain hashtag or set of hashtags on Twitter so for example the upcoming Aquaman movie has a lot of popularity on Twitter and now because of some recent trailer information that just dropped so let's say we want to find all the tweets that have hashtag Aquaman and have hashtag DC comics in them we'll go ahead and click enter here and we see that we get all the tweets that have hashtag Aquaman and have hashtag DC comics so scrolling through these tweets we see that we might want some other information on each tweet as well for example you might want the timestamp of tweet we might want information about the person that tweeted it for example their username but also such as how many followers they have we definitely want the text of the tweet and we also want information on all the other hashtags that are contained in this tweet you will need a valid Twitter account the Python programming language installed on your computer I'm using Python version 2.7 you can use Python version 3 with a few minor adjustments to your code and optional but highly highly recommended is the anaconda package installer this is something that's going to easily allow us to get a certain tool we need for this project so I'll include some information in the description below for how you can install this alright so the first step in our process is going to be visiting this create an application Twitter website in order to generate a series of access codes which basically tell Twitter hey I'm a valid Twitter user requesting this data please give it to me so it's pretty painless process to create this application let's just walk through it the first is give this application distinctive name so I'm going to call it my tweet scraping application go ahead and enter a description here so I'm going to say this is used to scrape tweets via hashtag okay this website you can just put a placeholder name so I'm going to put HTTP colon slash slash place holder comp this field here is optional go ahead and read this Twitter developer agreement if you would like to hit this check mark and create your Twitter application so give it a second here and your application has been created so that was pretty easy now you want to click on this keys and access tokens tab right here and you want to make note of four different coats so go ahead and copy them into a notepad file or put them somewhere safe the first is this consumer key the next is the consumer key secret right here the next you have to go down here and click on create my access token under token actions once you click on that give it a second scroll back down and you'll see two new codes appear when it's called access token which is right here and the last is called access token secret right here so copy all those down somewhere safe and try not to give them to anybody keep them as secret as possible because if someone has your access codes then they can start making requests to Twitter on your behalf and you don't necessarily want that all right so our final step before getting right into the code is going to be downloading a python library that we need that's gonna make our lives a lot lot easier when we start doing the code so that's why I said downloading this anaconda package installer is a really good idea because it's gonna make downloading this library a lot easier so go ahead and navigate into your anaconda terminal here go into your working directory for me that's just see users rittvik right here and all you have to type is pip install tweet hi tweet pi is a library I talked about so it's basically an interface between Python and Twitter that has a lot of pre built-in functions that will see that make our lives a lot easier click enter it's gonna go to the web and collect everything we read for a tweet pipe and that's it it says successfully installed tweet PI three point six point zero and that's literally all we have to do for this step alright without further ado let's jump right into the code and the code in this case could not be simpler thanks to the tweet pi library whereas in previous scraping videos we had to jump through the multiple functions and understand different kinds of components here there's only one function that we have to worry about and it's called search for hashtags now search for hashtag takes in five arguments the first four of which are very similar they're simply the consumer key the consumer secret access token and access token secret that we got from the Twitter application website the last one is simply the hashtag phrase which in our case is hashtag Aquaman and hashtag DC Comics now the first thing that this search for hashtags function does is creates an authentication object called auth which is created from are four different access codes and it's simply going to be used to tell Twitter that we are a valid user requesting this data now the next thing that this function does is creates an API object API stands for application programming interface but the best way to think about it is basically a language that we need to talk to Twitter in in order to request our data so that's why the tweet I library is so so useful and powerful because it has all this language built in we don't have to actually understand the language that Twitter requests data in because Tweep I will just convert our request into the language for us as long as we have passed in this authentication telling it that we are authorized to request this data the next thing we do is generate the name of the spreadsheet we'll be saving all our data too it's basically just created using the hash tags we want strung together using underscores and then the next thing we do is go ahead and open up that spreadsheet for writing alias it with W we can think of this as just the spreadsheet itself going forward the next thing we do is write the header row so each of the fields that we're going to be filling up the spreadsheet with us in our case that's going to be the timestamp of the tweet the actual text of the tweet the username of the person who posted this all the other hashtags in the tweet and how many followers this user has we get into this for loop here which says for each tweet that I gather and go ahead and write that tweet into my spreadsheet now the true meat of this function happens in getting this list of tweets that we actually care about and that happens right here the only two things that this tweet I doc cursor function which gets all those tweets takes in is first the API object that we created up there so remember this is a object which contains the language as well as the authentication that we need in order to talk to Twitter the next is of course the hashtag phrase that we passed in and here I add to filter out the retweets because I don't necessarily want every single retweet of a certain tweet but if you do you can just remove that I want only tweets that are in the English language you can change that or remove it if you want all tweets and I put here tweet mode equals extended because sometimes the text of the tweet gets cut off and I wanted to avoid that and I specified here I want only a hundred tweets right now you can always increase that if you want more and decrease it if you want less now for each tweet that I get back I go ahead and just extract the relevant piece of information from it so if I want the time it was created I want tweet created app if I want the whole text of the tweet I just do tweet da full text and a couple of cleaning up operations right here and similarly for every other field I want it's just some basic operation on the tweet I got back you can go ahead and read through that now let's go ahead and see this in action so I'm gonna go ahead and run this cell here and now it's just asking me for all the different codes that I extracted from the Twitter application website so I'll go ahead and speed that bit up for you guys now it asks me the hashtag phrase of course I want hash tag Aquaman and hash tag DC Comics know you can do a lot of things here you can change this and to an or if you want if you just want to match one of them you can also do a bunch of other things I'll include some documentation in the description below now go ahead and enter and it's done let's go ahead and see the results let's go ahead and open up our working directory we see there's a new file here called Aquaman underscore DC Comics the name of which remember was made using all the hash tags that we wanted strung together using underscores let's go ahead and open this up we can open it up in Excel now let's zoom in a little bit so you can see it better we see all the data that we want is right here so let me expand this so you can see it we see the time stamp for each tweet we see the tweet text we see the username of the person who tweeted this we see the list of all the hashtags that are in this tweet and we see the number of followers that this user has and as we want it we have exactly a hundred tweets in our results all right so all the code that was used in this video is freely available at my github so you can feel free to go through that at your own pace I'll go ahead and link that in the description for you guys now if you learn something in this video go ahead and leave a like and subscribe to my channel for more videos on how to data science