How to Collect Business Reviews Using Python - Part 1 - Python Yelp Sentiment Analysis

Key Takeaways

what's happening guys welcome to part 1 in the series on how to create a sentiment analyzer using Yelp business reviews in this video we're going to go through how to get your data off yoke using beautifulsoup and the requests module from Python and then later on we'll start analyzing our data cleaning our data in the next couple of series of videos so let's get right into it so the first thing that we're going to need to do in order to get our data off Yelp is to create a request and grab add data from there to begin with so to do that we're gonna use the requests module from Python and we're gonna import requests and we're also going to need beautiful soup so beautiful soup just makes life a whole lot easier when you're trying to traverse a HTML result so we're going to import beautiful soup I'm gonna do that by running the command to be s4 so from bs4 import beautiful soup alrighty cool so what we're going to do now is create a request to the website that contains our review data so I've just picked a random Tesla dealership so let's just use that for now and you can see the link there so we'll copy that and our request is so we're gonna create store our requests or a result in a variable so we're just gonna call the variable after now and we'll make a request by running request get and we'll pass at URL there if everything has gone successfully we should be able to get a status code of 200 so we can check that by getting the attribute add status code and you can say it's 200 now this doesn't initially return the text for the result so what we can do to grab that out is just use the text attribute so auto text and you can see we've got everything there now this isn't all that readable but beautiful soup is gonna make that whole lot easier in a second so our reviews now what we want to try to do is find patterns in terms of our reviews in order to help or use beautiful soup and and what I'm talking about when I mentioned patterns are patterns in the Dom so what we can see here is that each review sort of within its own div and you've got another one yeah so what we can do is inspect that and you can see it's in a paragraph and then it appears to be in another div called review content so let's just check in another one let's see if that holds true for more than one seems to be the case and then same thing and you can say this data review Aidid is changing age time let's check this one yeah it looks like they're in the review content and inside of the paragraph so all we can do is we can use beautifulsoup to traverse our result bring back the div which contains review content and then grab just the text within the paragraph from there so what we'll do in order to do that is will create a soup using beautiful soup and to that object we're going to pass our text which is our result and we're going to be using the HTML parser alrighty so that should be fun so what we can then do now is use the final function to grab all the review content dibs and we're going to say class equals review - content and that should return a list of divs which all have the class of review content if we run that looks like we've got let's just check the type ok so it's a result set which is really just a list and then what we're gonna do so we're gonna store that in a Diggs variable so this will last a loop through and grab out our paragraphs because remember that inside the actual text we want is in each paragraph so let's go ahead and do that so we can create a new variable called reviews reviews and that will be a blank list to begin with and then we're just going to loop through each one in the do so for deer and divs we are going to so div dot fine so we're gonna use the fine method and we're going to look for the paragraph and we let's print that out just to make sure we're getting the right thing to begin with okay so it looks like it's getting each paragraph that's looking good but it's still got the markup so we don't actually want that so we just want to test that's looking a little bit better just print a line break between each ones we can see what was actually good alrighty so it looks like we've actually got the review text so that's perfect so what we're gonna do is we're going to throw all of that you know array and eventually later I'm going to use pandas to throw it into a data frame so we'll put it appended to the reviews list by writing reviews dot append and what have we done wrong I'm sorry called this review haven't we we also had one too many brackets let's just do that alrighty and there we go so now got our reviews in an array and we're ready to jump over to step two so in video 2 we're going to go through how to analyze our review data so we're going to be calculating some useful metrics and we'll go from there