Python Requests Tutorial: HTTP Requests and Web Scraping

Key Takeaways

the request module in Python lets you make HTTP requests so that you can interact with any website or API directly from your python app it is one of the most highly downloaded python modules and also one of the most widely used in this video we'll walk through how to install the module and how you can use it this includes making get and post requests to websites sending python objects as Json data and also how you can handle common HTTP errors or timeouts if your request is taking too long we're going to go through all of that you using really simple Hands-On examples so let's go ahead and get started unfortunately the request module is not built into python by default so you'll first have to install it using this command to make a simple get request to a website create a new python file and use this code when you run the script python will make a get request to this example URL and this is just a simple boilerplate website for testing after it makes this request the results of that request will be stored in this response variable and you can call this anything you like it doesn't really matter but I'm going to call it response and if you run it you might see that nothing actually happens but that's because we haven't done anything with the response yet but we can print it out and then you'll see that the response is actually there now this request is exactly the same thing that a web browser does when you use it to visit a website so if you go to any website and then you click view page Source all of this content is sent to us from this HTTP request all of that uh information is actually contained here within this response that includes the HTML the CSS and the JavaScript but before we dig into the actual contents of the response you'll probably want to know the status code of the response and that's this 200 number here HTTP status codes are the first thing you'll probably want to know because they tell you the outcome of a request anything in the 200 range so anything between 200 and 299 uh you usually means a successful request anything in the 400 range is a problem with a client so for example a wrong password or you're trying to access a file that doesn't exist these codes are a universal standard as well so once you've learned them you can pretty much figure out how to handle responses from any website or API you can get the status code of your response by using this status code attribute of the response object if everything is working properly and you print out the status code you should get a two 100 which in this case means a successful request once you know your request is successful you'll probably next want to access the actual content from the endpoint if this is a website you might get HTML and CSS content like this and even though the content looks kind of raw it's actually what is ultimately used to render the web page on your screen so anything you see on a web page anything that you can click will be contained within this content alternatively if your endpoint isn't a website but it's an API instead you might get some Json data like this as your content response either way you can use the content attribute of the response object to get this data here's an example now if you're just trying to get raw content from a website it can be pretty hard to read it like this but if you want to use it then there's a chapter on web scraping at the end of this video where I'll show you how to parse and use this data now get requests are useful if you just want to read some information on a website or just view certain things um but if you actually want to send data to a website for example create a post about something or submit a blog or an article you probably need to use a post request you can use this request. poost method and this time I've changed the URL to http bin. org which is a website that gives you a bunch of endpoints that you can use for testing HTTP request so if you just go over to http bin. org you can see all the different methods that are available to you here so here if we click on HTTP methods you'll see that there's a post request the endpoint is just SL poost and this one will basically just Echo back the post parameters to us so in this example we are sending this post request to this post endpoint and then we're passing this python dictionary object as data as a Json object using this Json parameter similarly to view Json response data you can use this Json method on the response object and you'll get that data back as a dictionary also notice here that Json is a method not an attribute so you need to use the parenthesis for it to work so let's go ahead and try that out and here you can see that the response is a python dictionary now and we have our original data that we sent as the payload to this request um and you can change this to whatever you want so you can add new fields for example if I want to add account 100 um and I send that you'll see that the request is able to process it just fine so this is how you can send data to an HTTP endpoint when you work with HTTP request not everything is always going to be working well all the time time it's likely that you're going to run into a lot of errors it's important to handle errors properly so that your app can figure out what to do next with the request Library there's two ways to go about this first you can check for error codes directly using the status code attribute that we saw earlier and in this example if the status code is not 200 which means a successful response then an error message is printed and here I'm using a new endpoint that always fails with this 400 status so this is a really handy endpoint if you want the request to simulate an error for you and I can use this to test that my app does handle that error and print out the status message correctly with this condition so here you can see that it prints out correctly and you can also change this if you want to test a different type of error for example I can change this to a 500 and then I can run this again and you'll see that the error that it returned is different but now if I change this endpoint to return at 200 status which means it's a successful response you'll see that this error condition doesn't get activated ated and the error message is not shown that's because this request is successful alternatively to handle errors you can use this raise for status method to raise an exception if the request fails this will actually throw an error so you can use it as part of a TR catch expression or just force it to crash your app in this example if the request fails for example it returns a non2 200 status code then this HTTP error is thrown but because we try and catch it using this accept statement this exeption actually won't crash our app but instead it just print this error so it will do mostly the same thing as our previous example except it's just a different way to express it you might find that doing it this way actually gives you more information about the error because um before we only have the status code here it actually tells us the meaning of the 404 error which is that this item is not found and if you change it to a different code for example let's change it from 404 to 403 and then if you run it again you'll see that the message has changed so 403 means it's forbidden so this is an authentication error and maybe we'll change it to something completely different like 500 um and if you run this you'll see that it says an internal server error so this is actually more useful if you want a more verbose error message and you want to do something with that next let's talk about handling timeouts if your app has ever stalled or gotten stuck after an HTTP request it's probably because there was a network issue and you haven't set a Timeout on that request a Time out will force a request to fail if it doesn't respond after a number of seconds but by default the timeout value is set to none which means that it will wait forever this is usually not good because most HTTP requests shouldn't take longer than a few seconds to process the exceptions are if you need to download or upload a large file to set a timeout for your request you can pass this timeout parameter to the request method this parameter specifies the maximum number of seconds to wait for a response before raising this time out exception in this example the request to http bin SL delay sl10 has a timeout of 5 Seconds if the response is not received within 5 Seconds the timeout exception will be raised so if I change this delay to 2 seconds then this request will actually succeed because it's under our timeout of 5 Seconds but if I run this as it is and I leave the delay at 10 seconds then the response will fail after waiting for 5 seconds so let's go ahead and give that a try I have a delay of 10 seconds and my time out is 5 and let's run that and then after 5 Seconds it actually fails and it says read timeout um and if I don't set this at all then this will basically just wait until the request succeeds and if I set this delay to two and I run this then this request will succeed even though the time out's still here that's because the timeout can wait up to a maximum of 5 Seconds most of the time it is better to just fail explicitly rather than wait and see and not know what's going on with the request if you don't hear back from your request after a few seconds it's likely that something went wrong with the connection and the right move is usually just to try again one more time as soon as you can now this is getting a little bit into system design territory but if your request still doesn't succeed after the first retry it might be better to give up that's because if there's actually a real issue going on with a server then you could end up overloading it even more by always retrying this is sometimes called a brown out or a retry storm now let's move on to setting headers setting headers in a request allows you to include additional information in the HTTP header here's a simple example of an HTTP request and you can see that all this stuff here is the header most of the time you don't normally have to fiddle with configuring these things yourself uh but it's typically used for things like authentication or indicating what type of data you'll be sending or expecting to receive to set headers you can create a dictionary with the header values you want to use and then just pass it as the headers parameter in the request like this and here we set the authorization header with a beer token for authentication purposes so you might need to do this if you're using something like JWT tokens for authorization okay so this works nicely and again you can see here that this URL let us test setting headers to this and then it'll just Echo back the headers it's received now that you've learned the basics of making HTTP requests with the request module you can use it for simple web scraping web scraping is the process of extracting data directly from websites people use this to scrape anything from financial data job posts e-commerce listings and so on to scrape a website you can start by using this get request method to receive the raw HTML content of the page but as we saw earlier this is pretty hard to read in its HTML format so if you wanted to know the title of the page or the text content on the page or maybe you want to know what links there are and where they lead to then you're going to first need to parse all of this information which will turn it into a data structure that makes it easier for us to use to do that we can use another module called Beautiful soup so first install it by using this command once you have it you can make a get request to this website example.com and then retrieve the HTML content we then turn that into a soup object which is a data structure that makes searching and traversing the HTML content much easier so let's go ahead and try that out and here I'm going to print the soup object and then when you run that you can see that this content is now actually quite a lot easier to read even though we're not doing anything with it yet now I'm not going to go too in-depth into how you can use this but here's just a basic example of how you can get the title the content on the page and the links available on the page so let's go ahead and try that and now if I run it you'll see that it prints out the title it prints out the text that you see and then it also prints out a list with all the links available and there's just one which links to this URL so this is just a basic example of how you can use requests with beautiful soup to scrape information from a page you can also customize this code to scrape different websites and different types of data based on things like element IDs types class names and so on you will need to know the basics of HTML to be effective at this if you do want to build a website scraper just remember to be ethical about it and respect the website's terms of service and also be mindful of any legal restrictions of scraping that data so far the request module is looking pretty useful but how does it compare to its Alternatives well python actually has a built-in module called URL lib that you can also use to make HTTP requests as well the main difference between this and the request module is the level of extraction they offer to you as a user which directly impacts how easy they are to use here's the an example post request in url lib it does the same thing but compared to the request version it's less intuitive to work with I think if you are comparing between these two then I think that these are the two Dimensions that matter generally though I think that request is a better choice because his abstractions are higher level and you can get started much faster with it but if you are constrained by what modules you can install or if you only want to stick to built-in modules then URL lib will also get the job done that's it for this video tutorial if you enjoy this then please subscribe to the channel and let me know what type of content you'd like to see next otherwise I hope you found this useful and thank you for watching