Scrape HTML Table using rvest and Process them for insights using tidyverse in R

Key Takeaways

welcome to own a little coder in this video we are going to learn how to scrape an HTML table from internet and then do some data analysis on it this is mostly to introduce you how to scrape an HTML table so first of all what is an HTML table so on internet any webpage could have table and that table could be in different forms one they can simply use the tag HTML so in this case if you inspect element so you can see the tag table so this is the HTML tag for table but there are other instances where people can actually use div to define a table structure but in this case we are looking at only places where they have a table tag so what we are trying to do in this video is that we are trying to find a page in this case which is the highest-grossing films in the United States in Canada and then we are going to scrape this particular table which means we are going to programmatically extract this table as a data frame into our our session and then we are going to make some visualization on top of it so let us go ahead and see what we are going to do and how we are going to do so first you're going to use the package called harvest so Arvest is a package or package this is the package that we are going to use for our web scraping this package is very similar to beautiful scoop or scrapey on Python so if you are familiar with those packages so this package helps you extract the content on it from Internet let's say the page and then from that web page it helps you pass a particular set of nodes from selector and extract the content that is so important for you so you can see the example in this but in this video we are going to focus only on HTML table not anything else so first let us make sure that this package is installed so let us do a library always ok this package is installed here if this package is not there so if you are installing titleist this package becomes in image part of it so if you don't want to do any of those things and you want to install only August you can use installer packages and you install Arvest so that way this package is there next is you have to identify the URL from which you want to extract the table so in this case this is the URL from which you want to extract the table so let us copy the URL and then go to our code so first we have loaded the package successfully next we have to define the URL from where content has to be extracted okay so I'll just go to a new code instead of this I'll just give library ID was execute and then let's call it on content read underscore HTML is to extract the content of the HTML on the URL name so once we read it you can see that it has explained it to you if you want to see what is inside it you can just print content and see what is inside it so it's a bunch of HTML tags but without passing so now what we want is we are interested in the table so in this page you can actually see there are one two and three four like a lot of tables are there so we are in this particular case interested in this particular table so let us go ahead and say okay we want tables we'll call it tables content pipe this pipe operator from Agata package also can be used in this case and say HTML tables when you do this thing or you might get an error saying that this table has inconsistent number of columns do you want to fill it so this is to say that there are some instances where the column is not matching so do you want to fill it with some null value so let's say fill is equal to true and then extracted so now you can see that there is a tables object which is ideally a list and then you can see the length of tables sorry coming from the Python hole so I'm getting confused length of tables are part of 173 I'll sometime today okay so what we are interested in this we are interested in the first element the first table so let us say first table table so because it is a less realistic table so first now you can see that the first table the first table is a data frame so unlike this this has a data frame symbol so click the first two table and see what is inside it okay so we have got a decent table but the problem is now everything in the table is a character so let's do a glimpse glimpse of first table everything is a character because your first row is a character and titles of this was also we don't need this so what we need to do is we need to know remove this thing so the simple way we can do it is first table equal to first table remove the first element then now we have removed the first element and then you can see but still if you do glimpse you are going to have everything s character so in this case we have to parse those things that we want s number okay so before doing that we can also see either term the column names are wrong slightly bad this what do I mean by that that you have to use curly brace and then you have taxes the column name so for that what we are going to do is we are going to use a package called janitor janitor and there's a beautiful function and janitor called clean names so what we are going to do is first table first table okay so if you don't know how to do this pipe control-shift-n or command shift M on Mac command shift M 1 Mac ctrl shift M on windows clean names so now if you see you can see that names have changed so names of first table so earlier the names look like this now the names are lowercase with underscore no need to use curly braces about to hurt us so little blitz it says if we want to do this plot on this plot some idly using the lifetime gross and it is having the title and then the lifetime across so let's let us say that we want to do something similar to this thing so which means on this this these columns are still these columns are still character we want to pass it as number so what we can do is we can start building our first data pipeline first table mutate and what do we want we want the lifetime gross life I'm sorry you want the lifetime gross is equal to cars number lifetime gross okay and I think at this point we have our own values path so now we can say I arranged descending order life time gross okay now let's say we want only the top 20 head of 20 so let's say top 20 so you can see that there are top 20 films you have got 20 films with you can see that it is not formatted there is no dollar symbol there is no comma so this is this is no character proper character so now we have the top 20 films let's get into GZ plot to build a plot like this giome bar a yes X is equal to the title which is title and Y is equal to life time gross stat is equal to identity you exude this thing this is the plot that you're getting it's so so now you have to you have to change the orientation of the label but instead of doing that what we can do is we can use some coordinate flip - flip the coordinate but fortunately in the latest ggplot you don't have to really do coordinate flip what you can actually do is you can say you can inverse the coordinates instead of X Y you can swap so now your Yub comes from the categorical variable Attucks becomes a continuous variable so in this case it automatically does it for you a third coordinate flip and this is good but the problem now once again this plot is not ordered so what we can do is we can go back one step before say mutate mutate or title is equal to if fact the order and title comma lifetime gross okay then so we have an ordered plot now this is all good so let's try to hide our fill value which is filled is equal to let's hash one one two two zero cool okay so that's really gross red color so let's say okay I think the first caller was better let's say we can say wait agree raise okay cool so this is how our plot looks right now I'll just put blue I just cannot tolerate this color or plot isn't ever title or anything laps title is equal to top grossing top 20 top 20 causing movies in US and Canada and our source is caption data source Wikipedia is the plot your plot is ready it looks nice you can you can zoom in and see how it looks or so you have a nice beautiful not now let's say instead of based on lifetime gross you want it um based on let's say the other column which is initial cross the left hand goes to which is digested one so if you see the second one the first one is on adjuster the second one is adjusted let's say we want to adjust it we can still do it we can say notate lifetime growls gross underscore two which is stop to change here and we just have to change it everywhere we have this thing so you can copy this entire thing and paste it and change left and cross just into it nice touch left nostril and then you have different set of movies so yeah so the data set mostly so in this video the main thing that we try to learn is we try to learn how simple it is to read an HTML table using Arvest this function comes from our this house how very simple it is to read an HTML table so it is not just this table even if you want the second or third table you can still go ahead and read so just just to see that whether we can read the third table so let us try to say okay our third [Music] table is equal to tables three and let's see what is our table okay so this is all messed up this is because of because it has a lot of elements in here so so we have to fix a lot of things in this case let's let's redo the second table in this case the second table and we'll call our second table 100 yeah so we have all these things right so now it has red and now we can even see number of estimates per year so what we can actually do is we can say okay second table is there muted okay we want to clean up the column name screen names okay and after we clean the names we want to say if you don't know what are the names first we'll do clean names and then we'll assign it back second table done so we can see in names of second table and water to stand so we can do sec second table mutate I'm just a gross okay let's take a district cross gestured grow is equal to parse number of mr. cross then we can group it by ear and then we can say summarize total and gesture cross is equal to some more for just a close okay at this point we have this data for every year so what we have to do is let it is sorted by which year which has more number of this thing descending of total adjusted cross so now we can see the top ten years when your total cost so we can make a G G naught plus G own line yes X is equal to a year y is equal to adjuster cross there is something wrong okay it's called total adjustable ah sorry in terms of the topic top grossing movies so again the main thing in this video is that the main takeaway for you to do is that on the HTML table how easily you are able to read HTML table using HTML underscore table and then how you can convert on some raw table into some inside that you can get so the things to note here is always package how to extract it an HTML table and from that table how we are doing data pre-processing and then finally we are finding out some insight so I hope this video was helpful and if you have any comments please let me know I wanna see you in the next video thank you for listening see you