How Do Random Forests Work & What is Ensemble Learning

Key Takeaways

what is going on guys welcome back in this video today we're going to talk about Ensemble learning how it works why it works and how to implement it yourself so let us get right into it [Music] all right so we're going to talk about Ensemble learning in this video today and the basic idea behind Ensemble learning is to combine multiple machine learning models into one model so for example in the case of a classification task like classifying images of dogs and cats we would not just ask one individual model for its decision for its classification it would ask multiple different models and then we would average their decisions for example to reach a less biased and less error-prone decision so the idea is we have individual models these individual models can have or can make individual mistakes that are maybe quite unique to the model type or maybe quite unique to the hyper parameters so they might overfit the data or make some other mistakes in a very unique way that all the other models maybe don't do but those models again can have their own mistakes and by averaging out their decisions we hopefully reach a conclusion a classification that is less biased and more correct so when we evaluate it on the full test data set we get a higher performance than when we ask all the individual models that are part of the Ensemble that's the basic idea and today we're going to look at a specific implementation of that which is the random Forest classifier the random Force classification and the idea here is to take multiple individual decision trees that do the classification task and to then combine their decisions into a random Forest so multiple trees are combined into a forest and this is the random Forest classification that is based in multiple decision trees so we're going to do this first of all from scratch so we're going to train individual decision tree classifiers and we're going to then average their decisions and then we're going to look at the actual random Force classification from scikit-learn at least when I prepared this video I got the exact same results so let's see if this is the case again because there is some Randomness involved here but we're going to start here by importing or actually by installing uh the packages scikit-learn so scikit Dash learn and numpy those are the packages we're going to need today numpy just to work with arrays and scikit-learn because this is the package for traditional machine learning algorithms and models so we're going to start by saying import numpy S and P and then we're going to say from sklearn dot data sets we're going to just use the breast cancer data set which is just a data set containing some information about the sizes of tumors and then it tells us if it's malignant or benign it doesn't really matter what data set you choose here we're just going to use some data set that is useful for classification tasks so here we're going to import load breast cancer then from scikitlearn dot model selection we're going to import the train test split from scikit-learn dot uh tree we're going to import a decision tree classifier from sklearn dot Ensemble this is what we're doing here we're going to import the random Forest classifier again we're going to first create our own Ensemble with decision tree classifiers and then we're going to use the actual Ensemble classifier here from scikit-learn to compare the performance and then we're also going to say from sklearn dot metrics we want to import the accuracy score so those are the Imports and now we're going to start by loading the data by saying load breast cancer and then we're going to split it into X and Y so into the actual data into the actual uh features and then into the target variable as well so data dot data and data.targets now this XY data we're now going to split it into train and test so into training set into a testing set so that we can train the model on the training data and evaluate it on the testing data so we're going to say x underscore train X underscore test y underscore train y underscore test and this is going to be equal to train test split X and Y and the test size is going to be 0.2 so 20 of the data is going to be for testing uh this is now just our data what we're going to do now is we're going to say how many trees we want to have so let's say we want to have 100 trees what we're going to do is we're going to say 4i in range and then 100 because we want to have 100 trees and above that we're going to initialize an empty list so we're going to call this the tree list for example which is just going to be empty and we're going to Now train decision Tree classifiers on the data and we're going to add these classifiers to the tree list so tree equals decision tree classifier um and we're going to say Max features are going to be equal to square root so this is just a method for determining the Max features uh and then we're going to say now subset this is important I forgot to mention this of course we're training the classifiers on subsets on different subsets of the data because we want the models to be as independent as possible now this will not happen it will not be completely independent especially because they're all decision tree classifiers but we want to have at least as much difference as we can have as much Independence as we can have so we're going to use different subsets of data and for this we're going to say subset indices foreign equal NP random choice and we're going to choose from NP arrange length x train so we just pick some random indices here how many size equals integer so truncating it length of X train divided by two so we basically actually I can also do a floor division here right why did I do it like that I don't know I think this is this is better like this so we have the subset indices and what we want to do now is we want to actually get the data so actually from these indices get the actual rows here from the X strain and Y train so we're going to say x underscore train underscore subset is going to be equal to X train next come on X strain I'm still getting used to my new keyboard here um subset indices and then y train subset y train subset indices uh and then we want to fit a decision tree classifier onto this so we want to say tree fit X strain subset my train subset and then this tree shall be appended to the list that we just created Above So append this tree and again hopefully all of these are going to be somewhat independent we want to have 100 trees that are all kind of different and then we want to let them vote on the same issue so we're going to stay here now predictions pretz is going to be an empty list and then we're going to go through the test data now and we're going to let each model vote Let each decision tree vote on the issue whether this is malignant or benign and then we're going to see how correct The Ensemble is so how correct all of these models uh together are when we average the decision so for I Tree in enumerate and then we're going to enumerate of course the tree list and then we're going to say the individual underscore predictions are going to be equal to 3 dot predict X underscore test so we predict on the test data what the Y data is going to be in the individual accuracy so for this individual model the accuracy is going to be calculated by accuracy score y test and the individual prediction so we let this one tree that we're currently looking at we'll let it predict on the data and we evaluate its individual accuracy so not of The Ensemble of the individual decision tree and we want to then print here just as a message so that we can keep track of this tree and then we can say I plus 1 so that we start counting at one uh so three I plus one is going to have an accuracy or has an accuracy off and then individual accuracy so we can see what the difference is between the individual trees and The Ensemble and by the way this is not always going to be better not in all cases will you have better performance just because you use an ensemble but oftentimes you will so you can also use by the way from scikit-learn the voting classifier to combine any models I have a video on this by the way but not always will you outperform the individual classifiers especially when you have some very powerful models like random Forest classification in The Ensemble because of course you can have an ensemble inside of another Ensemble this is also possible um so yeah so we have the tree accuracy here and then we one of course append the predictions that we made to the list so individual pretz like this now with these predictions from all the trees we now want to average them we want to get the mean prediction we want to round it of course up or down so we want to get the mean since we have here between zero and one we will get something like 0.2 0.8 or something we want to round it to what it's closer to so if it's 0.8 you want to have a one if it's 0.2 we want to have a zero so what we do is we say predictions I want to turn this into a numpy array so predictions is going to be NP array predictions and then we want to say mean underscore predictions or actually let's yeah let's say meme predictions is going to be equal to NP dot mean and then we want to say predictions access is going to be equal to zero and around this we can now say np.round like this so this takes the mean and then rounds the mean up or down depending on what the value is and then we get our predictions and now we want to calculate the ensemble Ensemble underscore accuracy which is going to of course be just the accuracy score applied on these mean predictions and uh and the test data so let's do it the same way that we did it before y test and mean predictions maybe we should call this a different name maybe we can call this uh Ensemble underscore predictions like this um I think we can run this now this shouldn't take too long and of course we need to print The Ensemble accuracy but yeah you can see a clear picture I think in this case it's very obvious I mean not exactly I think you have a lucky tree here for example but if you look at most individual trees they have an accuracy around 0.9392 9089 stuff like that we have this 32 tree here which for some reason has 95.6 percent but you can see that the ensemble in the end has oh it actually also has 96 okay so it's actually quite good uh What was 32 exactly nine five okay so actually The Ensemble outperformed every single tree uh in The Ensemble this is the accuracy when you combine all of these votes but the individual votes as you can see here if you just look at one individual tree for every single classification all of them perform worse than the whole ensemble and now we can see if this is also if we get the exact same accuracy with a random first classification we can say here random Forest equals and then random Forest classifier train it of course on or actually first of all we need to specify how many trees we want to have uh to keep it consistent we're gonna go with 100 here oh it's not entries it's in estimators there you go and then we're going to say RF dot fit we're going to fit it on the training data this is automatically going to do everything behind the scenes that we did manually above so xtrain y train and then we want to just say RF predictions we don't want to use evaluate or score because we want to do it the same way that it that we did it up here with the accuracy score so why credits uh not y pets rfpreps is going to be equal to RF dot predict and we're going to predict on the test data and then we want to say print accuracy score of Y test and RF press and you can see in this case actually the random Forest classifier outperformed our Ensemble here there are the same here our classify outperformed the random force classifier from scikit-learn here they both have again the same accuracy yeah here again it seems like we're getting better and better yeah but you can see if we look at a very good example here um now can we find a counter example is any one of these trees yeah this one has uh this one here has a performance of 0.93 whereas our Ensemble performs worse but this is very rarely the case I mean here we also have an example but you can see that most of these individual trees are worse at the task than our Ensemble and this will also be the case if you have more data so the more data you have the less lucky the individual tree can be but yeah you can see that most of the time we get pretty decent scores here in the individual trees are not very powerful when it comes to the predictions so this is the power of Ensemble learning in a random Force classifier just means multiple decision trees working together so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you next video and bye