Random Forests - Explained

Key Takeaways

Ask one expert for a prediction and you might get a confident but wrong answer. Now, ask a hundred people, let each one judge independently and take the majority vote. The crowd, on average, outperforms any single individual. This is the idea behind the random forest. The building block is a single decision tree. We have data in two classes, blue circles and red triangles. The tree asks yes or no questions to split the data. Is X1 greater than minus 0.5? The draws a vertical line through the space. Within each region, another question draws another line. And each rectangle gets assigned to whichever class dominates it. So, the tree is just a series of axis-aligned splits, carving the space into boxes. But a single tree keeps splitting until every training point is perfectly classified, even the noisy ones. A simple boundary would do the job just fine, but that's not what we get. Instead, the tree produces this jagged staircase, memorizing noise rather than learning the true pattern. Small changes in the data produce completely different trees. A single tree has high variance. So, instead of relying on one tree, we create many random subsets by sampling with replacement. That's called bootstrapping. Each subset trains a separate tree. Some points appear multiple times, others are left out entirely. Every tree sees a slightly different version of the data. Now, if every tree sees all features, they all latch onto the same dominant one and end up looking similar. So, at each split, the tree can only choose from a random subset of features. This forces diversity. Each tree discovers different patterns in the data. When a new point arrives, every tree cast a vote and the majority wins. Because each tree trained on different data with different features, their errors are independent and they cancel out. The noisy predictions of individual trees average into a smooth, stable ensemble. That's the power of a random forest, many weak learners combining into one strong one. And that's basically how it works. If you found this helpful, hit that like button, subscribe for more and drop a comment if you have any questions. See you in the next one. Bye-bye.