Gradient Boost Part 2 (of 4): Regression Details

Gradient Boost is one of the most popular Machine Learning algorithms in use. And get this, it's not that complicated! This video is the second part in a series that walks through it one step at a time. This video focuses on the original Gradient Boost algorithm used to predict a continuous value, like someone's weight. We call this, "using Gradient Boost for Regression". In part 3, we'll walk though how Gradient Boost classifies samples into two different categories, and in part 4, we'll go through the math again, this time focusing on classification. This StatQuest assumes that you have already watched Part 1: https://youtu.be/3CC4N4z3GJc ...it also assumes that you know about Regression Trees: https://youtu.be/g9c66TUylZ4 ...and, while it required, it might be useful if you understood Gradient Descent: https://youtu.be/sDv4f4s2SB8 This StatQuest is based on the following sources: A 1999 manuscript by Jerome Friedman that introduced Stochastic Gradient Boost: https://jerryfriedman.su.domains/ftp/stobst.pdf The Wikipedia article on Gradient Boosting: https://en.wikipedia.org/wiki/Gradient_boosting NOTE: The key to understanding how the wikipedia article relates to this video is to keep reading past the "pseudo algorithm" section. The very next section in the article called "Gradient Tree Boosting" shows how the algorithm works for trees (which is pretty much the only weak learner people ever use for gradient boost, which is why I focus on it in the video). In that section, you see how the equation is modified so that each leaf from a tree can have a different output value, rather than the entire "weak learner" having a single output value - and this is the exact same equation that I use in the video. Later in the article, in the section called "Shrinkage", they show how the learning rate can be included. Since this is also pretty much always used with gradient boost, I simply included it in the base algorithm that I describe. The scikit-learn implementation