What each new tree in a boosted ensemble is actually fixing

Ask what gradient boosting does and most explanations start with the ensemble: many weak learners, combined, beat one strong one. That's true, but it skips the more useful question — what is each new tree actually trying to fix? The lab above answers it directly: step through the rounds and watch the residual strip shrink as the ensemble curve bends to close it.

Boosting is functional gradient descent, not a voting scheme #

Random forests and boosted ensembles both combine trees, and it's tempting to file them under the same idea. They aren't. A forest averages many independent, high-variance trees fit to bootstrap samples — the trees don't know about each other. Boosting fits trees sequentially, and each one is trained on what the current ensemble got wrong.

Formally, gradient boosting minimizes a loss $L$ over an additive model $F_M(x) = F_0(x) + \sum_{m=1}^{M} \eta\, h_m(x)$ by treating each $h_m$ as a step in function space. For squared-error regression, the step that most reduces $L$ is the one that fits the negative gradient of the loss with respect to the current predictions — and for squared error, that negative gradient is exactly the residual, $y_i - F_{m-1}(x_i)$. This is why the algorithm in the lab looks like "fit a tree to the leftover error" rather than anything you'd call gradient descent from a first course: it is gradient descent, taken in the infinite-dimensional space of functions, with a decision tree as the direction you're allowed to step in on each round.

That framing generalizes past squared error. Swap in the logistic loss and the "residual" a classifier's trees fit is $y_i - p_{m-1}(x_i)$, a probability residual, not a raw value difference — the mechanism doesn't change, only the gradient being fit does. This is the detail that separates understanding gradient boosting from having memorized one instantiation of it.

Why the weak learner has to stay weak #

Set tree depth to 1 in the lab and each round fits a stump: one split, two leaf values. That's a high-bias, low-variance model on its own — nowhere near enough to fit the sine-plus-jump function generating the data. It doesn't need to be. It only has to point the ensemble a little closer to the residual each round, and thirty rounds of "a little closer" compounds into a close fit.

Raise the depth to 2 and each tree gets more expressive: it can fit two splits' worth of local structure per round, so the ensemble converges faster — the training MSE curve drops more steeply in early rounds. Watch what that costs, though: the individual round's fit becomes less stable, more willing to bend around a handful of noisy points rather than the broad shape they're sampled from. Depth in a boosted tree isn't a knob you turn up for a better model; it's a bias–variance trade you're making explicit, round by round, and it interacts with the number of rounds you plan to run.

Shrinkage is a second, independent regularizer #

The learning rate here — often called shrinkage in the boosting literature to distinguish it from a neural network's — multiplies every tree's contribution before it's added to the ensemble. Set it to $1.0$ and each stump's fitted leaf value is applied in full; the ensemble can overfit the training residuals quickly, tracking noise as readily as signal. Drop it to $0.1$ and each round only closes a tenth of the gap it found, which means the ensemble needs more rounds to reach the same training error — and, more usefully, spreads that fitting across more trees, none of which gets to dominate the final function on its own.

This is the mechanism behind the standard advice to pair a low learning rate with a larger round budget: $\eta$ and $M$ aren't independent hyperparameters to tune separately so much as one knob split into two, where lower $\eta$ needs higher $M$ to reach the same fit, and gets a smoother, less variance-prone ensemble for the trouble. The lab's MSE-by-round chart makes the trade legible — compare $\eta=1.0$ at round 10 (train MSE around 0.21) against $\eta=0.3$ at round 30 (around 0.18) and you're looking at two ensembles that have reached close to the same training error with very different numbers of independent corrections behind them: ten large steps versus thirty smaller, more conservative ones.

What this lab doesn't show, and why that matters #

Everything above is measured on the training set the trees were fit to. A gradient boosting model's training MSE decreases monotonically, by construction, every round — that curve going down is not evidence the model is getting better at the thing you actually care about. The gap this leaves is exactly where held-out validation, early stopping on a separate validation loss, and subsampling (stochastic gradient boosting — fitting each round's tree to a random subset of rows) earn their place in a real training pipeline. None of the production libraries you'd reach for — XGBoost, LightGBM, CatBoost — ship without those controls, and none of them are optional extras; they're the difference between this lab's clean, monotone loss curve and what a boosted model's validation curve looks like once it starts fitting noise instead of signal.

The other simplification worth naming: this implementation splits on raw feature value with exhaustive threshold search, which is exactly right pedagogically and exactly what you would not do at scale. Production gradient boosting libraries bin continuous features into histograms before searching for splits, trading a small amount of split precision for an order-of-magnitude reduction in the search cost per node — a systems decision, not a statistical one, and a good example of how a library's defaults encode engineering constraints that a from-scratch implementation like this one is free to ignore.