When training loss stops improving, “make the network bigger” is one possible intervention. It is not yet a diagnosis. The model might lack the right function class, the optimizer might be taking poor steps, or the training objective might be rewarding a pattern that will not survive deployment.
The lab separates those explanations with a controlled problem. It uses 80 training points and 80 held-out points, generated from fixed seeds. You can choose a logistic classifier or a small tanh network, introduce noise into training labels and inspect both losses as the boundary changes.
First ask whether the function exists #
A logistic classifier computes . Its probability varies smoothly, but its 0.5 decision boundary is a line. XOR requires opposite quadrants to share a label, so a single line cannot represent the whole rule.
Train the linear model on XOR for 1,000 steps. The loss settles and the classifier can still score above chance on this finite sample. That does not mean it discovered XOR; it means a line can exploit an imbalance in the particular points it saw. The decision field makes that distinction visible.
Switch to the hidden layer and repeat. The model now computes
The nonlinear features can divide the plane into regions. With the default eight hidden units and fixed initialization, the network fits this example well. The change in architecture expands the available functions; additional optimizer steps alone could not do that for the linear model.
Inspect the objective before its curve #
Training uses full-batch gradient descent on binary cross-entropy, with an optional L2 penalty:
Biases are excluded from the penalty. The displayed training and held-out losses show cross-entropy alone, so their scales remain comparable when regularization changes. One “step” is one update from the entire training set, not one example or one arbitrarily named animation frame.
Try a smaller learning rate before attributing a slow curve to insufficient capacity. A large rate can also produce oscillation or saturation. Because this experiment uses deterministic full batches, changes come from the controls rather than minibatch sampling. That makes diagnosis easier, though it omits an important source of noise and implicit regularization in larger training runs.
A clean fit can become the wrong target #
Set training-label noise to 0.30. The held-out labels remain clean. Compare a narrow hidden layer with a wider one, then introduce a modest L2 penalty and train again from the same initialization.
Noise creates a disagreement between the finite training objective and the underlying rule. Extra capacity can spend effort on mislabeled points. A widening gap between training and held-out loss is evidence of that problem, but do not demand that every short run exhibit a perfectly shaped overfitting curve. Optimization, sample placement and model width all affect when the gap becomes visible.
Accuracy discards confidence. Cross-entropy can worsen while accuracy stays fixed if a model becomes more confidently wrong on a few points. That is why the lab shows both. Neither measure alone establishes probability calibration; that would require checking predicted probabilities against observed frequencies on sufficient independent data.
The held-out set eventually becomes part of development #
Once you repeatedly inspect a held-out curve to choose width, learning rate, regularization or stopping time, that set is functioning as validation data. It is no longer an untouched estimate of final performance. A separate test set, repeated splits or an appropriate temporal evaluation is needed for a stronger claim.
The distinction becomes more consequential when examples share users, documents or time periods. A random split can put nearly identical observations on both sides and make generalization look solved. The lab’s synthetic examples have none of those dependencies. Use it to reason about capacity and optimization; evaluate a deployed classifier under the dependencies and distribution shifts it will actually encounter.
Further reading #
TensorFlow Playground is the reference for inspecting features, architecture and training together. Goodfellow, Bengio and Courville’s Deep Learning develops the optimization and generalization arguments beyond the small network used here.