These experiments study small English language models trained with limited data. They vary state width, depth, FFN width, feedback mechanisms, and learning rates. The central question is whether information already computed inside a model can be reused to improve later prediction within a small parameter and data budget.

The main series uses BabyLM 2026 Strict-Small data for causal next-token prediction. These are pretrained completion models. Older series with different data or evaluation conditions remain in the catalogue, in separate comparison cohorts.

A token is a unit into which text is split for the model. It can be a word fragment or punctuation rather than a whole word. A state vector is a sequence of numbers representing information at a position during processing. Weights are model parameters updated during training; activations are intermediate values that change with the input.

Learning rate (LR) controls the scale of training updates. Weight decay (WD) gradually shrinks weights. Larger values are not necessarily better. “Main matrices” and “other parameters” are experiment-specific groupings, not standard categories shared by every language model.

Query, Key, and Value describe roles in retrieval. A Query represents what is being sought, a Key supplies features to match against, and a Value contains information to retrieve. The model generates these as vectors. A matrix memory superposes Key–Value associations, so it does not preserve every past sentence losslessly.

A token is represented by a vector with a few hundred coordinates. Attention retrieves information from the current and earlier positions; an FFN transforms the resulting features. Repeating these operations forms the model layers. A state width of 320 means 320 coordinates, while FFN width 1120 means the intermediate FFN representation has 1120 coordinates.

VISTARC passes changes made by one layer into the next layer’s FFN. It also feeds statistics of the previous position’s candidate predictions into the next position. MoS3 combines three output probability distributions. The model details identify which components each experiment actually contains.

A later family adds F, a small matrix memory of differences between predictions and arriving tokens, read according to the current state. A shape such as 16×16 describes feature dimensions, not a history of sixteen tokens.

The D/C family explores a different division: D stores changes produced by Attention and FFN processing, while C stores candidate-prediction information. Released model #1 has neither the later F module nor D/C matrices. These families reuse information in materially different ways.

Start with the measurement guide, then read the chapters on seeds, width and learning rate, depth, and FFN ratio. A model number is a permanent identifier, not a rank. Opening it reveals components, recorded effective learning rates, and results by training age.

The articles distinguish observations from interpretations and retain unsuccessful experiments. The catalogue contains 302 permanent identities and 988 recorded evaluation points. Missing scores are not treated as zeros.