Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality
Not provided in the content
Abstract
The paper investigates architectural differences in Transformers and linear models during pretraining, revealing that Canon layers improve performance while challenging traditional benchmark-based architecture research.
Reality Card
Canon layers consistently improve Transformers, saving about 30% of pretraining tokens for the same MMLU accuracy at 8B parameters.
At 1-2T tokens, Canon layers improve performance while linear models can match or exceed Transformers on benchmarks but struggle with basic in-context skills.
High scores on standard benchmarks can obscure weaknesses in basic in-context skills, and no model reliably beats random guessing on simple two-hop questions.
Paper to code
Verified implementation resources so builders can test the paper’s claims instead of stopping at the abstract.