Keel: Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
Listen to episode
About this episode
The January 27, 2026 ByteDance paper "Post-LayerNorm Is Back: Stable, ExpressivE, and Deep" introduces th Keel architecture which addresses the optimization instability of deep Transformers by reviving the Post-LayerNorm formulation, which theoretically offers better expressivity than the standard Pre-LayerNorm but historically fails to train at scale due to gradient vanishing. By replacing the standard ResNet->
The Keel paper argues that extending context length is limited to expanding information access rather than improving "fundamental expressivity," asserting that while longer contexts allow models to process more data, they do not inherently unlock the capacity for complex hierarchical reasoning. Whereas Gemini 1.5 demonstrates that long context drives In-Context Learning (ICL) and reduces predictive uncertainty (NLL) via retrieval-heavy tasks like learning a language from a manual, Keel contends that this does not equate to the "qualitatively new behaviors" unlocked by depth scaling. Consequently, Keel frames depth as the superior, albeit historically unstable, axis for improving model reasoning (particularly in math and code), contrasting the "diminishing returns" and high cost of context scaling against the robust expressivity gains achieved by stabilizing deeper networks.
Sources:
Keel:
https://arxiv.org/pdf/2601.19895
The Gemini 1.5 paper with its power law:
https://arxiv.org/pdf/2403.05530
More AI podcast episodes
Browse all →Want to find AI jobs?
Join thousands of AI professionals finding their next opportunity