Lab · from the stairs study
A transformer's feed-forward layer has a gate: a small function that decides how much of each hidden unit gets through. Most models today use SwiGLU. I trained a small model (10 layers, d_model 512) with 33 other gates: staircases meant to make the layer cheaper to quantize or skip, gates that borrow the rules of Conway's Game of Life, sparse gates that keep SiLU's gradient, and energy budgets on the layer's output. Five seeds each, 275 runs in all. On loss nothing beat SwiGLU; the best sparse gate came within about 0.01 of it with 77% of its outputs exactly zero.
Every gate in the study sits in the same spot of the feed-forward block. The layer computes two projections of the token, bends one of them with a gate, multiplies, and projects back. Pick a gate to see its shape, the slope training sees, and what it does to a live layer of hidden units.
64 hidden units of one layer. Each new token draws fresh pre-activations a and b; the selected gate turns them into g and h = g · u. Exact zeros and near-zero gates are what a sparse kernel could skip. Hover a unit for its numbers.
a and g (life_glu's gate lives in 0–1 and gets its own); h has its own. Gray bars are |g| < 0.05; a dot on the baseline is an exact zero. Shaded brackets mark 8-unit blocks with every gate near zero (the study counted 64-unit blocks across 1,344 units). Activations here are random draws, not the trained model's.This gate is pointwise: each unit sees only its own a. The life gates are different. They arrange the units on a ring (or, for conway2d_swiglu, a 2-D grid) and let each unit's gate depend on its 8 nearest neighbours, the way a cell in Conway's Game of Life looks at the 8 cells around it. Round 2 made the rule smooth; round 3 kept it exact and used it to scale SwiGLU; round 6 made it a layer's only nonlinearity, or used it only to scale gradients.