Lab
Interactive pages from my experiments, to poke at the results yourself.
-
Rewiring trained transformers
Twelve open-weight models changed at inference, with no retraining: query, key and value vectors swapped inside attention, the MLP run before attention, and the layers run in a different order. Read what they generate, and what their residual streams show layer by layer.
-
GLU Gate Lab
34 feed-forward gates against SwiGLU: staircases, sine ripples, Conway’s Game of Life, sparse gates and energy budgets. See each gate’s shape and slope, a live layer of hidden units, and how each did in training.