Author here. This post goes through the architectures of twelve open-weight models, including gpt-oss-20b, Qwen3.6-35B-A3B, Gemma 4, Nemotron 3 Nano and LFM3, and presents each design through the cost it reduces: GQA and sliding windows for the KV cache, MoE for compute per token, Gated DeltaNet, Mamba-2 and short convolutions as replacements for most attention layers, and quantization and ternary weights for the size of each weight. The last section follows a request through prefill and decode, and how VRAM and RAM affect each stage. Happy to answer questions.