Mar 31, 2026 information theory Shannon channel capacityThe Information-Theoretic Limits of Context WindowsThere are fundamental limits on how much information a fixed-width attention mechanism can extract from n tokens. Here's the math from Shannon's channel capacity to attention bounds.
