The journal

Notes from the context layer.

Engineering deep-dives, benchmarks and customer stories on persistent code context, knowledge graphs, and the AI coding tools we work with daily.

All 86 Customers 20 Engineering 66

31–40 of 86 · page 4 of 9

Mar 31, 2026 GPU memory memory wall VRAM requirements

The Quadratic Memory Wall: A Precise Analysis of GPU Memory Requirements per Context Length

Total GPU memory = model weights + KV cache + activations + workspace. Here's the exact formula to compute maximum context length for any GPU configuration.

Mar 31, 2026 Infini-Attention compressive memory infinite context

Infini-Attention and Compressive Memory: Unbounded Context with Bounded Memory

Google's Infini-Attention combines standard attention with a compressive memory that persists across segments — enabling theoretically infinite context at O(1) memory.

Mar 31, 2026 information theory Shannon channel capacity

The Information-Theoretic Limits of Context Windows

There are fundamental limits on how much information a fixed-width attention mechanism can extract from n tokens. Here's the math from Shannon's channel capacity to attention bounds.

Mar 31, 2026 KV cache VRAM calculation GPU memory

KV Cache Memory Math: Calculating Exactly How Much VRAM You Need

The exact formula for KV cache memory and worked examples for every major model architecture. Calculate your GPU requirements precisely.

Mar 31, 2026 KV cache GPU memory VRAM

The KV Cache: Why Your AI Needs So Much GPU Memory

During inference, the model stores Key and Value vectors for every token. This KV cache is often the biggest memory consumer. Here's the math behind it.

Mar 31, 2026 lost in the middle attention distribution prompt engineering

The 'Lost in the Middle' Problem: Why Your AI Ignores Information in the Center of Long Prompts

Research proved that LLMs attend strongly to the beginning and end of context but poorly to the middle. This is a mathematical property, not a bug.

Mar 31, 2026 MQA GQA multi-query attention

Multi-Query Attention and Grouped-Query Attention: Reducing KV Cache by 8× at the Architecture Level

Standard multi-head attention uses separate K and V for each head. MQA and GQA share them — reducing KV cache dramatically with minimal quality loss.

Mar 31, 2026 position encoding RoPE rotary embeddings

Position Encodings: How AI Knows Where Words Are (And Why It Breaks at Long Lengths)

Transformers have no inherent notion of order. RoPE encodes relative positions via rotation matrices. Here's the full math from sinusoidal to NTK-aware scaling.

Mar 31, 2026 prefill decode TTFT

Prefill vs. Decode: Why Long Context Makes Your AI Slow Before It Even Starts Talking

Before the AI generates its first word, it must process EVERY token in the context. Here's why time-to-first-token increases with context length.

Mar 31, 2026 prompt caching API optimization cache hit

Prompt Caching: How to Pay for Context Once Instead of Every Turn

Prompt caching saves 90% on repeated context. For coding agents carrying 20-40K tokens of system prompts, the savings are enormous.