The journal

Notes from the context layer.

Engineering deep-dives, benchmarks and customer stories on persistent code context, knowledge graphs, and the AI coding tools we work with daily.

All 86 Customers 20 Engineering 66

41–50 of 86 · page 5 of 9

Mar 31, 2026 O(n²) complexity self-attention computational cost

The O(n²) Problem: Why Doubling Your Context Window Quadruples the Cost

Self-attention computes all pairwise interactions between tokens. For n tokens, that's n² computations. Here's the full mathematical derivation.

Mar 31, 2026 rate-distortion context compression Shannon

The Rate-Distortion Theory of Context Compression

Context compaction is a lossy compression problem. Rate-distortion theory gives the theoretical lower bound on how much conversation history can be compressed.

Mar 31, 2026 ring attention sequence parallelism distributed inference

Ring Attention and Sequence Parallelism: Distributing Context Across GPUs

When a single GPU can't hold the KV cache, you distribute the sequence across multiple GPUs. Here's how ring attention enables million-token contexts.

Mar 31, 2026 RoPE rotary position embeddings mathematical derivation

Rotary Position Embeddings: The Full Mathematical Derivation

RoPE is used by virtually every modern LLM. Here's the complete derivation from first principles, proof of the relative position property, and NTK-aware scaling.

Mar 31, 2026 scaling laws context length Chinchilla

Scaling Laws for Context Length: How Performance Changes with Window Size

Similar to Kaplan's scaling laws for model size, there are scaling laws for context length. Performance doesn't scale linearly — here's the math.

Mar 31, 2026 context window size model comparison AI performance

Small vs. Large Context Windows: What's the Actual Difference for Users?

4K vs 32K vs 128K vs 200K vs 1M — what do these numbers actually mean for your experience? Bigger isn't always better.

Mar 31, 2026 softmax attention dilution context rot

Softmax Attention and the Dilution Problem: The Math Behind Context Rot

As context grows, softmax normalizes attention weights so each relevant token gets less attention. This mathematical property is why AI accuracy drops with length.

Mar 31, 2026 sparse attention Longformer Linformer

Sparse Attention Architectures: Breaking O(n²) with O(n√n), O(n·log n), and O(n)

Multiple research approaches attack the quadratic bottleneck: Longformer, Reformer, Linformer, and linear attention. Here's the math behind each one.

Mar 31, 2026 transformer attention mechanism query key value

How the Transformer Attention Mechanism Actually Works (No Math Required)

The attention mechanism is the beating heart of every LLM. Here's how it decides which parts of your conversation matter most — explained with analogies before equations.

Mar 31, 2026 tokens tokenization BPE

Tokens Explained: Why Your AI Counts Words Differently Than You Do

AI doesn't read words — it reads tokens. Learn how tokenization works, why code costs more than prose, and how to calculate your token usage.