ByteBell - AI for Engineering TeamsSkip to main content

Posts tagged with "NTK scaling"

2 posts found

Featured image for article: Position Encodings: How AI Knows Where Words Are (And Why It Breaks at Long Lengths)
Mar 31, 2026 position encoding RoPE rotary embeddings ALiBi NTK scaling YaRN

Position Encodings: How AI Knows Where Words Are (And Why It Breaks at Long Lengths)

Transformers have no inherent notion of order. RoPE encodes relative positions via rotation matrices. Here's the full math from sinusoidal to NTK-aware scaling.

Featured image for article: Rotary Position Embeddings: The Full Mathematical Derivation
Mar 31, 2026 RoPE rotary position embeddings mathematical derivation NTK scaling YaRN position encoding math

Rotary Position Embeddings: The Full Mathematical Derivation

RoPE is used by virtually every modern LLM. Here's the complete derivation from first principles, proof of the relative position property, and NTK-aware scaling.