3 posts carrying this tag.
At a conservative 1,000 tokens per source file, 10 million files is roughly 10 billion tokens. A 1 million token window holds 0.01% of that, and a 10 million token window holds 0.1%. Closing that by scaling needs something like a 10,000x increase rather than a 10x one, and quality per token moves against you while you try.
Repomix is the leader in context packing, with 26,000 stars and 255,000 npm downloads a month, and its tree sitter compression genuinely cuts token count by about 70%. Apply that to 10 million files and you go from three orders of magnitude short of the context window to slightly less than three orders of magnitude short. Every tool in this category publishes a token reduction number. None of them publishes a correctness number.
Compliance with an agent's own explicit constraints falls from 73% at turn 5 to 33% at turn 16 across 4,416 measured trials, and Claude Code auto compaction reduces 132,000 tokens of session state to roughly 2,300. Shorter sessions and cleaner instruction files treat the symptom. The window fills with low value tokens because the agent rebuilds an understanding of your codebase from raw source on every task.
