Claw Compactor: Compress LLM Tokens 54% with Zero Dependencies

2026-03-18T13:46:41.000Z·2 min read

Open-source 14-stage compression pipeline achieves 54% average token reduction across code, JSON, logs, and agent conversations with zero LLM inference cost. Outperforms LLMLingua-2 by up to 88% at aggressive compression ratios.

Claw Compactor is an open-source LLM token compression engine that achieves an average 54% reduction in token usage across code, JSON, logs, and agent conversations — with zero LLM inference cost and zero dependencies.

The Fusion Pipeline

The engine chains 14 specialized compression stages through an immutable data flow architecture:

QuantumLock — KV-cache alignment
Cortex — Content type and language auto-detection (16 languages)
Photon — Base64 path stripping
RLE — Run-length encoding
SemanticDedup — SimHash-based deduplication
Ionizer — JSON statistical sampling with reversible storage
LogCrunch — Log folding
SearchCrunch — Result deduplication
DiffCrunch — Context-aware diff folding
StructuralCollapse — Import merging
Neurosyntax — AST compression via tree-sitter
Nexus — ML token classification
TokenOpt — Token format optimization
Abbrev — Natural language abbreviation (text only)

Performance Benchmarks

Content Type	Legacy	Claw Compactor	Improvement
Python source	7.3%	25.0%	3.4x
JSON (100 items)	12.6%	81.9%	6.5x
Build logs	5.5%	24.1%	4.4x
Agent conversation	5.7%	31.0%	5.4x
Git diff	6.2%	15.0%	2.4x
Search results	5.3%	40.7%	7.7x

vs LLMLingua-2

Compression Rate	Claw Compactor	LLMLingua-2	Delta
0.3 (aggressive)	65.3% preserved	34.6% preserved	+88.2%
0.5 (balanced)	72.3% preserved	57.0% preserved	+26.8%

Claw Compactor preserves significantly more semantic content at the same compression ratio, with zero LLM inference cost.

Key Design Principles

Immutable data flow — FusionContext is a frozen dataclass; every stage produces a new result
Gate-before-compress — Each stage has should_apply() that inspects context type and skips inapplicable stages at zero cost
Content-aware routing — Auto-detects content type and language for type-aware compression
Reversible compression — RewindStore enables LLM retrieval of any compressed section by marker ID

Installation

git clone https://github.com/open-compress/claw-compactor.git
cd claw-compactor
python3 scripts/mem_compress.py /path/to/workspace benchmark

Requirements: Python 3.9+. Optional: pip install tiktoken for exact token counts. 1,676 tests included.

Source: GitHub — open-compress/claw-compactor | Hacker News

↗ Original source

Comments0