Hypura: Storage-Aware LLM Inference Scheduler Optimizes Performance on Apple Silicon

Hypura Brings Tiered Storage Optimization to LLM Inference on Apple Silicon

A new open source project called Hypura introduces a storage-tier-aware scheduler for running LLM inference on Apple Silicon Macs. The tool optimizes how model data moves between RAM and storage during inference, addressing a key bottleneck for running large models on memory-constrained devices.

The Problem

Running large language models on Apple Silicon is popular but challenging:

• Unified memory limits — Even high-end M-series Macs have 128GB or 192GB of unified memory
• Model sizes growing — 70B+ parameter models often exceed available RAM
• Storage speed matters — When models must be offloaded to SSD, I/O speed becomes the bottleneck

How Hypura Works

Hypura adds intelligence to the inference pipeline:

• Storage tier awareness — Understands the performance characteristics of RAM vs. NVMe SSD vs. slower storage
• Intelligent prefetching — Predicts which model layers will be needed and loads them proactively
• Layer scheduling — Optimizes the order and timing of layer loading from storage
• Zero-copy operations — Minimizes data copying between memory tiers

Why Apple Silicon

Apple Silicon's unified memory architecture is both a strength and a constraint. While the memory bandwidth is excellent (400 GB/s on M3 Max), the total capacity is fixed at purchase time. Hypura maximizes the effective model size that can run on any given Mac configuration.

Impact

• Run larger models on existing hardware without upgrades
• Reduce inference latency for partially-offloaded models
• Extend the useful life of older Apple Silicon Macs for AI workloads
• Complement tools like llama.cpp and MLX for the Apple AI ecosystem

The project is available on GitHub as t8/hypura.