Show HN: Reame – a CPU inference server that gets faster as it runs

Category: ai-ml

Tags: llm-inference, cpu-optimized, speculative-decoding

Score: 7.5/10 (Innovation: 7, Technical: 8, Documentation: 8, Utility: 7)

Reame is an LLM inference server optimized for cheap CPU hardware, using persistent KV cache snapshots, n-gram-based drafting, and self-regulating speculative decoding to get faster over time on repetitive workloads. Its novel combination of techniques like Palimpsest, Il Suggeritore, and the Conclave makes cheap hardware a viable first-class citizen for narrow AI tasks.

Target audience: backend devs, data engineers, devops

Repository: https://github.com/swellweb/reame · C++ · MIT

View on Hacker News