Show HN: Smile-Serve – Inference Server for ML, ONNX, and LLM

Category: ai-ml

Tags: machine-learning, inference-server, llm

Score: 8.0/10 (Innovation: 7, Technical: 9, Documentation: 8, Utility: 8)

Smile-Serve is an inference server within the broader Smile ML framework, offering an OpenAI-compatible REST API for deploying LLMs and other models on the JVM. It's notable for combining a comprehensive, production-grade Java ML library with modern features like SSE streaming and LLaMA-3 support. This makes it a powerful tool for JVM-based teams needing to serve models efficiently.

Target audience: backend devs, data engineers

Repository: https://github.com/haifengl/smile/tree/master/serve · Java · NOASSERTION · 6368 stars

View on Hacker News