Show HN: Ril: a parallel data streaming tool for Python

Category: cli-tool

Tags: python, parallel-computing, data-streaming

Score: 6.8/10 (Innovation: 6, Technical: 7, Documentation: 8, Utility: 6)

Ril is a Rust-based CLI tool that parallelizes CPU-bound Python data processing by streaming Arrow RecordBatches across worker processes, providing near-multiprocessing.Pool speed with automatic memory streaming and simple pipeline configuration. Its approach of combining Unix pipes, automatic worker allocation, and zero-infrastructure parallelism makes it interesting for ad-hoc large data jobs.

Target audience: data engineers, scientists, backend devs

Repository: https://github.com/dannypesic/ril · Rust · MIT · 3 stars

View on Hacker News