Show HN: High-performance parallel save/load for large NumPy

Category: library

Tags: numpy, parallel-i/o, shared-memory

Score: 5.8/10 (Innovation: 5, Technical: 6, Documentation: 6, Utility: 6)

Parallel Saver is a lightweight Python utility that speeds up saving and loading large collections of NumPy arrays by using shared memory and multiprocessing to parallelize I/O. It offers both pickle-based and .npz archive storage options, which is interesting for data-heavy workflows where sequential I/O is a bottleneck.

Target audience: data engineers

Repository: https://github.com/NoteDance/parallel-saver · Python · Apache-2.0 · 1 stars

View on Hacker News