Show HN: DemandMap – Memory Mapping S3 into Polars on macOS Without FUSE

Category: library

Tags: memory-mapping, s3, lazy-loading, polars, rust

Score: 6.0/10 (Innovation: 7, Technical: 8, Documentation: 4, Utility: 5)

DemandMap implements memory-mapped lazy loading of data from S3 using macOS userfaulting, enabling on-demand chunk downloads without FUSE. It cleverly exploits OS-level page fault callbacks to fetch only the needed data, achieving near-instant random access to large remote arrays. The project is interesting for its novel use of an esoteric OS API, but is currently macOS-only and very rough.

Target audience: data engineers

Repository: https://github.com/sonthonaxrk/demandmap · Rust · 4 stars

View on Hacker News