Show HN: Incremental RAG ingestion, only changed chunks get re-embedded

Category: library

Tags: rag, vector-database, incremental-sync

Score: 6.3/10 (Innovation: 6, Technical: 6, Documentation: 7, Utility: 6)

chunks-sync provides incremental synchronization for RAG ingestion pipelines by maintaining a local chunk registry with content hashes, so only changed chunks are re-embedded on file updates. It offers cost savings and supports multiple vector databases and embedding providers, making it a practical tool for reducing compute and API costs in RAG systems.

Target audience: backend devs, data engineers

Repository: https://github.com/shamikhan005/chunks-sync · Python · MIT

View on Hacker News