Show HN: Super fast pipeline for finding specific paths across mills of domains

Category: cli-tool

Tags: dns, http-probing, security

Score: 6.5/10 (Innovation: 6, Technical: 7, Documentation: 7, Utility: 6)

Webcensus is a fast, pipelined tool for scanning millions of domains for specific file paths (e.g., security.txt) using a Rust-based HTTPS prober and massdns for DNS resolution. Its funnel-shaped architecture and resumable stages make it efficient on a single machine, and it publishes curated domain lists as datasets.

Target audience: security researchers, system administrators, devops

Repository: https://github.com/Kirill89/webcensus · Rust · MIT · 1 stars

View on Hacker News