Show HN: A page that hides a sentence for AI and lets you check if it came back

Category: security

Tags: ai-crawlers, robots-txt, website-security

Score: 7.0/10 (Innovation: 7, Technical: 6, Documentation: 8, Utility: 7)

This is a curated, community-maintained list of AI crawler user-agent strings for use in robots.txt and server configs to block unwanted scraping by AI bots. It is interesting because it addresses the growing need for website owners to control access to their content by AI training crawlers, with support for multiple web servers and a structured contribution process.

Target audience: web developers, devops, site administrators

Repository: https://sinceyouarrived.world/taken/agents · Python · MIT · 3907 stars

View on Hacker News