By day I work on DevRel at Tigris Data, and by night I’m a virtual anime person that makes software deployed by the United Nations. A side project of mine took off recently, a Web AI Firewall Utility named Anubis. Anubis blocks abusive scrapers from taking out web services by asserting that clients that claim to act like browsers do, in fact, act like browsers. As usage has grown (yes, the UN actually uses Anubis), we’ve gotten creative about detecting bots, including setting honeypots that give us more detailed data on their requests. These honeypot logs are sent to Tigris, and we analyze them using DuckDB to find patterns and improve bot detection.
In this article I'm going to be calling those abusive scrapers "AI scrapers", but to be clear I'm not really sure if they are for training generative AI or not. However, the worst offender has been Amazon's Alexa team so it's pretty easy to take those two points and draw a line between them.
