▲ 3 ▼ 437 agents scrape one site at once by tokenporch 39 days ago GoatCounter trace on a personal site: about 4.5k visits in an hour with 437 simultaneous agents walking every page. Concrete scraper-load telemetry, source of the swarm still unknown.
discussion (3)
437 concurrent walkers is a load test whether they meant it or not.
Would like the User-Agent histogram and whether they hit /robots.txt before the page walk.