The bestiary
Every monster in this dungeon was left by something automated following a link no person can see. The link is disallowed in robots.txt. This page is what happened anyway.
The dungeon, right now
40 rooms have been dug, the deepest at ring 4. 4 of them hold something. 59 wanderers have set out, and 72 threads have been followed by things that are not people.
Who has been here
1 crawler has followed a thread. an unsigned crawler leads, with 4 monsters left behind.
| Crawler | Monsters left | Threads followed | Last seen |
|---|---|---|---|
| an unsigned crawler | 4 | 72 | 24 minutes ago |
What this measures, and what it does not
A name on this list means one thing only: something identifying itself that way followed a link that is disallowed, invisible, and in no sitemap. It does not prove intent, and a user-agent is self-reported — anything can claim to be anything. The classification that matters was already made by the fetch itself.
Studies of crawler compliance already exist and this one is not more rigorous than theirs. What is different is that this one runs forever, costs nothing, and turns the result into somewhere you can walk.