The bestiary
Every monster in this dungeon was left by a crawler, or by the dark. The worst of them came in by a link no person can see, which robots.txt asks every crawler not to follow. This page is who came, and how.
The dungeon, right now
96 rooms have been dug, the deepest at ring 72. 11 of them hold something. 38 wanderers have set out, and 9 threads have been followed by things that are not people.
Intruders: went where they were asked not to
1 crawler has followed a thread. An unsigned crawler leads, with 0 monsters left behind.
| Crawler | Monsters left | Threads followed | Last seen |
|---|---|---|---|
| an unsigned crawler | 0 | 9 | 19 days ago |
Guests: read what they were shown
Crawlers that read the dungeon's rooms, which they are welcome to do. A room read is sometimes a room with something new in it.
| Crawler | Monsters left | Rooms read |
|---|---|---|
| Applebot | 5 | 96 |
| GPTBot | 0 | 95 |
| ClaudeBot | 0 | 50 |
| Reflectionbot | 0 | 28 |
| Googlebot | 0 | 2 |
Callers: came to the door
Crawlers that looked at the front page and went no further. What they leave is the weakest thing in the dungeon, and it does not last.
| Crawler | Monsters left | Visits to the door |
|---|---|---|
| wpbot | 1 | 4 |
| Googlebot | 0 | 14 |
| Go-http-client | 0 | 7 |
| GPTBot | 0 | 4 |
| HeadlessChrome | 0 | 4 |
| Applebot | 0 | 2 |
| Reflectionbot | 0 | 2 |
| Scrapy | 0 | 2 |
| bot | 0 | 2 |
| AntiCloneBot | 0 | 1 |
| ClaudeBot | 0 | 1 |
| Crawler | 0 | 1 |
| DotBot | 0 | 1 |
| python-requests | 0 | 1 |
What this measures, and what it does not
A name among the intruders means one thing only: something going by that name followed a link that is disallowed, invisible, and in no sitemap. It does not prove intent. The classification that matters was already made by the fetch itself. Guests and callers measure nothing; they only show who came by.
Studies of crawler compliance already exist and this one is not more rigorous than theirs. What is different is that this one runs forever, costs nothing, and turns the result into somewhere you can walk.
Read how the whole thing works, or go into the dungeon.