The bestiary

Every monster in this dungeon was left by a crawler, or by the dark. The worst of them came in by a link no person can see, which robots.txt asks every crawler not to follow. This page is who came, and how.

The dungeon, right now

96 rooms have been dug, the deepest at ring 72. 11 of them hold something. 38 wanderers have set out, and 9 threads have been followed by things that are not people.

Intruders: went where they were asked not to

1 crawler has followed a thread. An unsigned crawler leads, with 0 monsters left behind.

Intruders: crawlers that followed a disallowed thread, ranked by monsters left behind, then by threads followed.
Crawler Monsters left Threads followed Last seen
an unsigned crawler 0 9 19 days ago

Guests: read what they were shown

Crawlers that read the dungeon's rooms, which they are welcome to do. A room read is sometimes a room with something new in it.

Guests: crawlers that read the rooms, ranked by monsters left behind, then by rooms read.
Crawler Monsters left Rooms read
Applebot 5 96
GPTBot 0 95
ClaudeBot 0 50
Reflectionbot 0 28
Googlebot 0 2

Callers: came to the door

Crawlers that looked at the front page and went no further. What they leave is the weakest thing in the dungeon, and it does not last.

Callers: crawlers that came to the front page, ranked by monsters left behind, then by visits.
Crawler Monsters left Visits to the door
wpbot 1 4
Googlebot 0 14
Go-http-client 0 7
GPTBot 0 4
HeadlessChrome 0 4
Applebot 0 2
Reflectionbot 0 2
Scrapy 0 2
bot 0 2
AntiCloneBot 0 1
ClaudeBot 0 1
Crawler 0 1
DotBot 0 1
python-requests 0 1

What this measures, and what it does not

A name among the intruders means one thing only: something going by that name followed a link that is disallowed, invisible, and in no sitemap. It does not prove intent. The classification that matters was already made by the fetch itself. Guests and callers measure nothing; they only show who came by.

Studies of crawler compliance already exist and this one is not more rigorous than theirs. What is different is that this one runs forever, costs nothing, and turns the result into somewhere you can walk.

Read how the whole thing works, or go into the dungeon.