—Sites checked

—Crawlers tracked

—Collecting since

Blocked Partially restricted Allowed

Loading live data…

What is and is not recorded Recorded: which crawler names appeared as blocked, partially restricted or allowed, and a running count of checks. Not recorded: the domain checked, your IP address, any cookie or identifier, or the contents of any robots.txt. The counters cannot be traced back to a person or a site.

Why this dataset exists

AI crawler access is a genuinely new question and nobody has good public data on it. Individual sites can check themselves, but until now there was no way to see the wider pattern: is blocking GPTBot common or rare? Do sites block AI crawlers more than traditional search crawlers?

The only way to answer that is to aggregate real checks, which is what this page does. It becomes more useful the more people use the tool — and it costs nothing to contribute to, because you get your own result either way.

How to read it

Each bar shows the proportion of checked sites where that crawler was blocked, partially restricted (allowed on some paths, denied on others), or allowed. The percentage on the right is the blocked share, which is the number most people care about.

A caveat worth stating plainly: this is a sample of sites that someone chose to check, not a random sample of the web. People often check sites they suspect have a problem, so blocking rates here are likely higher than across the internet generally. Treat it as directional rather than definitive.

ADD TO THE DATA

Check a site and contribute a data point

You get your own result immediately, and the aggregate above gets one row more accurate. No account, no email, nothing stored about you.

Run the free checker → Why AI crawler access matters

Results are shareable by link if you need to send one to a developer.