Can AI assistants actually read a website? This server answers from the **Agent Web Index**: 45,000+ domains probed wit…
Can AI assistants actually read a website? This server answers from the **Agent Web Index**: 45,000+ domains probed with real HTTP requests — once as a browser, once as each published AI crawler user-agent (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and others) — so it can tell apart what `robots.txt` says from what the server actually does. **Tools** - `domain_readiness` — readiness score, per-crawler verdict and the edge vendor answering for a host. - `web_readiness` — the aggregate state of the measured web. - `edge_blocking` — refusal rate per CDN/WAF vendor, over domain×crawler pairs robots.txt allows. - `measure_domain` — measures any domain on demand: six real requests made while you wait. No key, no signup. Dataset is CC BY 4.0. Most AI-crawler blocking is not in robots.txt — it is an edge default nobody chose, and this is the data that shows it.
pfmcp-hq
mcp
free
Others in the same category, ranked by how often they are opened.
