← All endpoints
api.agentstools.dev aeo crawlers
https://api.agentstools.dev/aeo/crawlers
payable
robots.txt AI-crawler access matrix: for the 2026 cohort of AI and answer-engine crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended and more) reports allowed, blocked or unlisted per bot, whether the default rule blocks everything, declared sitemaps and a score. One network operation.
We sent a real
GET request to this endpoint and it
answered 402 with payment requirements we could parse.
- Price
- $0.005
- Network
- base
- Pay to
- 0xF22e558a00D91Ee12A1F50C52186FecB8dDFf493
- Spec dialect
- v2 (declares x402Version 2)
- Transport
- PAYMENT-REQUIRED header
- HTTP method
- GET
- Response latency
- 166 ms
- Payment options
- 1 usable of 1
- Last checked
- 2026-08-10T09:11:44.677Z (1d ago)
- Category
- finance-crypto
Re-check it yourself:
curl "https://api.the402.dev/validate?url=https%3A%2F%2Fapi.agentstools.dev%2Faeo%2Fcrawlers"