← All endpoints
x402.forgemesh.io robots-txt-ai-check
https://x402.forgemesh.io/robots-txt-ai-check
payable
Fetches a domain's robots.txt and parses its Content-Signal and AI-preference directives (search, ai-input, ai-train) into structured JSON plus a plain-English summary of what's allowed — the "can this domain be crawled by an AI agent" check. Run before scraping content or building a RAG index, ahead of the industry's move toward stricter bot-gating defaults later this year.
We sent a real
POST request to this endpoint and it
answered 402 with payment requirements we could parse.
- Price
- $0.005
- Network
- base
- Pay to
- 0x850363a27F0aC6fEb9C7a3eC4C1d295262dF9432
- Spec dialect
- v2 (declares x402Version 2)
- Transport
- PAYMENT-REQUIRED header
- HTTP method
- POST
- Response latency
- 40 ms
- Payment options
- 1 usable of 1
- Last checked
- 2026-08-10T09:16:22.965Z (4d ago)
- Category
- finance-crypto
Re-check it yourself:
curl "https://api.the402.dev/validate?url=https%3A%2F%2Fx402.forgemesh.io%2Frobots-txt-ai-check"