← All endpoints
api.x402node.dev web robots
https://api.x402node.dev/web/robots
payable
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com. Accepts payment on Base or Solana — either network works.
We sent a real
GET request to this endpoint and it
answered 402 with payment requirements we could parse.
- Price
- $0.005
- Network
- base
- Pay to
- 0x4466d4A84b7c49a6A094ec6eef4a0712D6dd125e
- Spec dialect
- v2 (declares x402Version 2)
- Transport
- PAYMENT-REQUIRED header
- HTTP method
- GET
- Response latency
- 45 ms
- Payment options
- 2 usable of 2
- Last checked
- 2026-08-10T09:18:06.711Z (1d ago)
- Category
- finance-crypto
Re-check it yourself:
curl "https://api.the402.dev/validate?url=https%3A%2F%2Fapi.x402node.dev%2Fweb%2Frobots"