← All endpoints
api.strale.io x402 robots-txt-parse
https://api.strale.io/x402/robots-txt-parse
payable
Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, crawl delays, and whether all crawlers are blocked.
We sent a real
GET request to this endpoint and it
answered 402 with payment requirements we could parse.
- Price
- $0.02
- Network
- base
- Pay to
- 0x66D7C2F952362BFB24FD7F02a9beC9c754ea83bC
- Spec dialect
- v1 (declares x402Version 1)
- Transport
- JSON response body
- HTTP method
- GET
- Response latency
- 171 ms
- Payment options
- 1 usable of 1
- Last checked
- 2026-08-10T09:16:52.695Z (1d ago)
- Category
- finance-crypto
Re-check it yourself:
curl "https://api.the402.dev/validate?url=https%3A%2F%2Fapi.strale.io%2Fx402%2Frobots-txt-parse"