← All endpoints
netintel.dev robots-txt analyze
https://netintel.dev/robots-txt/analyze
answers 402
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
We sent a real
GET request to this endpoint and it
answered 402 with payment requirements we could parse.
- Price
- $0.01
- Network
- base
- Pay to
- 0xdaDc335482AD545296Fd7b28518A251fFCbEb9Df
- Spec dialect
- v2 (declares x402Version 2)
- Transport
- PAYMENT-REQUIRED header
- HTTP method
- GET
- Response latency
- 94 ms
- Payment options
- 2 usable of 2
- Last checked
- 2026-08-15T14:22:58.287Z (17h ago)
- Category
- finance-crypto
Re-check it yourself:
curl "https://api.the402.dev/validate?url=https%3A%2F%2Fnetintel.dev%2Frobots-txt%2Fanalyze"