← All endpoints
crawl-preflight.postnov01.workers.dev crawl-check
https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain
not probed
Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.
We did not probe this one. templated path, needs a real parameter to probe
- Price
- 20000 atomic
- Network
- not established
- Pay to
- -
- Spec dialect
- -
- Transport
- -
- HTTP method
- -
- Response latency
- -
- Payment options
- 0 usable of 0
- Last checked
- never (never)
- Category
- finance-crypto
Re-check it yourself:
curl "https://api.the402.dev/validate?url=https%3A%2F%2Fcrawl-preflight.postnov01.workers.dev%2Fcrawl-check%2F%3Adomain"