Crawl Pre-Flight
crawl-preflight.postnov01.workers.dev
Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.
- Category
- Service
- Networks
- Base
- Starting price
- $0.02
- Paid calls, 30 days
- 1
Paid endpoints
1 endpoint, sorted by usage. Prices are per request.
| Endpoint | Price | Network | Facilitator | Calls / 30d |
|---|---|---|---|---|
GET/crawl-check/:domain Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain. | $0.02 | Base | Coinbase CDP | 1 |
Is this your project? Suggest an edit.