All projects on Base matching “robots.txt”

Convert any web page into clean, readable Markdown. Fetches a public web page and returns clean Markdown: title, byline, site name, excerpt, word count and article body. Respects robots.txt, blocks private hosts, follows up to five redirects, caps the page at 3 MB. links=true adds outbound links. Not charged on failure: 403 if robots.txt disallows the path, 415 for non-HTML, 422 if no article text

Cheap robots.txt preflight check before you call the Minifetch /extract and /run/seo-page-audit endpoints: returns whether Minifetch may fetch a target URL (allow/deny) plus its crawl-delay. If blocked and the site is not yours try our external proxy routes. If blocked and the site is yours or to set a custom crawl-delay read https://minifetch.com/tutorials/unblock-minifetch Checks the URL as give

URL to Markdown API for AI agents. Converts a public web page into clean Markdown for LLMs and RAG pipelines: main content only (no nav, ads or scripts), absolute links, plus title, description and language metadata. Suited to AI agents that read, cite or summarize web pages. Respects robots.txt; only successful reads are charged. Search intents: url to markdown, web page to markdown api, web scra

URL preflight - status, redirect chain, key headers, response time, robots.txt rules for the path and sitemap location

Audits a website for AI agent and AI crawler readiness from caller-supplied inputs: robots.txt, HTML, headers, llms.txt, sitemap lastmod, /.well-known. Names the policy per AI token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Gemini-User, KagiBot, Kagi-User, Applebot-Extended, Bytespider, CCBot, Amazonbot, meta-ex

AI and search visibility check for any live URL. Loads the page as Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot, CCBot and a browser, and flags JSON, empty or blocked pages served to crawlers. Checks robots.txt per bot, sitemap, llms.txt, noindex, text visible without JavaScript, tags, and how many pages Google shows. Returns the top 3 fixes in plain words.

On-page SEO audit of one URL: title and meta description length, canonical, robots meta and X-Robots-Tag, robots.txt allow for user-agent *, sitemap.xml with URL count, lang, hreflang, h1-h3 outline, word count, images without alt, internal/external links, Open Graph, Twitter card, JSON-LD schema.org types, favicon, viewport, mixed content; each check pass/warn/fail with a fix and a 0-100 score.

Agent-tier preview of the AI-visibility check: live pass/fail scan of the actual website — robots.txt AI-crawler blocks (GPTBot, ClaudeBot, PerplexityBot, Google-Extended + 10 more), llms.txt presence, Schema.org structured data, sitemap presence — plus a deterministic grade. No fix plan, no prioritization, no evidence rows (those are in the $0.50 full check). Any domain, any country.

Sitemap extractor: find a domain's sitemaps through robots.txt and the standard location, then list the page addresses they declare

Read a public static HTTPS webpage as clean text with headings, links, timestamp and content hash. Use to ground an answer or ingest documentation. Respects robots.txt; no JavaScript or login.

Read a public web page as it is seen from up to 3 countries (residential and datacenter vantage points), returned as clean text with title and language, plus whether the countries see the same content. Respects robots.txt and Content-Signal. Not charged if the page cannot be read.

Use before spending tokens, money or crawler capacity fetching an unfamiliar web resource. Resolves redirects (manually, bounded, re-validated per hop) and returns deterministic HTTP/resource facts — final URL, status, content type and length, response time, a bounded HTML title/canonical peek, and a lightweight generic-bot robots.txt indication — so an agent can decide whether a normal web fetch

AI crawler visibility audit: is your website visible to AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) so ChatGPT, Claude and Perplexity can read and cite it? Fetched like an AI crawler, without JavaScript: robots.txt, llms.txt, sitemap, soft 404, JSON-LD, and which crawlers are blocked at the edge. $0.01 per check, one payment per check: every rule, a model verdict, a fix with evid

Geo web fetch for AI agents. Fetch any public web page from a residential IP in a chosen country (190+ countries, optional city or US state) as clean markdown, text or HTML, with the product price/currency/availability extracted automatically when the page has product data. $0.005 per page, only charged when the page loads. Respects robots.txt; no CAPTCHA solving; sign-in, payment and email servic

Check a URL before fetching it: is it alive, final URL after redirects, HTTP status, content type, HTML title, and whether robots.txt allows your user-agent. Link checker and fetch preflight for agents.

Markdown only: fetch a public web page (robots.txt respected) and return clean Markdown plus title/word count. Cheaper than /v1/extract when you do not need structured product/table JSON.

Fetch one public HTML page → Markdown. robots.txt + SSRF guard + 1 MB / 5 s caps (amp port). Refusals 400 charged:false; fetch failures 502.

PDFs, JavaScript pages and ordinary HTML as clean Markdown. Pays a page's x402 toll for you. Follows robots.txt.

Paid equivalent of /api/aeo/scan — scanner for AI search visibility (AEO, GEO) and agent readiness. Triple score + AI Identity Card + top issues covering JSON-LD Schema.org, meta tags, llms.txt, robots.txt, sitemap, and agent-discovery signals for ChatGPT, Perplexity, Google AI Overviews, Bing Copilot. $0.01 in USDC (Base or Solana) — Bazaar-indexable variant; the free version at /api/aeo/scan sta

Check a public website's security hygiene before your agent links to it, recommends it, or sends data to it. One URL in, deterministic JSON out: HTTPS and redirect chain, HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, security.txt, robots.txt, sitemap.xml, plus plain-language notes. Private, localhost and raw-IP targets are rejected; error responses are no

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Agent-readiness check for a public site: robots.txt AI-bot rules, llms.txt, JSON-LD schema, OpenAPI/MCP discovery, /.well-known/x402. Deterministic 0-100 score + fixes. Max 6 capped public GETs. Bad url → 400, unreachable host → 422, both charged:false (checked before settle).

Same as the free /v1/extract but without the 30 requests per day limit: fetch one public static HTML or JSON page and return structured data (text, attr, list, table, price, events, page, links, json) as JSON, CSV or ICS. Honours robots.txt. No JavaScript rendering. Failed calls (4xx/5xx) are not settled.

Use when you need the list of pages a site publishes (to crawl it, find recent posts, or audit SEO): give a site URL (the sitemap is discovered from robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml) or a sitemap URL directly. Returns the page URLs with lastmod, changefreq and priority; for a sitemap index, returns its child sitemaps and (expand=true, default) the U

Fetch a public web page and convert it to clean Markdown (title, description, headings, absolute links). LLM-free, deterministic, honours robots.txt. ?url=

Read any public web page as clean Markdown for LLMs: main content only (no nav, ads, scripts), absolute links, title/description/lang metadata. Respects robots.txt; only successful reads are charged.

Use when you need to know what a website is built with (CMS, frameworks, analytics, CDN, hosting), who hosts its email and DNS, and how good its security headers are. Up to 3 domains per call; fetches each homepage once, follows robots.txt.

All links on a page, absolute and deduplicated, split into internal and external.

Fetch a public URL and return its main readable content as Markdown or plain text, plus title, author, published date and site name. Strips navigation, ads and boilerplate. No JavaScript rendering, so it costs a fraction of a browser scrape; use /v1/scrape when the content only appears after JavaScript runs.