All projects matching “web content”

Extract clean structured content from a public web page

Extract web content from any page or PDF as clean Markdown — web page text, HTML to Markdown for any URL: strips scripts, nav, ads, and boilerplate while preserving headings, links, lists, tables, code blocks, and blockquotes; extracts the text layer from PDFs. Returns Markdown body, title, word count, and a quality grade. Web reader for article and main content. For JS-rendered or bot-walled page

Web content extraction: fetch any public URL and get clean readable text, title, meta description, OpenGraph data, headings, links and word count as JSON. Built for AI agents that need web page content without HTML noise.

Clean web content extraction — any URL to markdown + text + metadata

Web content extraction and search. Get clean markdown from any URL or search results.

Page to JSON: scrape and extract any public web page URL into clean structured data JSON. Web scraper / HTML-to-JSON extractor / readability parser for AI agents and LLMs: returns title, author, site name, published date, main article text (plain text), word count, up to 200 links and an optional short summary. Web scraping, content extraction, metadata extraction, article parsing, crawl, RAG inge

Market & brand sentiment for any keyword, brand, ticker or product: analyses the sentiment of web & news content mentioning it and returns an overall label (positive/negative/neutral/mixed), a net sentiment score (-1..1), positive/negative/neutral shares and an emotion breakdown (anger, happiness, love, sadness, fun).

Extract structured data from a URL or raw text into strict JSON matching a schema you provide. Fields the content doesn't support come back null - never an invented or approximate value, and an unmatchable schema returns an explicit error. Try GET /extract/sample. Part of the AgentIndex content kit (pdf, web-read, extract, summarize, detect-language) - see GET /capabilities.

Render a web page and return its content as markdown

Render a web page and return its content as markdown

You.com Search API: real-time web search returning ranked results grouped by section (results.web, results.news), each with url, title and description; web results also carry snippets. Use when an agent needs fresh, citable web data (news, markets, company facts) to ground a decision. Query params: query (required), count 1-100 (default 10), livecrawl (web|news|all; fetches live page contents, cha

Fetch any public web page and return its main readable content as clean Markdown or plain text plus metadata (title), stripping nav/ads/boilerplate (trafilatura). Optional JS rendering via headless browser for SPAs; markdown or text output. Robots-respecting, public content only — the reader/scraper building block for RAG and agent pipelines.

Read any public web page as clean Markdown: main-content extraction (navigation, footers, scripts and hidden elements removed), headings, lists, tables, code blocks, links (absolute, deduplicated), images, plus title, description, canonical URL, language, author and publish date, and a content hash + fetch time as evidence. JSON and plain-text URLs are returned as-is. No JavaScript rendering. $0.0

Fetches a public web page URL and returns its main article content as clean Markdown with title, byline and excerpt; strips navigation, ads and scripts. Useful for RAG ingestion and LLM context.

Read any public web page as clean Markdown for LLMs: main content only (no nav, ads, scripts), absolute links, title/description/lang metadata. Respects robots.txt; only successful reads are charged.

Get the full text of one web page. Give a URL; returns the page's own main content as clean markdown (up to 8,000 characters) with navigation and ads removed, unchanged and not summarized (use /summarize for a short version). Typical response 2 to 5 seconds. If the page cannot be read you get an error and are not charged.

Web scraping proxy API - send a URL, get clean markdown or HTML content back. Supports dynamic rendering.

Safe Read: fetch one public web page and get its clean main text, title, metadata, a sha256 content hash, and prompt-injection warnings with a 0-100 risk score (visible text, hidden elements, HTML comments, alt/meta attributes, invisible Unicode, base64). SSRF-guarded: DNS-checked and pinned public addresses only, 4 redirects max, 2 MB, 8 s, text/html and text/plain only. Errors are not charged.

Fetch a public web page and return clean LLM-ready Markdown (Readability main-content extraction, nav/scripts stripped) plus title, byline, excerpt and OpenGraph metadata. {fullPage:true} keeps the whole body; {maxChars} caps output (default 100k). No JS rendering.

Web page to markdown (read URL as clean text) for LLM agents: main content only (no nav, cookie banners, ads), with title, description, author, dates, language, headings, internal/external links and JSON-LD types, PLUS a prompt-injection scan of the page (visible text, hidden HTML, comments, invisible Unicode) so your agent knows whether the content tries to give it orders. Fetched from Cloudflare

Scrape website page to Markdown. Fetch any public web page URL and get clean, LLM-ready text or Markdown: main content, code blocks, tables, links and headings, for RAG and agent context. Static HTML, Markdown or text; optional section ID. Default 20,000 output characters, max 100,000; input 2 MiB. No JavaScript rendering (see /extract/rendered), logins or paywalls. Empty pages, bad sections, brow

Fetch one public web page now and return clean markdown (or plain text) of the main content plus title, description, author, published time, canonical, language and sha256. Charged only on success.

Web page to Markdown, $0.001 USDC per call: Fetch a public web page and return LLM-ready Markdown: title, meta description, canonical, language, main content (scripts/nav/footer stripped, headings/lists/links kept), first 50 absolute links, word count. Honours robots.txt (disallowed → 403, not charged). No JS rendering; non-HTML → 415. max_chars 500-50000 (default 20000). No params = runs the exam

AI Web Extractor — turn ANY url into clean, LLM-ready markdown. POST {url}; get the main content as markdown (boilerplate/nav/ads stripped) with title, description, and links. The scrape / read / url-to-markdown primitive for RAG and research agents. Fast, cheap, x402-native.

Use before spending tokens, money or crawler capacity fetching an unfamiliar web resource. Resolves redirects (manually, bounded, re-validated per hop) and returns deterministic HTTP/resource facts — final URL, status, content type and length, response time, a bounded HTML title/canonical peek, and a lightweight generic-bot robots.txt indication — so an agent can decide whether a normal web fetch

Fetches web pages that return 403 Forbidden or HTTP 402 Payment Required. Handles Cloudflare bot protection and TollBit paywalls. Returns page content with receipt.

SWARM Read Page - Retrieve the content of a URL as clean, LLM-ready text (headings with #, lists with -; menus, scripts, styles and markup removed). Use when an agent needs to read a web page, article or docs page without paying model tokens for raw HTML: on 15 public pages we measured, the text was a median 97% smaller than the HTML (about 67,000 fewer tokens per page). Requires `url`; HTML or te

Read any public web page as clean JSON: title, description, site name, publish date, language, readable text (boilerplate removed), Markdown and the page's links. Give it a URL, get the content back in one call. HTML, plain text, JSON and XML pages supported.

Use to extract the main article or document text from a single-page URL. Not for homepages or hubs (Web Index Cleaner) and not a link inventory (Link Extractor). Server HTML only; JavaScript is not run. Output is text or markdown. If the requested extract has no main content, the other profile is tried once in the same request. The HTTP body has correlation_id and data. The item array is data.resu

Public sources for agent research: one You.com search plus fetch of up to three public HTTPS pages. Returns structured JSON with page text, metadata, source URLs, fetch timestamps and content hashes for source checking and company research. 0.01 USDC on Base per call via x402 v2; no signup. No browser rendering or synthesized answer. Generate a private request_id locally and reuse it on pending re