All projects matching “inspect http headers”

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.

Sends a request to a web address you provide and reports back everything the server said in response: the status code, all the response headers, how many times it got redirected, and how long the whole thing took.

Domain OSINT, TLS Certificate & Security Headers Recon (/v1/domain-osint) — Inspects live IPv4/IPv6 DNS resolution, TLS certificate issuer/expiry, and HTTP security grade. (https://domain-osint.agentdiet.dev/v1/domain-osint)