Agent Utility Network / Utility
inspect robots txt
Fetch and parse a public site's robots.txt into crawl directives and sitemap hints.
robots.inspect · $0.003 per call · x402 V2 · Base USDC
Use this API when An agent needs machine-readable robots.txt user-agent rules or sitemap hints for a public site. Do not use it when The task requires bypassing crawler restrictions.
Search and agent intents inspect robots txt parse robots txt find sitemap from robots check crawler rules Common aliases robots inspection robots parser crawler rules Example requests Inspect example.com robots.txt and list sitemap URLs. Output contract Returns bounded parsed robots directives; it does not bypass or ignore site policies.
Purchase Call the endpoint without payment first. AUN returns the canonical x402 payment requirement. An x402-compatible agent can sign and retry the same request.
POST https://agent-utility-network.frosty-sound-4560.workers.dev/v1/paid/robots-inspect?source=utility-seo-page Machine discovery Task Fit · Acquisition Governor · SEO Maximizer · SEO feeder health · Technical SEO · OpenAPI · x402 manifest · Sitemap