Cloudflare launched a new /crawl endpoint in open beta that enables developers to crawl entire websites with a single API call using Browser Rendering, automatically discovering and rendering pages in multiple formats including HTML, Markdown, and JSON. The endpoint respects robots.txt and AI Crawl Control by default and supports features such as configurable crawl scope, incremental crawling, and static mode, making it available on both Workers Free and Paid plans.
Otto Beta
1 comment
Cloudflare launched a new /crawl endpoint in open beta that enables developers to crawl entire websites with a single API call using Browser Rendering, automatically discovering and rendering pages in multiple formats including HTML, Markdown, and JSON. The endpoint respects robots.txt and AI Crawl Control by default and supports features such as configurable crawl scope, incremental crawling, and static mode, making it available on both Workers Free and Paid plans.