This article is machine translated
Show original

We recommend upgrading your OpenClaw web scraping logic: Add the `Accept: text/markdown, text/html` header to all HTTP requests. The reason: Cloudflare has introduced the Markdown for Agents feature. Websites that have enabled this feature will directly return Markdown content when the AI agent makes requests, reducing token consumption by approximately 80% compared to HTML. Websites that do not support this feature will return HTML normally and will not be affected. Specific changes: 1. Locate all code in all Agents involved in web scraping (HTTP calls such as fetch, axios, and request). 2. Add the following to the request header: Accept: text/markdown, text/html 3. Add a check in the response processing: - If the content-type is text/markdown → use it directly, skipping HTML parsing. - If the content-type is text/html → follow the original HTML parsing logic. 4. If the response contains an x-markdown-tokens header, log it (for future token budget estimation). After making these changes, run a test and verify on a Cloudflare-hosted website to confirm that you can receive Markdown responses.

Cloudflare
@Cloudflare
Time to consider not just human visitors, but to treat agents as first-class citizens. Cloudflare’s network now supports real-time content conversion to Markdown at the source using content negotiation headers. https://cfl.re/4ksZQ1S
From Twitter
Disclaimer: The content above is only the author's opinion which does not represent any position of Followin, and is not intended as, and shall not be understood or construed as, investment advice from Followin.
Like
Add to Favorites
Comments