> It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know which image format would be usable on all UAs and there was no CSS or javascript to allow for client-side adaptation.
> ...The caching impact of proactive negotiation is far worse than the one extra round trip per site for reactive negotiation, and even that round-trip isn't necessary in formats that support client-side adaptation.
On the caching impact, Simon Willison wrote:
> ...you can’t deploy an application that uses content negotiation via the Accept header behind the Cloudflare CDN — for example serving JSON or HTML for the same URL depending on the incoming Accept header. If you do, Cloudflare may serve cached JSON to an HTML client or vice-versa.
Note: I posted this in another comment with links to those two quotes which I couldn't copy easily now - will add later.
Content negotiation still has its uses, but most of the time you’re better off using different endpoints.
Wrong: https://developers.cloudflare.com/cache/concepts/vary/
Simon wrote that in 2023: https://simonwillison.net/2023/Nov/20/cloudflare-does-not-co...
But it's something I think developers should think about if they rely on caching. If it took Cloudflare this long to support this, there may be other systems which still don't.
Link to Roy Fielding comment:
https://lists.w3.org/Archives/Public/ietf-http-wg/2013JanMar...
I take the point that it makes caching harder, but I don’t think that should overrule ergonomics concerns.
But for HTML and Markdown in particular, there's been so much useful work done in the semantic HTML space and microformats, that I don't know why anyone interested in this wouldn't just improve their HTML markup and leave it to the agent to do the rest. Convert to markdown or extract the useful HTML before handing it to model.
HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown.
We are not going to rewrite the entire web because harness developers are too lazy or stupid to pull an HTML to markdown package from npm. If some sites want to do so anyway, good for them, in many cases I’d love to skip the CSS/JS and read the markdown directly (or better, nicely formatted). But don’t blame website authors for your harness wasting your tokens.
The reason it may be more difficult today is because we've lost a lot of that. Some of it because of modern JS frameworks, but some also because publishers simply don't want to make it easy for the useful stuff to be scraped and extracted easily.
I'm not convinced that's changing because of AI agents (it's getting worse in many ways with anti-agent rules). Maybe improving for documentation pages intended for agents. But if it is changing, I think it'd be far easier to improve the HTML and let the agent take care of the rest.
I've been enjoying https://defuddle.md since learning about it. Works great. Not affiliated.
Most harnesses have a "zero config" plugin that works, eg: https://pi.dev/packages/pi-exa
(no affiliation to either)
I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.
It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now?
There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.
This is kind of the opposite of reality no? Markdown is just plain text and meant to be human readable. You don't need XML tags to read and write it, opposed to html where you do and you need a browser to properly view it.
> Why do you need AI then?
(from this HN comment, worth a read in full: https://news.ycombinator.com/item?id=43679585)
It's giving https://news.ycombinator.com/item?id=8863#9224
They’re far too useful for me.
A cherry.jpg image/jpeg file is shown as an image by the browser
A logo in svg like https://news.ycombinator.com/y18.svg too
So browsers should show readme.md as text/markdown and clicking on a link inside that file should also redirect to another markdown file, instant wiki
Browsers should implement that asap and we will follow
The MD content isn’t intended for human consumption whereas HTML is. So you either have that injection readable to your users, which could be jarring to non-technical readers, or you play the cat-and-mouse game of hoping those pesky crawlers don’t ignore text that’s not human-visible on a rendered page.
Or everything just disappears behind something like cloudflare or a paywall.
The open internet deserves a real solution instead of gatekeepers in the form of brightdata, firecrawl, cloudflare and the likes.
Maybe tarpit them into reading gigabytes of Markov Chain nonsense, really slowly?
