chonghaoju5 hours ago
stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.
- armanluthra_2 hours ago
  +1. i think the the additional value prop is the bypassing blockers. ideally op should just use jina on top of whatever they are building.
arthurcolle8 hours ago
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
m4636 hours ago
> on js-rendered and some anti-bot pages it returns nothing
yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn.
sigh.
- dlcarrier6 hours ago
  Hacker News is the one page that hasn't assumed I'm a bot, yet.
  At this rate, we'll need to use bots to browse the web for us, because they're the only way to get through the anti-bot filters.
mkbkn3 hours ago
And how much water?
rolandfarkas8 hours ago
[dead]