If you haven’t already, try https://r.jina.ai for cleaning up web pages for agents. It’s pretty good, until you hit JS heavy pages.
I spin up tiny headless chrome into LLM pipeline for this case, give it a try at Unrender
Please share anything you have on how to make it better