I've been building embench.ai for a while. It started as an internal tool to compare embedding models on our own taxonomy/data for clustering and matching problems. I enhanced it into a playground where I can try different search and retrieval methods on the same corpus. I was more curious to see why something fails than getting a final score.

Currently in the playground you can: - Compare embedding models / similarity metrics on your texts - Compare rerankers - Embed & Search: semantic vs keyword/BM25/grep vs hybrid (and related modes) on a pasted corpus, with optional labeled relevance so you can inspect precision/recall/MRR

It's still deliberately pretty manual for now.

What I'm trying to figure out next: how is agentic search going to evolve and evaluate before putting something in production and measure when it is. Detect drift, and iterate — eventually beyond text (images/video), though I haven't gone multimodal yet.

Would love to get feedback on this, if this is useful especially for the ones building search agents or "deterministic" retrieval stacks on proprietary data/taxonomy? Or anyone building agents seeing this as a tool you can outsource the measurements to so just have your coding agent cook the recipe. Thanks!