I’ve been building a search system for long form content (talks, interviews, books, audio) where the goal isn’t “find the right document,” but more precise retrieval.

On paper, it looked straightforward: embeddings, a vector DB, some metadata filters. In reality, the hardest problems weren’t model quality or infrastructure, but how the system behaves when users are vague, data is messy, and most constraints are inferred rather than explicitly stated.

Early versions tried to deeply “understand” the query up front, infer topics and constraints, then apply a tight SQL filter before doing any semantic retrieval. It performed well in demos and failed with real users. One incorrect assumption about topic, intent, or domain didn’t make results worse it made them disappear. Users do not debug search pipelines; they just leave.

The main unlock was separating retrieval from interpretation. Instead of deciding what exists before searching, the system always retrieves a broad candidate set and uses the interpretation layer to rank, cluster, and explain.

At a high level, the current behavior is:

Candidate retrieval always runs, even when confidence in the interpretation is low.

Inferred constraints (tags, speakers, domains) influence ranking and UI hints, not whether results are allowed to exist.

Hard filters are applied only when users explicitly ask for them (or through clear UI actions).

Ambiguous queries produce multiple ranked options or a clarification step, not an empty state.

The system is now less “certain” about its own understanding but dramatically more reliable, which paradoxically makes it feel more intelligent to people using it.

I’m sharing this because most semantic search discussions focus on models and benchmarks, but the sharpest failure modes I ran into were architectural and product level.

If you’ve shipped retrieval systems that had to survive real users especially hybrid SQL + vector stacks I’d love to hear what broke first for you and how you addressed it.