I can talk for literally hours about how good it is when you connect it to your company's Confluence or Jira or Slack or Google Drive or a ton of other things. At a scale of many tens of thousands of documents too.
Their team is awesome too and completely tuned into exactly what their users need. And that it's open source is the cherry on top. No secret in how your data is being used.
- An incredibly happy user looking forward to more from Onyx
The best way we’ve found to do this is to build a document index instead of relying on application native searches at query time. The document index is a hybrid index of keyword frequencies and vectors. The keyword component addresses issues like team specific terminology and the vector component allows for natural language queries and non-exact matching. Since all of the documents across the sources are processed prior to query time, inference is fast and all of the documents have already been mapped to an LLM friendly representation.
There are also other signals that we can take into account which are applied across all of the sources. For example, the time that a document was last updated is used to prioritize more recent documents. We also have models that run at indexing time to label documents and models that run at inference time to dynamically change the weights of the search function parameters.
When you talked about "document index is a hybrid index of keyword frequencies and vectors", I am a bit curious of how to get them. In pre-processing, do you have to use LLM / models go through documents to get keywords? What about vectors? Are you using embed model to generate them? Does that imply preprocess has to be done whenever there is new doc or any modification in existing doc? Would that be costly in time? Any spicy cook to make the preprocessing more efficient?
We checked the recall at 4K tokens (which was a pretty typical token limit of the previous generation of LLMs) and we were at over 94% recall for our 10K document set. We also added a lot of noise to it (Slack messages from public Slack workspaces) to get hundreds of thousands of documents but recall remained at over 90%.
So basically you can have it completely airgapped from the outside world, the only tough part is the local LLM but there are lots of options for that these days.
- What would you say is the agentic approach's special sauce over a typical RAG pipeline, ie query->multi-query generation->HyDE->vector search->bm25 search->RRF->rerank->evaluate->(retry|refuse|respond) that differentiates the approach?
- If a user has 20 services connected, how does the agent know how to call/search/traverse the information in the right order?
- Do you have any internal evals on how well the different model affect the overall quality of output, esp for a "deep search" type of task? I have model-picker fatigue.
- Do you plan to implement knowledge graphs in the future?
The agent part is the loop of running the LLM over RAG system and letting it decide which questions it wants to explore more (some similarities to retry|refuse|respond I guess?). We also have the model do CoT over its own results including over the subquestions it generates.
Essentially it is the deep research paradigm with some more parallelism and a document index backing it.
How does the agent traverse the information: there are index-free approaches where the LLM has to use the searches of the tools. This gives worse results than approaches that build a coherent index across sources. We use the latter approach. So the search occurs over our index which is a central place for all the knowledge across all connected tools.
Do you have any internal evals on how well the different model affect the overall quality of output, esp for a "deep search" type of task? I have model-picker fatigue: Yes, we have datasets that we use internally. It comprises of "company type data" rather than "web type" data (like short Slack messages, very technical design documents, etc.) comprising about 10K documents and 500 questions.
For which model to use: it was developed primarily against gpt-4o but we retuned the prompts to work with all the recent models like Claude 3.5, Gemini, Deepseek, etc.
Do you plan to implement knowledge graphs in the future? Yes! We're looking into customizing LLM based knowledge graphs like LightGraphRAG (inspired by, but not the same).
Would you ever extend your app to search the web or specialized databases for law, finance, science etc?
Danswer Show HN: https://news.ycombinator.com/item?id=36667374
Danswer Launch HN: https://news.ycombinator.com/item?id=39467413
Different apps have different permissions models, not everyone is allowed to see everything. Do you attempt to model this complexity at all or normalize it to some general permissions model?
For example, Google Drive docs have permissions like "global public", "domain public", "private" where "private" is shared with users and groups and there's also the document owner.
Slack has public channels, private channels, DMs, group DMs.
So we need to map these external objects and their external users/groups into a unified representation within Onyx.
Then there are additional challenges like rate limiting so we cannot poll at subsecond intervals.
The way that we do it is we have async jobs that check for object permission updates and group/user updates against the external sources at a configurable frequency (with defaults that depend on the external source type).
Of course, always failing-closed instead of failing-open and defaulting to least permissive.
edit: added clarification
For example,board exec asks senior exec a question about a particular product. The senior exec then has to fire off emails to say 5 managers, who might go down their tree to ICs, all the info is gathered and synthesised into a response.
Normally this response takes into account some angle the senior exec might have.
A lot of knowledge working tasks follow this pattern that I have somewhat simplified.
At rest, the data is stored in Postgres and Vespa (the hybrid index), both of which are part of the deployment so it's all local.
The part that typically goes external is the LLM but many teams also host local LLMs to use with Onyx. In either case, the LLM is not being finetuned, the knowledge relevant to the question is passed in as part of the user message.
We built Onyx with data security in mind so we're very proud of the way the data flows within the system. We made the system work well with models that can run without GPUs as well so our users can get good quality results even if deploying on a laptop.