ThalamusDB: Query text, tables, images, and audio(github.com)
53 pointsby itrummer6 days ago8 comments
  • ilaksh2 days ago
    Does this use CLIP or something to get embeddings for each image and normal text embeddings for the text fields, and then feed the top N results to a VLM (LLM) to select the best answer(s)?

    What's the advantage of this over using llamaindex?

    Although even asking that question I will be honest, the last thing I used llamaindex for, it seemed mostly everything had to be shoehorned in as using that library was a foregone conclusion, even though ChromaDB was doing just about all the work in the end because the built in test vector store that llamaindex has strangely bad performance with any scale.

    I do like how simple the llamaindex DocumentStore or whatever is where you can just point it at a directory. But it seems when using a specific vectordb you often can't do that.

    I guess the other thing people do is put everything in postgres. Do people use pgvector to store image embeddings?

    • itrummer2 days ago
      LlamaIndex relies heavily on RAG-style approaches, e.g., we're using items whose embedding vectors are close to the embedding vectors of the question (what you describe). RAG-style approaches work great if the answer depends only on a small part of the data, e.g., if the right answer can be extracted from a few top-N documents.

      It's less applicable if the answer cannot be extracted from a small data subset. E.g., you want to count the number of pictures showing red cars in your database (rather than retrieving a few pictures of red cars). Or, let's say you want to tag beach holiday pictures with all the people who appear in them. That's another scenario where you cannot easily work with RAG. ThalamusDB supports such scenarios, e.g., you could use the query below in ThalamusDB:

      SELECT H.pic FROM HolidayPictures H, ProfilePictures P as Tag WHERE NLFILTER(H.pic, 'this is a picture of the beach') AND NLJOIN(H.pic, P.pic, 'the same person appears in both pictures');

      ThalamusDB handles scenarios where the LLM has to look at large data sets and uses a few techniques to make that more efficient. E.g., see here (https://arxiv.org/abs/2510.08489) for the implementation of the semantic join algorithm.

      A few other things to consider:

      1) ThalamusDB supports SQL with semantic operators. Lay users may prefer the natural language query interfaces offered by other frameworks. But people who are familiar with SQL might prefer writing SQL-style queries for maximum precision.

      2) ThalamusDB offers various ways to restrict the per-query processing overheads, e.g., time and token limits. If the limit is reached, it actually returns a partial result (e.g., lower and upper bounds for query aggregates, subsets of result rows ...). Other frameworks do not return anything useful if query processing is interrupted before it's complete.

    • bobosha2 days ago
      We use a vector db (Qdrant) to store embeddings of images and text and built a search UI atop it.
      • ilaksh2 days ago
        Cool. And the other person implies that the queries can search across all rows if necessary? For example if all images have people and the question is which images have the same people in them. Or are you talking about a different project?
        • itrummer2 days ago
          I think the previous post refers to a different project. But yes: ThalamusDB can process all rows if necessary, including matching all images that have the same persons in them.
  • AmazingTurtle2 days ago
    You say it's a DB, given the execution time of up to 600s per query, I say: its an agent.
    • itrummer2 days ago
      Well, it definitely goes beyond a traditional DBMS, but yes :-) If processing the same amount of data via pure SQL versus SQL with LLM calls, it will be slower and more expensive when using LLMs. Note that 600s is just the default timeout, though. It's typically much faster (and you can set the timeout to whatever you like; ThalamusDB will return the best result approximation it can find until the timeout). More details in the documentation: https://itrummer.github.io/thalamusdb/thalamusdb.html
  • 6 days ago
    undefined
  • catlifeonmars2 days ago
    Dumb question: why is this its own DB vs being a Postgres extension (for example).
  • satisfice2 days ago
    How is it tested?
    • itrummer2 days ago
      We use mocking to replace actual LLM calls when testing for the correctness of the ThalamusDB code. In terms of performance benchmarking, we ran quite a few experiments measuring time, costs (fees for LLM calls), and result accuracy. The latter one is the hardest to evaluate since we need to compare the ThalamusDB results to the ground truth. Often, we used data sets from Kaggle that come with manual labels (e.g., camera trap pictures labeled with the animal species, then we can get ground truth for test queries that count the number of pictures showing specific animals).
      • satisfice2 days ago
        When someone claims that a system can search “approximately” or “semantically” that means there some sort of statistical behavior. There will be error. That error can be systematically characterized with enough data. But if it can’t or isn’t, then it’s a toy.

        A problem I have with LLMs and the way they are marketed is that are being treated as and offered as if they were toys.

        You’ve given a few tantalizing details, but what I would really admire is a link to full details about exactly what you did to collect sufficient evidence that this system can be trusted and in what ways it can be trusted.

        • itrummer2 days ago
          The approximation in ThalamusDB is relative to the best accuracy that can be achieved using the associated language models (LLMs). E.g., if ThalamusDB processes a subset of rows using LLMs, it can reason about possible results when applying LLMs to the remaining rows (taking into account all possible outcomes).

          In general, when using LLMs, there are no formal guarantees on output quality anymore (but the same applies when using, e.g., human crowd workers for comparable tasks like image classification etc.).

          Having said that, we did some experiments evaluating output accuracy for a prior version of ThalamusDB and the results are here: https://dl.acm.org/doi/pdf/10.1145/3654989 We will actually publish more results with the new version within the next few months as well. But, again, no formal guarantees.

          • satisficea day ago
            With humans we don’t need guarantees, because we have something called accountability and reputation. We also understand a lot about how and why humans make errors, and so human errors make sense to us.

            But LLMs routinely make errors that if made by a human would cause us to believe that human is utterly incompetent, acting in bad faith, or dangerously delusional. So we should never just shrug and say nobody’s perfect. I have to be responsible for what my product does.

            Thanks for the link!

  • petre2 days ago
    Seems like a good tool for police work.
  • tarwich3 days ago
    What a cool idea
  • cyanydeez2 days ago
    Bizarre coding solutions that reqhire OPENAI