What every developer needs to know about in-process databases(www.graphgeeks.org)
12 pointsby semihs9 months ago1 comment
  • semihs9 months ago
    In-process (aka embedded/embeddable) databases are not new. In fact SQLite is the most widely deployed database in the world. However, starting with DuckDB, there is a new set of in-process database systems, such as Kuzu and Lance. As a co-developer of Kuzu, I hear several frequently asked questions (some of which are misconceptions) about in-process databases.

    - What are their advantages/disadvantages compared to client-server databases? - Does in-process mean databases are in-memory/ephemeral? (NO!) - Can in-process databases handle only small amounts of data? (NO!) - What are some common use cases of in-process databases? - What if my application needs a server?

    I tried to answer some of these questions in a blog post with pointers to several other resources that articulate several of these points in more detail than I get into.

    I hope it's helpful to clarify some of these questions and help developers position in-process DBMSs against client-server ones.

    • emmanueloga_9 months ago
      The article suggests running Kuzu in a FastAPI frontend for network access. A caveat: production Python servers like Uvicorn [1] typically spawn multiple worker processes.

      A simple workaround is serving HTTP through a single process language like Go or JavaScript, since Kuzu has bindings for both. Other processes could access the database directly in read-only mode for analysis [2].

      For better DX, the ideal would be Kuzu implementing the Bolt protocol of Neo4J directly in the binary, handling single-writer and multi-reader coordination internally. Simpler alternative: port the code from [3] to C++ and add a `kuzu --server` option.

      --

      1: https://fastapi.tiangolo.com/deployment/server-workers/#mult...

      2: https://docs.kuzudb.com/concurrency/#scenario-2-multiple-pro...

      3: https://github.com/kuzudb/explorer/tree/master/src/server

      • semihs9 months ago
        Yes this makes sense and we plan to eventually do something along what you are suggesting. We also have a plan to have a built-in server/GUI, where users can directly launch a web-based explorer through our CLI by typing "kuzudb -ui".