I've partially solved this with separate databases for different steps in the data pipeline(s) and have even experimented with Clickhouse as a complete alternative, but I really like way too many things about duckdb to replace it.
[1]: If you care: https://skaldmaps.com/blog/2026/07/zip-codes-are-a-bad-spati...
And now with things like quack the sharp points around concurrency are relaxing, and it feels like the compromises of using it this way are disappearing.
I discovered DuckDB looking for a way to analyze Nginx access.log's and it's an amazing tool. I believe it should be a standard tool like ripgrep for devs.
Thanks to the team for everything!
Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?
- ETL pipelines running on K8s nodes. Using their streaming processing engine means I can run smaller pods/nodes if needed, for datasets that may have required large dataframe-like transformations that may have buffered a big dataset into memory previously.
- A CLI distributed to an internal team to do a postprocessing step on a large modeling dataset - to get it into a consumable format and upload it to a bucket as a .db file.
- A SvelteKit app that used the node duckdb bindings to attach to the .db on the bucket and explore the results through a suite of BI tools. These tables have millions of rows, and would be pretty heavy to store in PG. The DuckDB version works really, really well.
Excellent performance.
Does that make this account an alias as well?
I have used it with WASM for some web applications for web use. I have also used with locally for querying 100 gigs of data. And I have used it in the cloud as the serverless gold layer for Apache superset.
"If we use this we get indefinite RAM indefinite CPU and do not need to host a server".
I had an impression that DuckDB was not great at distributing work to other machines, but good at doing it locally? Am I wrong?
I made a dry run extension a few months ago (https://github.com/aleda145/duckdb-dryrun), will be so nice to build it just once and know that it will always work.
Also urge anyone to make an extension, the template makes it quite smooth: https://github.com/duckdb/extension-template
All that being said, I’m quite excited about DuckDB 2.0. I want to give the improved VARIANT support a try.
[0] Documentation on DuckDB’s native format is rather sparse AFAICT. But the DDL has nothing resembling an ordered table.
DuckDB is seriously an incredible utility.
What about the runtime size? I care this because I intend to run a stripped WASM version of DuckDB in browser.
Here is a wild idea: is it really so hard to edit out sentences structured and punctuated like this - it's so painfully obvious and distracts from the content. The effect is real.
Writing similar to this: is quite a normal way of writing for technical articles - especially when you are trying to make the point clear and well organized.
For me, this sentence pops out like a sore thumb
I do not enjoy dealing with text encodings and collations in databases.