Hi HN, I'm Isaac. This is Duckstring, an open source data engineering platform, using DuckDB as the engine. I've built it to address some of the main frustrations I've had in my years as a data engineer: modularising logic so that it can grow in complexity over time, methods for upgrading components without breaking everything, and implying the DAG through dependencies so that I don't have to manage it.

I'm convinced DuckDB is the only engine needed to cover most real-world data pipelines. I've tested it on data scales up to terabytes, and running on appropriately-sized single machines, it's consistently beaten distributed engines like Spark. I think the age of distributed computing for most pipelines might be waning.

You can see the benchmark results for some tests here, where at 1 TB DuckDB 2.0 ran 10 times faster than (untuned) Spark: https://duckstring.com/blog/duckdb-pipeline-engine

Some specific features of Duckstring:

- Data transformations are treated like software packages, allowing the DAG to be implied by dependencies

- A pull-based orchestration system, throttling every process both upstream and downstream of the bottleneck so that nothing runs if it can't be consumed

- An incremental framework based on DBSP that handles updates and deletes efficiently

- CLI-first, so that you can point your own AI agents at the platform rather than relying on a janky integration

The link is to an intro story, which passes on to a Playground app to help get a feel for the orchestration method. Link if you want that directly: https://playground.duckstring.com

You can also just go straight to the site or the project on GitHub: https://duckstring.com https://github.com/duckstring-dev/duckstring

It's pre-1.0, though has been in active development since January, and is ready to try on real pipelines. I'd love to hear your feedback!