Postgres rewritten in Rust v0.2, now faster than Postgres and ClickHouse(github.com)

15 pointsby malisper4 hours ago5 comments

westurner3 hours ago
Would there be an advantage to reusing SIMD-optimized libraries for this?
Polars Rust is built on Arrow and packed_simd.
pola-rs/polars: https://github.com/pola-rs/polars
polars - Rust API docs: https://docs.pola.rs/api/rust/dev/polars/#simd
lancedb's data format; Lancedb/lance works with [Pandas, DuckDB, Polars, PyArrow,]; https://github.com/lancedb/lance
Narwhals' df interface (Python) https://narwhals-dev.github.io/narwhals/
substrait's portable query plans: https://substrait.io/ , https://github.com/ibis-project/ibis-substrait
Arrow RecordBatch, https://news.ycombinator.com/item?id=45495738#45546244
cargo-fuzz, TLA+
- malisper3 hours ago
  At least for ClickBench, the remaining bottleneck is memory bandwidth. I think that's mostly going to be solved by better data representations than SIMD.
  One of the biggest wins was our hash table implementation. Depending on the cardinality of the data, it switches between design that is optimized for L2 cache vs something that is outside of L2.
convolvatron4 hours ago
there's some great stuff in here, particularly the direct-to-binary codegen, threads instead of processes and sync batching. you must have some intuitive notion about where the performance gains come from.
jauntywundrkind3 hours ago
Port in OrioleDB next please. If our wish-granting really is at such a state. https://github.com/orioledb/orioledb
Great submission slipped by yesterday (!) on it's beta15 and beta16 released, on a big stability push. It now passes Postgres's own test suite! https://www.orioledb.com/blog/orioledb-beta15-16-stability https://news.ycombinator.com/item?id=49099108
- malisper3 hours ago
  Once it's stable, that's definitely something I'm going to look at doing
jauntywundrkind3 hours ago
Port in OrioleDB next please. If our wish-granting really is at such a state. https://github.com/orioledb/orioledb
KinetiNode3 hours ago
[dead]