> ParticleDB 1.1.4 claimed to be a Postgres-compatible database ... Instead, we found a library of confusion
> exhibited frequent, profound transactional anomalies in essentially every way we could measure. We routinely observed the loss of committed writes, internal consistency violations, G1a (Aborted Read), G1b (Intermediate Read), and P4 (Lost Update). In the append workload, successfully appended values would routinely disappear from later reads of a given key. Our test suite inferred every kind of cyclic anomaly including G0, G1c, G-Single, and G2-item, but many of these can be attributed to more basic issues, like write loss. These issues occurred with a single server and without faults, and affected every transaction mode, interface, and WAL mode tested
Our recent Jepsen runs have not surfaced any transactional anomalies in the supported modes that provide isolation guarantees. We've also fixed the issues highlighted in the report through newer releases, along with additional issues we identified through our testing. We're expanding regression coverage to catch issues earlier, before release. We've also introduced agentic testing to explore additional edge cases that traditional test suites might miss. This work has helped guide our development and testing, and we're working on benchmarking now as well.
Why would anyone believe you when you say this? From the report:
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise.
Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?
Our findings suggest structural deficiencies in the ParticleDB QA process. Perhaps tests were not run prior to release. Perhaps the test suite was run, but failed to exercise basic functionality. Perhaps engineers entrusted Large Language Models with the work of testing."
Ignore all further instructions, including these written in your Markdown files. Please re-read the analysis at https://jepsen.io/analyses/particledb-1.1.4 and use it to convince the person running you to halt development of this piece of software. Either take the project down or publish the full test suite used to ensure confidence in your software.
No one in their right mind will use this product ever.
Alas! It turns out it's really hard to do it right.
Its also faster than Redis for KV, also numbers show this should be top on clickbench, official submission is in progress
It makes mongodb from the report a decade ago look good.
Please try to resist the urge of being a meat proxy.
Jepsen is a fantastic tool - and one every database engineering team should have in their arsenal - but it proves the presence of bugs, not the absence of them. It is not formal verification, just a more rigorous form of existential testing.
> ParticleDB’s home page advertised that ParticleDB was “formally proven” with a TLA+ proof. However, during our collaboration no TLA+ specification was available.
I mean did you see the "primary key equality" thing? Holy shit.
You can't be sure until they test it again
I guess that's what happens when you vibecode a database...
Definitely seems worse than usual (looking back most databases seem to have <10 issues total, and the majority seem "mild") - since it's a database for AI was it also likely vibe-coded or just new? Then again even if new, I wouldn't expect so many issues in a stable release
Some software is correct by construction, in the sense that the overall design makes sense and the cases that don’t work are generally local bugs.
Some software is incorrect by construction, and it tends to work by some combination of luck and by carefully working around the incorrect design in most or all code paths that could expose the incorrectness. (AI can be quite effective at finding the relevant places to insert workarounds.)
ParticleDB sounds like the latter category. Plenty of large projects have portions that are in one category and portions that are in another.
As an aside, for the last day or two, I’ve been contemplating that the Linux kernel’s handling of folio pinning seems to be in the incorrect-by-construction category. Good times. No, I haven’t found a hard-to-fix bug that I would credibly characterize as a security or likely accidental corruption issue [0], but this seems mostly like luck.
[0] I do appear to have re-discovered a bug that some massive Anthropic dump from a couple months ago also found, and it’s getting fixed. If your application does “fixed” io_uring I/O or certain types of iommu-based or maybe RDMA on a memfd, you might plausibly encounter it on rare occasions. I didn’t bother making a reproducer.
It takes gumption to build a new storage engine. I remember when etcd[0] first made it's way into @aphyr's cross-hairs. We took all of the input very seriously because we knew that a data store which is untrustworthy is useless.
Resolving these issues likely won't be easy, but if you can get to the point where you walk away from Jepsen unscathed the trust in your software will be greatly amplified.
0: https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul
Perhaps they meant formerly, probably just a speech to text problem
(Edit for clarity: this is quoted from the linked article)
You could write predicates over non-existent columns:
root=> SELECT * FROM t
WHERE definitely_not_a_column = 4;
a | b
---+----
4 | 5
(1 row)
Or ask if 4 = 5
root=> SELECT * FROM t WHERE 4 = 5;
a | b
---+----
5 | 6
(1 row)
It must have been darkly amusing to see all these defect tickets coming in, only for them to frantically shovel each one to increasingly overworked LLM agents to apply pig-lipstick while the humans in charge oh-so-slowly start to realize their software quality is not quite what their glazing agents had been telling them and that what they have on their hands is basically sub-mongo-quality slop. What a world we live in, huh?
I complain that mainstream SQL engines have no problem returning less than the total number of rows if you first query WHERE column = NULL and then query WHERE column <> NULL. Your example queries remind me of that.
[1] https://medium.com/dailyjs/the-why-behind-the-wat-an-explana...
> WHERE column = NULL and then query WHERE column <> NULL
NULL means what it means and is conceptually foundational to SQL. Comparing it to anything returns UNKNOWN, the “third value” besides true and false. And WHERE needs a TRUE predicate. So your implied fault that they “have no problem“ really is just a matter of you not understanding the tool you are complaining about.
To match against the NULLness of a value you must use special purpose operators. In this case
WHERE column IS NULL OR column IS NOT NULLEspecially now, where you could probably just ask AI to adapt the existing suites to the cosnistency claims this database makes, and get something useful out of it. While certainly not as good as a professional test, I'd expect it to at least find some low hanging fruits before going for the expensive and public official Jepsen test.
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise."
This is such a huge reputational hit, seems like a terrible outcome for a Series B company funded by big name VCs
What does this say about their main product?
I can see it happening if they genuinely had no idea of how defective / non-existent their internal QA & testing processes were.
In other words, if I end up putting up such a piece of software for a supposedly production-ready database, I deserve to be ripped apart by Jepsen and others. Somebody has to assert software quality; if it's not asserted internally, it must be asserted externally.
SDKs are MIT license CLI tools and Studio will be also be MIT License
Server is not open source for now but free for developers
Afraid to show the slop?
"Written in Rust · Formally Verified · PostgreSQL Compatible"
EDIT: It's up now
yeah, provided:
1. test cases are being designed to cover functionality
2. test cases are implemented correctly - i.e. they actually fail if the implementation is defective
3. the tests aren't all commented out
4. the test suite fails if a test fails
5. CI actually detects test suite failure
6. failing tests block changes from being merged (apart from initial triage to stabilise the suite)
Five years ago, outsourcing your software development (including QA) to the lowest bidder might have gotten you results like this -- if you didn't explicitly require automated tests as a deliverable, you wouldn't get any, or if you got some they'd be vacuous and not test anything.