The way to build reliable software systems is to have multiple independent paths to success.
This is the Erlang "let it crash" strategy restated, but I've also found it embodied in things like the architecture of Google Search, Tandem Computer, Ethereum, RAID 5, the Space Shuttle, etc. Basically, you achieve reliability through redundancy. For any given task, compute the answer multiple times in parallel, ideally in multiple independent ways. If the answer agrees, great, you're done. If not, have some consensus mechanism to detect the true answer. If you can't compute the answer in parallel, or you still don't get one back, retry.
The reason for this is simply math. If you have n different events that must all go right to achieve success, the chance of this happening is x1 * x2 * ... * xn. This product goes to zero very quickly - if you have 20 components connected in series that are all 98% reliable, the chance of success is only 2/3. If instead you have n different events where any one can go right to achieve success, the chance of success is 1 - (1 - y1) * (1 - y2) * ... * (1 - yn). This inverse actually increases as the number of alternate pathways to success goes up and fast. If you have 3 alternatives each of which has just an 80% chance of success, but any of the 3 will work, then doing them all in parallel has a 97% chance of success.
This is why complex software systems that must stay up are built with redundancy, replicas, failover, retries, and other similar mechanisms in place. And the presence of those mechanisms usually trumps anything you can do to increase the reliability of individual components, simply because you get diminishing returns to carefulness. You might spend 100x more resources to go from 90% reliability to 99% reliability, but if you can identify a system boundary and correctness check, you can get that 99% reliability simply by having 2 teams each build a subsystem that is 90% reliable and checking that their answers agree.
In the 1930's, yes, component redundancy was the way to reliability. This worked at the time because components were flaky and technical systems were simple aggregations of components. Today, components themselves are more reliable, but even when they are not, redundancy adds only a little reliability because there's a new, large, source of failure: interactive complexity.
Today's systems are so complicated that many failures stem from insufficient, misunderstood, or ambiguous specifications. These errors happen not because a component failed -- all components work exactly as they were intended to -- it is only that in their intended interactions they produce an unintended result. Failure is an emergent property.
The solution is to approach reliability from a system theoretic perspective. This very early draft contains the core of the idea, but not yet fleshed out or edited: https://entropicthoughts.com/root-cause-analysis-youre-doing...
And "do something simpler" is actually a core part of this strategy. You're right that "today's systems are so complicated that many failures stem from insufficient, misunderstood, or ambiguous specifications". In most cases, yesterday's system worked just fine, you just can't sell it as a competitive advantage. So build simple, well-understood subsystems as fallbacks to the complex bleeding-edge systems, or even just take the software that's been working for a decade.
I’m working on the idea that there is better set of assumptions to use for directing technical development.
Failovers, redundancies, and backups are all important for building systems that are resilient in the face of problems, for reasons you've pointed out.
However, failovers, redundancies and backups are inefficient. Solving a problem with 1 thing is always going to be more efficient that solving the same problem with 10 things.
It's interesting to see this tradeoff play out in real-life. We see people coalescing around one or two services because that's the most efficient path, and then we see them diversifying across multiple services once bad things happen to the centralised services.
Yes, but notice that most things on the GP's comment have an exponential impact on reliability (well, on 1 - reliability), so they are often non-brainiers as long as they follow that simple model (what they stop doing at some point).
Just like every technique on the article also can turn out to reduce your reliability too.
Just to be clear, while this particular technique is valid and used in space software, it isn't common at all in Erlang and not part of the "let it crash" principle.
That's a heuristic that might work sometimes.
If you really want to build reliable software systems, then at least prove them correct. There are some tools and methodologies that can help you with this. And of course even a proof isn't everything since your assumptions can still be wrong (but in more subtle ways).
5 independent systems with 90% chance of success is mathematically as reliable as one that is 99.999%.
100x 90% systems would get you to 100 "9s" of reliability aka 99.99999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999%
(a) failures are correlated, not independent, and
(b) many failures happen not at the component level but at the plane where components interact, and regardless of how much redundancy there is at the component level, there is ultimately just one plane at which they finally interact to produce a result.
Math rarely applies so easily to real life. Talking about "independent" systems is cheap.
If at all possible. How would you transport yourself to work using two independent systems?
Your example of "how do you transport yourself to work using two independent systems" is actually very apropos, because I and many other commuters do exactly that. If the highway is backed up, I bypass it with local roads. If everything is gridlock, I take public transportation. If public transportation isn't functioning (and it generally takes a natural disaster to knock out all the roads and public transportation, but natural disasters have happened), I work from home and telecommute. Each of these subsystems is less favored than the alternative, but it'll get me to work.
Your two teams might well both use in some place in the system the same hardware or software component. This will make the probability of failure between the systems not be completely independent, for all that you paid two teams and they worked separately. Spent a lot of money, and the results will not be as expected. If they both use x86 Intel, and a Meltdown kind of thing happens, your "independent" systems will both fail from the same cause.
The transport analogy works great if you somehow imagine the transportation to be instantaneous, and only the decision to matter. But if you are already on a train and the train is delayed, you are not walking back home and taking the car. You have multiple options for transport, but you do not have a system built of independent components. You are not using the train and the car and the highway and the local roads all simultaneously.
I don't think you understand the requirements for the formula you wrote to be valid. Your examples do not fit, for all that they are reasonable and useful approaches. Your actual reliability with these approaches falls way below the multiple nines you think of.
When designing software, you first need to nail down the requirements, which I didn't really find in TFA.
It sucks, because nobody likes the idea of the "squeaky wheel getting the grease." At the same time, nobody is surprised that the yard equipment that they haven't used in a year or so is going to need effort to get back to working. The longer it has been since it was relied on to work, the more likely that it won't work.
To that end, I'm not arguing that all things should be the critical path. But the more code you have that isn't regularly exercised, the more likely it will be broken if anything around it changes.
Of course, as a programmer, this is by far not my first instinct. I am a programmer, my function is programming, not purchasing.
Of course buying something is always cheaper (compared to the cost of my time) and will be orders of magnitude cheaper once the costs to maintain written-by-me code is added in.
Things that are bought -tend- to last longer too. If I leave my job I leave behind a bunch of custom code nobody wants to work on. If I leave Redis behind, well, the next guy just carries on running Redis.
I know all this. I advocate for all this. But I'm a programmer, send coders gotta code:) do it's not like we buy everything, I'm still there, still writing.
Hopefully though my emphasis is on adding value. Build things that others will take over one-day. Keep designs clean, and code cleaner.
And if I add one 'practice' to the list; Don't Be Clever. Clever code is hard to read, hard to understand, hard to maintain. Keep all code as simple as it can be. Reliable software is software that mostly isn't trying to be too clever.
I've been thinking a lot lately about the cost of off-the-shelf solutions from the perspective of sustainability, and there is a cost beyond money. The performance of software almost always degrades over time. By buying Foo off-the-shelf, you are saying, "I am ok with getting on the same bloat-dictated hardware upgrade cycle as Foo."
Of course you have the option of buying Foo and never upgrading, unless Foo has a license that forces you to. But that also walls you off from security bugfixes. But by replicating the essential features of Foo in-house, you can actually set and stick to a complexity and performance budget.
Of course if you are a business of any real size, you're already on the hardware upgrade treadmill anyway, and probably all of your customers are too, so what does it matter if your software is a little slower and a little more resource hungry year after year after year? Other than maybe a little twinge of guilt every now and then.
I suppose the key is to understand the hidden costs with both approaches. The salary vs subscription cost is part of it, but there also subtle things, like flexibility (or lack thereof in bought systems), security (or lack thereof in homegrown systems) and so on.
It does get difficult in more complicated cases thanks to a lack of information on what a good solution looks like. This article attempts to straighten it out a little: https://entropicthoughts.com/build-vs-buy
Yes, once you factor in transaction costs, integration costs, risks contamination from that 3rd party, risks from lack of value alignment with that 3rd party (remember the Unity game engine?)...
Or, in other words, people that say that phrase you said very often don't know the actual cost of buying. But well, nobody knows the actual cost of building before they try either.
Only build software if the cost of maintaining it is 1/3 or less than the cost of buying a license.
(this has the nice second-order effect of being more robust to errors in the maintenance estimate, hence making it quicker to estimate).
I don't think this is counter intuitive at all... this is the whole premise behind free software. Why write it yourself when someone else already has and there is a community around using and updating it. We all buy the vast majority of our software and it is usually our go to move, unless there is an itch.
But in the context of "you're at work paid to be a programmer" the instinct is to look for things to program.
var thread = new Thread(() => {
Thread.Sleep(10);
throw new Exception("Uh oh!");
});
thread.Start();
Thread.Sleep(100);
Console.WriteLine("Done!");
You can opt into unobserved task exceptions terminating the application if that's what you are looking for, and maybe subscribe to TaskScheduler.UnobservedTaskException event too: https://learn.microsoft.com/en-us/dotnet/api/system.threadin...
Notably, this is an issue in Go where a package might spawn a goroutine with uncaught panic, like dereferencing a nil which is common, and you have no recourse to this at all. Perhaps it did historically make sense in Go, but it continues to bite people and requires more careful vetting of the dependencies. Moreover, in type-safe memory-safe languages uncaught exception might be a perfectly fine thing to ignore.
When you fire and forget a task and it ends up throwing, GC will simply collect all the objects that no longer have GC roots, and the finally blocks will be ran, and finalizers will be called eventually on Gen2 GC if there are any - the standard library and most community abstractions that interact with manual memory management through interop or otherwise end up being watertight as a result of that.
But why do we invest so much complexity into outputting html/js/css.
It’s always funny to me when people act like “websites” are some trivial, silly little area of software, when in fact for a lot of people, it’s their primary use of a computer.
If you would build an in-memory cache, how would you do it?
It should have good performance and be able to hold many entries.
Reads are more common than writes. I know how I would do it already,
but I’m curious about your approach.
Let's assume that keys accessed follow a power law, so some keys get
accessed very frequently and we would like them to have the fastest
retrieval of all.
Even with that tweak, the hottest keys will still be scattered around memory. Wouldn't it be best if their entries could fit into fewer pages? So, maybe a much smaller "hot" table containing say the 1,000 most accessed keys. We still want a high load factor to maximize the use of cache pages so perhaps perfect hashing?
Why?
* it works
* it's available now
* it scales
* it's capable of HA
* it has bindings for every language you probably want to use
Why bother writing your own cache, unless it's for an exercise? Cache management is complicated and error prone. Unless the roundtrip kills you just use redis (or memcached).
The fastest retrieval will be a cache hit, so really once the data structures are not the bottleneck then the focus should switch to the hit rates. That's where the Count-Min sketch, hill climbing, etc. come into play in the Java case. There's also memoization to avoid cache stampedes, efficient expiration (e.g. timing wheels), async reloads, and so on that can become important. Or if a dedicated cache server like memcached, one has to worry about fragmentation, minimizing wasted space (to maximizing usable capacity), efficient I/O, etc. because every cache server can saturating the network these days so the goals shifts towards reducing the operational cost with stable tail latencies. What "good performance" means is actually on a spectrum because one should optimize for overall system performance rather than any individual, narrow metric.
1. Make or buy
2. Release a MVP
3. Keep it simple
4. Prepare for the worst
5. Make it easy to tests
7. Benchmark, monitor, log...
Not sure about this tbh. In a lot of cases yeah maybe. But when you are dealing with complicated business logic where there is a lot of bells and whistles required, building a simple reliable version can lead you into a naive implementation that might be reliable but very hard to extend, while making an unstable complicated thing can help you understand the pit falls and you can work back from there into something more reliable. So I think this depends very much on the context.
Simple in my mind has abstractions where they are needed which should naturally lead to easy to extend code.
If someone posed this question to you in an interview and you used these principles, would you get the job?
Probably not.