It shows zero contributions in the past year, on this account. This is a huge, huge red flag.
I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.
"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. "
Not all growth is good, especially growth that is actively hurting the company.
There is an equilibrium in both nature and software. Purposely designing systems that mimic the effects of cancer is going to benefit who exactly?
Github isn’t small startup, where other 10x threshold is as cheap as buy bigger box in your IaaS.
When you are already biggest player in the ecosystem and you suddenly get 10x persisted traffic, with at least 30x+ forecast “soon” - I am not surprised they have issues.
[0] even at current MS owned github
The worst outages I've been part of always have some version of this :(
Do you know of a documented instance where it would have helped?
The outage is due to massive load increase. In 4 months the number of commits doubled to 2.9 Billions. Anyone worked with high load systems knows that’s it’s not a normal growth and how difficult even to keep on horizontally scaling in a short time period such a complex system.
GitHub should charge at least maybe 5$ monthly fee and most of the entitled freeloaders would leave the platform and it would free up resources
Now remind me again, who trained a coding-assistant without consent on those "freeloaders" code and sold it for profit?
It's a service that is owned and operated by Microsoft Corporation, and we're the product of it.
Do you have a source for that factoid? (I suspect the vast majority of Github resource usage is paid. And we are upset.)
Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.
Github is ripe for disruption and I hope it is disrupted soon.
And from the RCA [1]:
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.
It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage, unless they have no reporting or their tooling is completely incompetent. If things can fall apart so easily, throwing more capacity at the problem won't fix it.
If it was failing slowly, why didn't they fix it then? Why didn't they notice?
Could it be because they fired all the people who know how to monitor and identify such problems? That's what SREs are for.
Please share the information you have which contradicts the conclusions I have drawn from Github's statement.
(And we know they're liars. They report very few of the actual incidents they have; see for example https://mrshu.github.io/github-statuses/)
Look, I hate Microslop as much as anyone but you'd have to purposely misinterpret TFA in order to arrive at this interpretation. C'mon.
These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.
The art of large system design is to identify and avoid these kinds of chokepoints. And when something happens, propagate the "backpressure" up the stack to avoid queuing.
AWS got a fair share of similar outages, so the newer SDKs now try to not exacerbate these kinds of issues: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retr...
The original AWS EBS outage is probably the canonical example: https://aws.amazon.com/message/65648/
If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.
Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but it's unattractive to me - as someone who agrees with most of their current politics - because the day they decide they don't like me, I'm screwed.
I've actually worked with a couple of companies who do use GitHub for their code, and they all use Linear in addition to GitHub.
I understand the concern you're talking about wrt. Codeberg, but I wouldn't view it as a significantly bigger risk than anything else. Any platform can suddenly decide that your project is against ToS (GitHub will absolutely not accept just anyone's use of their platform either) and Codeberg introducing some rules recently doesn't, in my mind, drastically increase the risk of a dramatic ToS change in the future. But we all have to make our own risk evaluations and I won't judge yours. Luckily, moving between Git hosts isn't that difficult; setting up CI again and losing merge request history does suck but it's not the end of the world, unlike something like, say, losing your AWS/GCP/whatever account.
Lots of companies moved because it was cheap, but it's not anymore. Ironic that companies might choose to migrate to them now for stability, rather than price.
Also for accuracy, Codeberg has a pro-human and anti-corporation stance. Both of which are definitely en vogue at the moment.
Much better than GitHub's pro slop sentiment, which is doing nothing but destroy their reputation.
My personal GitLab on the other hand really does take only a day or two per year.
That said, a week or two per year is just what it costs to maintain any one thing period. I spent about that much time maintaining PCs in the office, or my personal proxmox setup. It's not onerous at all.
GitLab is super bloated and a little sucky to admin, but it's not too bad all things considered. I'm admin in my new job's GitHub org and it sucks a whole lot more to maintain.
It's an expensive, low revenue generating site.
There are, and have always been, competitors, including "host it all yourself" solutions, but nothing has really stuck.
How is it "ripe" for disruption?
That can't be cheap.
Compute and Storage for Free Tiers: Hosting code for over 150 million developers and processing over 2 billion GitHub Actions (CI/CD) workflows a month requires astronomical server power and data storage. The "Free" tier is a massive cost sink that Microsoft treats as a loss-leader marketing expense
Let me know when you understand how that's not free.
What I read from this is: * Scaling is hard, we don't have enough capacity * We give away a shitton of compute for free * I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.
Notice there's nothing about paid customers, I'll add in what they are missing:
Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.
Central US data center failed to scale with it
I don't like paying for free stuff but gh certainly worth it.
Might force people to review their slop before pushing it.
Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem
It is specific and technical.
GitHub outages seem to be very normal and even more emotional these days. Yet, that’s no reason to treat each other destructively. It’s sad.
So let’s turn this energy into creating something better.
GitHub has alternatives, but no replacement
https://news.ycombinator.com/item?id=49135365
Why developers are ditching GitHub for Codeberg and self-hosting alternatives
https://news.ycombinator.com/item?id=48842611
and new entry:
Cursor Origin Code Hosting
GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.
I self-host it and its pretty easy to have uptime higher than github when you have 3 users total.
Absolutely do not recommend using GitLab.
That says absolutely nothing. The "What is Forgejo?" question is unanswered and instead you get a lot of words about their values, their inclusivity, etc. And the next thing in the docs is how to install it. It's ridiculous.
I still don't know what it is or what it does.
Seems like we are headed for Tech Gridlock.
I don't think GH has any of those, but just keep scaling, but that scaling failed.
Just bad architectural decisions from the postmortem.
--
It will only get worse due to AIs spawning massive commits, and they don't have unlimited cloud resource.
They can scale but not scalable in terms of effort, resources, and $
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.
GitHub does use queueing for all kinds of stuff internally, though, because they're not morons.
You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.
6 years ahead [1] on not going all in an centralizing everything on GitHub.
"If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.
I’m done.
"You've seen what we've done. The August 21st outage comes next. See you then!"
> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations
How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.
Well that's because in comparison to the absolute flood of traffic brought on by AI, they really haven't been operating on this scale before.