The worst part of all this is that GitHub's CTO and VP of Engineering sent out the usual "here's what we'll do to fix things" letter to their larger customers and, without exaggeration, it boiled down to: 1) "Here's a bunch of stuff we already did!" which... clearly isn't working, and 2) "We're continuing our Azure migration." also clearly not working.
So needless to say, if you depend on GitHub for critical business operations, you need to start thinking about what a world without GitHub looks like for your business and start working your way toward that. I know my confidence in GitHub's engineering leadership is at rock bottom.
It's starting to really look like the AI effect. It might be coincidence but I've noticed a lot more downtime and bad software lately. The last Nvidia drivers gave me a blue screen (last week or so), and speaking about Windows, I froze updates last year because it was clear they were introducing a bunch of issues with every update (not to mention unwanted features).
I like AI but actually not for coding because code quality is correlated to how well you understand the underlying systems you're building on, and AI is not really reasoning on this level at all. It's clearly synthesizing training data and it's useful in limited ways.
Man, a while ago I thought: "It happens often, alright, but every 2 weeks? Sounds like a slight exaggeration." But it really is every 2 weeks, isn't it? If I imagine in a previous job anything production being down every 2 weeks ... phew, would have had to have a few hard talks and course corrections.
How much of this is due to Microsoft Culture of not innovating and buying leading companies with their revenue from windows/office and slowly destroying the aspects of companies that made them great in the first place?
Is all the recent GitHub downtime entirely attributable to GitHub AI Copilot related development? How hard can it be to reduce the blast radius of new AI features to not affect the core parts of hosting repositories? Because of Copilot everywhere, The UX has become bad and I had to click all over the place and on my profile to find repositories.
Does anyone else ever think "that code I just pushed into my repo just took down all of github..." whenever it goes down around the same time you sync your changes?
Just moved a project of mine to Gitlab. Created this very simple component with codex that will keep a mirror updated on GitHub for me, so I can focus development on Gitlab.
I do not care about much of it other than the git and API. I also sometimes use the Issues, although only with the API. But if it stops working sometimes, that is not too significantly a problem since the files can be sent after they start to work again; it does not have to be immediately.
I remember back in the early Windows XP era when things got so bad that Microsoft basically had to make a hard pivot towards security and reliability.
I think they may need to do that once again. Almost every product of theirs feels like a dumpster fire. GitHub is down constantly, Windows 11 is a nightmare and instead of patching things they're adding stupid features nobody asked for. I think they need to stop and really look closely at what they're prioritizing.
I'm surprised nobody has tried to throw together a commercial alternative to GitHub. 50% of it is available as FOSS, the other 50% you can vibecode in a month (you can vibecode reliably, Microsoft/Google just suck at it). Afaict, reason we all keep using GitHub is it has a million features and isn't as ugly, difficult and slow as GitLab. (sorry GitLab, I love your handbook, hate your UX)
I've been sitting here waiting for a critical deploy to happen via GitHub Actions (I know, hour fault, we should have left ages ago). My patience for this bullshit is gone, I'm going to be pushing very hard to get us off of GitHub entirely except for public code mirrors going forward.
Edit: oh look, their site says all good, but I still have jobs stuck. What a pile of garbage.
So am I the only one thinking that maybe GitHub is succumbing to the weight of AI slop that's coming in from all the vibecoding, clawbots, and other AI workflows?
GitHub goes down at least once a week as I said before. [0] thanks to Copilot, Tay.ai and Zoe chatbots wrecking the platform instead of humans maintaining it.
If there was a prediction market for when GitHub experiences an outage every week, then you would make a lot of money.
Github CEO must be on HN, right? If so, any comments?
They have not even bothered to implement entra login when they have their competitors login for years, do they even know what their product is? Or are you just a middle man for slop?
Why don't companies with chronic outages mimic their stack from top to bottom (i.e. starting with a new domain), then before making a change, make the change on the duplicate stack and blast it with mock requests.
Might catch 90% of problems before they make it into the real stack?
E.g. every step of GitHub's migration to Azure could be mimicked on the duplicate stack before it's implemented on the primary stack. Is this just considered too much work? (I doubt cost would be the issue, because even if it costs millions, it would pay for itself in reduced reputational damage from outages).
EDIT: downvotes - why? - I think this is a good idea (I'd do it for my sites if outages were an issue).
200 comments
So needless to say, if you depend on GitHub for critical business operations, you need to start thinking about what a world without GitHub looks like for your business and start working your way toward that. I know my confidence in GitHub's engineering leadership is at rock bottom.
I like AI but actually not for coding because code quality is correlated to how well you understand the underlying systems you're building on, and AI is not really reasoning on this level at all. It's clearly synthesizing training data and it's useful in limited ways.
Is all the recent GitHub downtime entirely attributable to GitHub AI Copilot related development? How hard can it be to reduce the blast radius of new AI features to not affect the core parts of hosting repositories? Because of Copilot everywhere, The UX has become bad and I had to click all over the place and on my profile to find repositories.
https://gitlab.com/gabriel.chamon/ci-components/-/tree/main/...
addressing-githubs-recent-availability-issues-3news postI think they may need to do that once again. Almost every product of theirs feels like a dumpster fire. GitHub is down constantly, Windows 11 is a nightmare and instead of patching things they're adding stupid features nobody asked for. I think they need to stop and really look closely at what they're prioritizing.
https://about.gitea.com/
I can’t be specific but we are constantly complaining.
Edit: oh look, their site says all good, but I still have jobs stuck. What a pile of garbage.
I'm so sick of this.
If there was a prediction market for when GitHub experiences an outage every week, then you would make a lot of money.
[0] https://news.ycombinator.com/item?id=47487881
They have not even bothered to implement entra login when they have their competitors login for years, do they even know what their product is? Or are you just a middle man for slop?
Might catch 90% of problems before they make it into the real stack?
E.g. every step of GitHub's migration to Azure could be mimicked on the duplicate stack before it's implemented on the primary stack. Is this just considered too much work? (I doubt cost would be the issue, because even if it costs millions, it would pay for itself in reduced reputational damage from outages).
EDIT: downvotes - why? - I think this is a good idea (I'd do it for my sites if outages were an issue).