Others have pointed out that there are architectural steps you can take to minimize this pain, like keeping all CI operations isolated within scripts that can be run locally (and treating GitHub Actions features purely as progressive enhancements, e.g. only using `GITHUB_STEP_SUMMARY` if actually present).
Another thing that works pretty well to address the feedback loop pain is `workflow_dispatch` + `gh workflow run`: you still need to go through a push cycle, but `gh workflow run` lets you stay in development flow until you actually need to go look at the logs.
(One frustrating limitation with that is that `gh workflow run` doesn't actually spit out the URL of the workflow run it triggers. GitHub claims this is because it's an async dispatch, but I don't see how there can possibly be no context for GitHub to provide here, given that they clearly obtain it later in the web UI.)
Lefthook helps a lot https://anttiharju.dev/a/1#pre-commit-hooks-are-useful
Thing is that people are not willing to invest in it due to bad experiences with various git hooks, but there are ways to have it be excellent
Production runs should be immutable, but we should be able to get in to diagnose, edit, and retry. It'd lead to faster diagnosis, resolution, and fixing.
The logs and everything should be there for us.
And speaking of the logs situation, the GHA logs are really buggy sometimes. They don't load about half of the time I need them to.
2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts.
3. Having standalone scripts will allow you to develop/modify and test locally without having to get caught in a loop of hell.
4. Design your entire CI pipeline for easier debugging, put that print state in, echo out the version of whatever. You don't need it _now_, but your future self will thank you when you do it need it.
5. Consider using third party runners that have better debugging capabilities
Me, typically I have found it to be a sign of over-engineering and found no benefits over just using shell script/task runner, as all it should be is plumbing that should be simple enough that a task runner can handle it.
> If it works it works, as long as you can run it locally, it'll be good enough,
Maybe when it is your own personal project "If it works it works" is fine. But when you come to corporate environment there starts to be issues of readability, maintainability, proprietary tooling, additional dependencies etc I have found when people start to over-engineer and use programming languages(like Python).
E.g.
> never_inline 30 minutes ago | parent | prev | next [–]
> Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands.
However,
> and sometimes it's an even better idea to keep it in the same language the rest of the project is
I'll agree. Depending on the project's language etc other options might make sense. But personally so far everytime I have come across something not using a task runner it has just been the wrong decision.
Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering" causing similarly sized future issues. But then I also know everyone's history is vastly different, my views are surely shaped by the specific issues I've witnessed (and sometimes contributed to :| ), than anything else.
Strong opinions, loosely held :)
> Typically I see larger issues being created from "under-engineering" and just rushing with the first idea people can think of when they implement things, rather than "over-engineering"
Funnily enough running with the first idea I think is creating a lot of the "over-engineering" I am seeing. Not stopping to consider other simpler solutions or even if the problem needs/is worth solving in the first place.
> Yeah, tends to happen a lot when you hold strong opinions with strong conviction :) Not that it's wrong or anything, but it's highly subjective in the end.
I quickly asked Claude to convert one of my open source repos using Make/Nix/Shell -> Python/Nix to see how it would look. It is actually one of the better Python as a task runners I have seen.
* https://github.com/DeveloperC286/clean_git_history/pull/431
While the Python version is not as bad as I have seen previously, I am still struggling to see why you'd want it over Make/Shell.
It introduces more dependencies(Python which I solved via Nix) but others haven't solved this problem and the Python script has dependencies(such as Click for the CLI).
It is less maintainable as it is more code, roughly x3 the amount of the Makefile.
To me the Python code is more verbose and not as simple compared to the Makefile's target so it is less readable as well.
I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI.
If your CI can do things that you can't do locally: that is a problem.
Probably most of the times when this is an actual problem, is building across many platforms. I'm running Linux x86_64 locally, but some of my deliverables are for macOS and Windows and ARM, and while I could cross-compile for all of them on Linux (macOS was a bitch to get working though), it always felt better to compile on the hardware I'm targeting.
Sometimes there are Windows/macOS-specific failures, and if I couldn't just ssh in and correct/investigate, and instead had to "change > commit > push" in an endless loop, it's possible I'd quite literally would lose my mind.
IME this is where all the issues lie. Our CI pipeline can push to a remote container registry, but we can't do this locally. CI uses wildly different caching strategies to local builds, which diverges. Breaking up builds into different steps means that you need to "stash" the output of stages somewhere. If all your CI does is `make test && make deploy` then sure, but when you grow beyond that (my current project takes 45 minutes with a _warm_ cache) you need to diverge, and that's where the problems start.
Completely agree.
> I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI.
That is what I am doing, having my GitHub Actions just call the Make targets I am using locally.
> I mean, at some point you are bash calling some other language anyway.
Yes, shell scripts and or task runners(Make, Just, Task etc) are really just plumbing around calling other tools. Which is why it feels like a smell to me when you need something more.
Of course, if you use something else as a task runner, that works as well.
Neither do most people, probably but it's kinda neat how they suggested fix for github actions' ploy to maintain vendor lock-in is to swap it with a language invented by that very same vendor.
This isn’t only a problem with GitHub Actions though. I’ve run into it with every CI runner I’ve come across.
For my actions, the part that takes the longest to run is installing all the dependencies from scratch. I'd like to speed that up but I could never figure it out. All the options I could find for caching deps sounded so complicated.
You shouldn't. Besides caching that is.
> All the options I could find for caching deps sounded so complicated.
In reality, it's fairly simple, as long as you leverage content-hashing. First, take your lock file, compute the sha256sum. Then check if the cache has an artifact with that hash as the ID. If it's found, download and extract, those are your dependencies. If not, you run the installation of the dependencies, then archive the results, with the ID set to the hash.
It really isn't more to it. I'm sure there are helpers/sub-actions/whatever Microsoft calls it, for doing all of this with 1-3 lines or something.
If I can avoid it, no. Almost everything I can control is outside of the Microsoft ecosystem. But as a freelancer, I have to deal a bunch with GitHub and Microsoft anyways, so in many of those cases, yes.
Many times, I end up using https://github.com/actions/cache for the clients who already use Actions, and none of that runs in the local machines at all.
Typically I use a single Makefile/Justfile, that sometimes have most of the logic inside of it for running tests and what not, sometimes shell out to "proper" scripts.
But that's disconnected from the required "setup", so Make/Just doesn't actually download dependencies, that's outside of the responsibilities of whatever runs the test.
And also, with a lot of languages, it doesn't matter if you run an extra "npm install" over already existing node_modules/, it'll figure out what's missing/there already, so you could in theory still have "make test" do absolute everything locally, including installing dependencies (if you now wish this), and still do the whole "hash > find cache > extract > continue" thing before running "make test", and it'll skip the dependencies part if it's there already.
For caching you use GitHubs own cache action.
For things like installing deps, you can use GitHub Actions or several third party runners have their own caching capabilities that are more mature than what GHA offers.
https://docs.github.com/en/actions/how-tos/manage-runners/la...
What? Bash is the best scripting language available for CI flows.
This is probably why they invented a whole programming language and then neglected to build any debugging tools for it.
For me what worked wonders was adopting Nix. Make sure you have a reproducible dev environment and wrap your commands in `nix-shell --run`, or even better `nix develop --command`, or even better your most of your CI tasks derivations that run with `nix build` or `nix flake check`.
Not only does this make it super easy to work with Github Actions, also with your colleagues or other contributors.
Of course, the platforms would rather have you not do that since it nullifies their vendor lock-in.
1. When the build fails, you can SSH into the machine and debug it from there.
2. You can super easily edit & run the manifest without having to push to a branch at all. That makes it super easy to even try a minimum reproducible example on the remote machine.
Other than that, self-hosting (with Github or preferrably Forgejo) makes it easy to debug on the machine, but then you have to self-host.
But almost every company uses GitHub, and changing to Bitbucket isn't usually viable.
i just checked and in 2025 there was at least 2 outages a month every month https://x.com/swyx/status/2011463717683118449?s=20 . not quite 3 nines.
There was a time I wanted our GH actions to be more capable, but now I just want them to do as little as possible. I've got a Cloudflare worker receiving the GitHub webhooks firehose, storing metadata about each push and each run so I don't have to pass variables between workflows (which somehow is a horrible experience), and any long-running task that should run in parallel (like evaluations) happens on a Hetzner machine instead.
I'm very open to hear of nice alternatives that integrate well with GitHub, but are more fun to configure.
Who here has been thinking about this problem? Have you come up with any interesting ideas? What's the state of the art in this space?
GHA was designed in ~2018. What would it look like if you designed it today, with all we know now?
For the script getting run, there's one other thing. I build my containers locally, test the scripts thoroughly, and those scripts and container are what are then used in the build and deploy via Action. As the entire environment is the same, I haven't encountered many issues at all.
Like most of the glaring nonsense that costs people time when using msft, this is financially beneficial to msft in that each failed run counts against paid minutes. It's a racket from disgusting sleaze scum who literally hold meetings dedicated to increasing user pain because otherwise the bottom line will slip fractionally and no one in redmond has a single clue how to make money without ripping off the userbase.
GitHub action is a totally broken piece of s !! I know about that broken loops cause I had to deal with it an incredible number of times.
I very often mention OneDev in my comments, and you know what ? Robin solved this issue 3 years ago : https://docs.onedev.io/tutorials/cicd/diagnose-with-web-term...
You can pause your action, connect through a web terminal, and debug/fix things live until it works. Then, you just patch your action easily.
And that’s just one of the many features that make OneDev superior to pretty much every other CI/CD product out there.
Maybe that has changed.
gg watch action
Finds the most recent or currently running action for the branch you have checked out. Among other things.
edit: Just a quick note, the `gg` and `gg tui` commands for me don't show any repos at all, the current context stuff all works perfectly though.
Being able to run your entire "pipeline" locally with breakpoints is much more productive than whatever the hell goes on in GH Actions these days.
Setting up my github actions (or gitlab) checks in a way that can easily run locally can be a bit of extra work, but it's not difficult.
Are there other platforms allowing that? Genuinely interested.
Ah, that's like 90% of the way there, just need to enable so the SSH endpoint is created at the beginning, rather than the end, so you could for example watch memory usage and stuff while the "real" job is running in the same instance.
But great to hear they let you have access to the runner at all, only that fact makes it a lot better than most CI services out there, creds to SourceHut.
Maybe it is, I've never tried :-). I don't see a reason why not, probably it is.
Because you can extend and override jobs, you can create seams so that each piece of the pipeline is isolated and testable. This way there is very little that can go wrong in production that's the CI fault. And I think that's only possible because of the way that Gitlab models their jobs and stages.
> For the love of all that is holy, don’t let GitHub Actions
> manage your logic. Keep your scripts under your own damn
> control and just make the Actions call them!
The problem was the environment setup? You couldn't get CUE on Linux ARM and I am assuming when you moved to Makefiles you removed the need for CUE or something? So really the solution was something like Nix or Mise to install the tooling, so you have the same tooling/version locally & on CI?
"GitHub actions bad" is a valid take - you should reduce your use to a minimum.
"My build failed because of GitHub actions couldn't install a dependency of my build" is a skill issue. Don't use GitHub actions to install a program your build depends on.
And that’s where there’s a Mac Studio that sits sadly in the corner, waiting for a new check in so it has something to do.
Before that, most people would avoid Jenkins and probably never try Buildbot (because devs typically don't want to spend any time learning tools). Devs would require "devops" to do the CI stuff. Again, mostly because they couldn't be arsed to make it themselves, but also because it required setting up a machine (do you self-host, do you use a VPS?).
Then came tools like Travis or CircleCI, which made it more accessible. "Just write some kind of script and we run it on our machines". Many devs started using that.
And then came GitHub Actions, which were a lot better than Travis and CircleCI: faster, more machines, and free (for open source projects at least). I was happy to move everything there.
But as soon as something becomes more accessible, you get people who had never done it before. They can't say "it enables me to do it, so it's better than me relying on a devops team before" or "well it's better than my experience with Travis". They will just complain because it's not perfect.
And for the OP's defense, I do agree that not being able to SSH into a machine after the build fails is very frustrating.
(In general, I think a lot of criticisms of GitHub Actions don't consider the fully loaded cost of an alternative -- there are lots of great alternative CI/CD services out there, but very few of them will give you the OS/architecture matrix and resource caps that GitHub Actions gives every single OSS project for free.)
1) New technology comes out, people get excited
2) People start recognising the drawbacks of the technology
3) Someone else makes an improved version that claims to fix all of the issues. GOTO 1
As soon as I need more than two tries to get some workflow working, I set up a tmate session and debug things using a proper remote shell. It doesn't solve all the pain points, but it makes things a lot better.
Honestly, this should be built into GitHub Actions.
> For the love of all that is holy, don’t let GitHub Actions
> manage your logic. Keep your scripts under your own damn
> control and just make the Actions call them!
Honestly, I wonder if a better workflow definition would just have a single input: a single command to run. Remove the temptation to actually put logic in the actions workflow.
This is not even specific to GitHub Actions. The logic goes into the scripts, and the CI handles CI specific stuff (checkout, setup tooling, artifacts, cache...). No matter which CI you use, you're in for a bad time if you don't do this.
If you can't run the same scripts locally (minus external hosted service/API) then how do you debug them w/o running the whole pipeline?
I find GitHub Actions abhorrent in a way that I never found a CI/CD system before...
everything is including some crappy proprietary yaml rather than using standard tooling
so instead of being a collection of easily composable and testable bits it's a mess that only works on their platform
That's just the good old Microsoft effect, they have a reverse-midas-touch when it comes to actually delivering good UX experiences.
I agree that that should be reasonable but unfortunately I can tell you that not all developers (including seniors) naturally arrive at such conclusion no.
Granted, if you are working on "Windows 12", you won't be building, installing, testing, and deploying that locally. I understand and acknowledge that "as tight as possible" will still sometimes push you into remote services or heavyweight processes that can't be pushed towards you locally. This is an ideal to strive for, but not one that can always be accomplished.
However, I see people surrender the ability to work locally much sooner than they should, and implement massively heavyweight processes without any thought for whether you could have gotten 90% of the result of that process with a bit more thought and kept it local and fast.
And even once you pass the event horizon where the system as a whole can't be feasibly built/tested/whatever on anything but a CI system, I see them surrendering the ability to at least run the part of the thing you're working on locally.
I know it's a bit more work, building sufficient mocks and stubs for expensive remote services that you can feasibly run things locally, but the payoff for putting a bit of work into having it run locally for testing and development purposes is just huge, really huge, the sort of huge you should not be ignoring.
"Locally" here does not mean "on your local machine" per se, though that is a pretty good case, but more like, in an environment that you have sole access to, where you're not constantly fighting with latency, and where you have full control. Where if you're debugging even a complex orchestration between internal microservices, you have enough power to crank them all up to "don't ever timeout" and attach debuggers to all of them simultaneously, if you want to. Where you can afford to log every message in the system, interrupt any process, run any test, and change any component in the system in any manner necessary for debugging or development without having to coordinate with anyone. The more only the CI system can do by basically mailing it a PR, and the harder it is to convince it to do just the thing you need right now rather than the other 45 minutes of testing it's going to run before running the 10 second test you actually need, the worse your development speed is going to be.
Fortunately, and I don't even how exactly the ratio between sarcasm and seriousness here (but I'm definitely non-zero serious), this is probably going to fix itself in the next decade or so... because while paying humans to sit there and wait for CI and get sidetracked and distracted is just Humans Doing Work and after all what else are we paying them for, all of this stuff is going to be murder on AI-centric workflows, which need tight testing cycles to work at their best. Can't afford to have AI waiting for 30 minutes to find out that its PR is syntactically invalid, and can't afford for the invalid syntax to come back with bad error messages that leave it baffled as to what the actual problem is. If we won't do it for the humans, we'll do it for the AIs. This is definitely not something AI fixes, despite the fact they are way more patient than us and much less prone to distraction in the meantime since from their "lived experience" they don't experience the time taken for things to build and test, it is made much worse and more obvious that this is a real problem and not just humans being whiny and refusing to tough it through.
But it is true that GitHub Actions don't make it easy to debug (e.g. you can't just SSH into the machine after the build fails). Not sure if it justifies hating with Passion, though.
Took me a while to figure that out. While I appreciate occasional banters in blog articles, this one seems to diverge into rant a bit too much, and could have made its point much clearer, with, for example, meaningful section headers.
This way we can test it on local machine before deployment.
Also as other commenters have said - bash is not a good option - Use Python or some other language and write reusabe scripts. If not for this then for the off chance that it'll be migrated to some other cicd platform
No. It's cargo cult science.
Not by GitHub, but isn't act supposed to be that?
Don't put your logic in proprietary tooling. I have started writing all logic into mise tasks since I already manage the tool dependencies with mise. I tend to write them in a way where it can easily take advantage of GHA features such as concurrency, matrixes, etc. But beyond that, it is all running within mise tasks.