I didn't really expect them to write code on the spot (hate that in interviews), but just to describe a possible solution; there were no wrong answers. Seeing how people came up with a quick hack for 10 and then being able to evolve their thinking for thousands was the point of the question and it could be enlightening to see.
I had a lot of people come up with things that I never even thought of, such as SSH'ing in to all 10 machines and then using a terminal feature to type into all of the windows at once.
For more than that, I used Puppet at the time (this was a decade and a half ago); I was a contributor to Puppet and standardized on it in my company. Eventually we moved to Ansible and I sold that business but last I heard, they are still using Ansible and likely using playbooks that were ported over from my Puppet stuff
* As sibling notes, there's ansible (or chef/puppet/salt/...)
* The traditional solution was https://github.com/duncs/clusterssh which opens an xterm to each target plus a window that you type into to mirror input to all of them
* I do the same-ish in tmux with
bind-key a set-window-option synchronize-panes
* Likewise, there are terminal emulators that let you do splits and then select a menu option to mirror input to all of them
It's main advantage is that it allows you to do SSH agent forwarding that actually works at scale, since it limits concurrency when talking to SSH agent to a configurable amount (by default 128, the default connection backlog in OpenSSH ssh-agent)
I know it is easy to be a hater, but sincerely do not see a reason to use something like that over Ansible or just pure sh, ssh and scp. All you have to do is to set up keys and the inventory. Takes 10 minutes, even if you are doing it for the first time. And you can expand it if you need it.
The reasons I find it over Ansible are:
- takes the same syntax and options as plain SSH, just run over multiple hosts. So if you already know SSH, you know how to use pssh that is an extension of the command. Ansible requires to study it. The configuration format is trivial, just a file that contains on each line one host, no need to study complex formats like Ansible
- doesn't require dependencies on the target machine. Ansible, as far as I know, requires a python3 installation on the target machine. Something that, for example, is not granted in all settings (e.g. embedded devices, that are not strictly GNU/Linux machines, for example consider a lot of network devices that espose an SSH server, like Microtik devices, with PSSH is possible to configure them in batch), or in some settings you maybe need to work on legacy machines that have an outdated python version.
- sometimes simpler tool that just do one thing are just better. Especially for tools like pssh that are to me like a swiss army knife, the kind of tool that you use obviously when you are bodging something up to make something work because you are in an hurry and saves your day
Of course if you already use Ansible to manage your infrastructure you may as well use it to run a simple command. But if you have to run a command on some devices, that were not previously setup for Ansible, and devices trough which you may not have a lot of control (e.g. a bunch of embedded devices of some sort), pssh is a tool that can come handy.
I do agree with your point, sometimes it's just easier to use native tools or simple wrappers around native tools. Use whatever makes your job easier
Stop assuming your method works across the universe of edge cases.
printf 'started: %s\n' "$(utcdate)"
(
trap 'kill 0' SIGINT
for REMOTE in "${REMOTES[@]}"
do
ssh -- "$REMOTE" "$COMMAND" "$@" &
done
wait
)
printf 'ended: %s\n' "$(utcdate)"
To send the same command to multiple servers, use pdsh: https://linux.die.net/man/1/pdsh
To collect all the results and show which ones are the same or different, use dshbak (i.e., "pdsh <parameters including servers>|dshbak"): https://linux.die.net/man/1/dshbak
Similar things, sometimes more convenient but less efficient for a large number of servers, are to use the konsole terminal program and link multiple window tabs together so the same typed command goes to all, and quickly view the results across the tabs; or to use tmux and send the same commands to multiple windows (possible useful "man tmux" page terms: link-window, pipe-pane, related things to those, activity, focus, hooks, control mode).
And others that I haven't used but which also look possibly interesting for platforms where pdsh and dshbak might not be available (like OpenBSD at least):
- https://github.com/duncs/clusterssh/wiki (available on OpenBSD as a package)
- https://www.gnu.org/software/parallel/ (also available as a package on OpenBSD 7.6: named "parallel-20221122"; might relate to "pdksh")
- Also clusterit.
sshsync group web-servers "sudo systemctl restart nginx"
And Yeah, now that you've mentioned it multiple shorter gifs would be better.
I only needed a very small fraction of what it can do to bail a client out of a problem their customer caused on several hundred computers the night before an event, but it absolutely saved the day and a lot of money.
Personally, I'm a tmux synchronized-panes kind of guy, so that I can see the result of each output immediately.
I have made scripts to do this with filter parameters over VMs on cloud providers, which is very valuable. Maybe you can extend this to have those options, so potential users are more attracted to it?
Maybe I should set a limit or let the user set a limit to how many results should be shown once the process is completed. Showing m and n results from the start and end
For the most common cases I have it aliased to just `p`: https://github.com/Julian/dotfiles/blob/main/.config/zsh/com...
Or https://github.com/Julian/dotfiles/blob/4d36e6b17e9804a887ba...
and I would be afraid to run SSH commands on multiple machines at once in case one of them errored out and needed manual intervention. ansible or puppet would let me know about that stuff.
maybe it's me! there is approximately zero chance that running the same script on four of my machines would result in four cleanly run scripts. one or more would fail, and if more than one failed, they would each fail for a different reason.
I personally do not prefer "run command in multiple places at once" over "run command in four places in sequence", however. I think it would just be a "random" choice in my case, and I might just write a script that does either. I do not mind as long as it is a one-time thing, but if I would have to do this more than once, I would just automate it, via scripts. I would probably just have it run in sequence.
Like a basic list of servers can also have this done via "ansible -m shell -a 'echo something' <server group>"