Some suggestions for improvement for your website/Github:
- I wanted to understand how it differs from rsync. A comparison page would be helpful.
- A comparison to rclone would also be interesting, as it also uses multiple parallel connections.
- As it target more advanced users I think it would be helpful to provide more advanced technical insights on how your transfers exactly work. Also how are edge cases handled, what tests are done? Moving and copying files is critical, and you need to inspire trust in your project among users.
As for edge cases, I will also try to document that more. In general, when I'm not sure what semantics to go for, I try to either copy what rsync does or do something safer. For instance, one thing I am looking at now is the best way to handle cases where a source path cannot be represented on the target filesystem or where two source paths would collide (e.g. due to unicode normalization or case insensitivity). Tentatively, my inclination is to try to fail before copying any files when possible. Currently, syq doesn't do this (but neither does rsync).
For me, the killer feature would be "ease of use."
Sure, I _can_ make rsync do any number of rarely-used optimizations, if I feel like studying the manpage for half an hour and figuring out how to fit it to my exact use case.
If your tool has the same features but is automatically adaptive -- I'd use it.
Copying thousands of tiny files in deeply nested subdirs? Just works. Copying a huge file that's already encrypted? Just works. Copying a mix? Just works.
No special flags to set, zero config. It just works, optimally, every time.
"Reducing the user's cognitive load" is the killer feature here.
rsync already supports this via OpenSSH's ControlMaster directive. Bonus, it speeds up every connection to that server rather then a single tool's.
> the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop).
You can control your agent forwarding in your local ssh client configuration. And with modern ProxyJump, you don't need to forward your agent at all regardless of how many bastion hops are between you and your final target.
ProxyJump doesn't help when you want to copy files from server A to server B without giving server A an agent that can do arbitrary things on server B. There is really no alternative to using a restricted authorized key on server B for this scenario.
As an aside, It's perhaps an indictment of our networking landscape that multiple parallel connections between 2 specific machines would accelerate a transfer. I would've expected a single TCP connection to be able to saturate a line. Or perhaps the parallel connections seeks to amortize the per-file setup overhead?
There is some info on the optimizations in the docs, but I agree that a more complete technical explanation of all the things syq does could be useful. I will work on one. On the other hand, I also tried hard to make it just go fast without needing the user to understand why it is fast or tune anything. For instance, the number of connections is auto-tuned by default.
First, thank you for taking the feedback :)
To be clear, I'm focusing on the first thing you should be telling users. It seems that, as a CLI program, the target audience of this tool is fairly technical users who are likely already familiar with SSH, rsync, and everything. As demonstrated in this thread, your first communication task is to convince them why it's worth their time to use your tool, considering they're probably satisfied with the pre-existing ones (or a replacement would already exist!).
My point is, explaining why it's better/faster than rsync shouldn't be buried in the docs. It should be a paragraph on the front page.
Hope this helps, and keep it up. HN is a rough audience to go public on ^^;
There's several factors at play that make this (usually) not the case.
Besides physical latency (which includes those added by any VPNs/tunnels/etc., some of which may be internal to an ISP along the route and outside of your control), there's other things like the TCP window sizes / window scaling option[1] that can affect single stream performance, and those type of parameters can differ by OS/interface type on both ends.
Also for SSH specifically, it has its own fixed buffer size that also limits throughput unless you're using the HPN-SSH fork[2].
How does the direct TCP mode work?
Getting into the unnecessary details: in classic live ISO fashion, the root (/) filesystem is tmpfs. Then I have at most 1 persistence drive and/or at most 1 9pfs mount. From there, I use bind mounts to only persist the few folders/files that I want preserved. For example, I'll have my persistence drive mounted at /persist and I'll bind mount "/var/lib/etcd" to "/persist/var/lib/etcd". Then when I reboot, everything outside of "/persist" is wiped out and I'm back to the initial iso + anything in /persist. So it is very similar to running a docker image with a volume mount, or running tails with persistence.
I build it all via NixOS and I use Impermanence to manage the bind mounts to the persist drive: https://wiki.nixos.org/wiki/Impermanence
You cannot do this without help from the filesystem. That's why rsync itself does not support this, and it has to be done by a specialized btrfs tool in this case.
That, and still claiming it’s “better than rsync”? Seems like you shouldn’t compare it with rsync (yet, if doing delta merges is planned)
- Single contributor. Software bugs are generally caused by developers writing code. By reducing the number of contributors, syq has cleverly reduced the surface area for defects to sneak in.
- Distribution via shell script. rsync is bundled in most linux distributions, which means you have to deal with annoying software updates from time to time. Syq, OTOH, tells you to pipe curl into bash to execute a shell script, which means one-and-done installation and maintenance.
- UI clarity. If you search StackOverflow for rsync, you'll see thousands of questions asking how to accomplish various tasks with rsync, both straightforward and arcane. By contrast, `syq` isn't even a tag on SO. The obvious conclusion here is that rsync's interface is so byzantine, and its documentation so poor, that users must resort to asking strangers for help, a problem obviously not shared by syq.
- You can install by piping curl to bash, using brew, or by compiling yourself. One thing about distribution I would argue syq gets right is that you are never relying on whatever version of syq happens to be installed on a server. The client syq always talks to a server syq tagged with the exact same version, and when your local copy of syq installs a remote copy, it verifies that the remote binary is signed by me.
- I never said or implied that rsync has a byzantine interface or poor documentation. I think rsync is well documented and the interface is overall fine. Syq even has an rsync compatibility mode, so you don't have to learn any new syntax if you don't want. Still, I tried to provide good docs for syq too, and honestly even if there were a gap in the docs, all you have to do these days is ask AI to look at the source code and tell you what to do. I think this also mitigates the trust issue with a new project from a single contributor.
Really serious benchmarks never contain any information about network characteristics, options used or buffer sizes, and these are indeed very serious. The "details" button produces huge bar graphs. No disappointment there.
Absolutely looking forward to 10x my speed, perhaps even transferring gigabytes per second on my gigabit link.