> How deranged would it be to have every nfs client establish a wireguard tunnel and only have nfs traffic go through the tunnel?

Sounds good to me. I have my Wireguard tunnel set up so that only traffic intended for hosts that are in the Wireguard network itself are routed over the Wireguard tunnel.

I mostly use it to ssh into different machines. The Wireguard server runs on a VPS on the Internet, and I can connect to it from anywhere (except from networks that filter Wireguard traffic), and that way ssh into my machines at home while I am away from home. Whereas all other normal traffic to other places is unaffected by and unrelated to the tunnel. So for example if I bring my laptop to a coffee shop and I have Wireguard running and I browse the web with a web browser, all my web browsing traffic still gets sent the same normal way that it would even if I didn’t have the tunnel running.

I rarely use NFS nor SMB, but if I wanted to connect either of those I would be able to that as well over this Wireguard setup I have.

I built a NFS3-over-OpenVPN network for a startup about a decade ago; it worked “okay” for transiting an untrusted internal cloud provider network and even over the internet to other datacenters, but ran into mount issues when the outer tunnels dropped a connection during a write. They ran out of money before it had to scale past a few dozen nodes.

Nowadays I would recommend using NFS4+TLS or Gluster+TLS if you need filesystem semantics. Better still would be a proper S3-style or custom REST API that can handle the particulars of whatever strange problem lead to this architecture.

This is the correct answer, routing between subnets is how it’s suppose to work. I think there are some edge cases like DR where it seems like stretching L2 might sound like a good idea, but it practice it gets messy fast.

This.

Instead you can create multiple Wireguard interfaces and use policy routing / ECMP / BGP / all the layer 3 tricks, that way you can achieve similar things to what vxlan could give you but at layer 3.

There's a performance benefit to doing it this way too, in some testing I found the wireguard interface can be a bottleneck (there's various offload and multiple core support in Linux, but it still has some overhead).

EVPN/VXLAN fabrics are becoming industry standard for new deployments. MACSEC/IPsec is industry standard for site-to-site.

You'd be surprised to know that this is especially popular in cloud! It's just abstracted away (:

> Noting Tailscale, since it has been mentioned, has comparatively extremely slow performance.

Isn't this mainly because Tailscale relies on userspace WG (wireguard-go)? I'd imagine the perf ceiling is much higher for kernel WG, which I believe is what Netbird uses.

How is IPSec performance better than wg? I never heard this before, it sounds intriguing.

VXLAN is for L2 between campuses. It is commonly used in enterprise and cloud networks.

Tinc is a fantastic piece of software. Very easy to use and configure, and it just works.

These days I lean towards WireGuard simply because it's built into Linux, but Tinc would be my second choice.

You can see my reply below: https://news.ycombinator.com/item?id=46609044 I believe our setups are pretty equivalent.

I achieve load balancing by running native wireguard on a vps at hetzner, I've got a native wireguard mesh, I believe Talos can do the same, where the peers are manually set up, or via. tailscale etc. I then tell k3s that it should use the wireguard interface for vxlan, and boom my kubernetes mesh is now connected.

flannel-iface: "wg0" # Talos might have something similar.

I do use some node-labels and affinities to make sure the right pods end up in the right spot. For example the metallb annoucer always has to come from the hetzner node. As mentioned in my reply below, it takes about 20ms roundtrip back to my homelab, so my sites can take a bit of time to load, but it works pretty well otherwise, sort of similar to how cloudflare tunnels would work, except not as polished.

My setup is here if it is of help

https://git.kjuulh.io/kjuulh/clank-homelab-flux/src/branch/m...

Is this your use case?

https://docs.zerotier.com/bridging/

ZeroTier does L2.

If you actually want to use vxlan ids to isolate l2 domains, like if you want multiple hypervisors separated by public networks to run groups of VMs on distinct l2 domains, then vxlan over WG seems like the way to go.

I've used wireguard for a while, not sure why I never considered doing BGP over it, might make for a fun weekend project.

Reduced MTU chopping off your maximum packet size from all the extra headers and other overhead you're adding?

Internal is fine because you control things like MTU so you don't have to worry about packet fragmentation/partial loss.

That seems like an awful amount of overhead for questionable gain.

What gave you that idea? Internally, Google uses GRE/GENEVE-like stuff but for reasons that have nothing to do with "preventing compromise" or whatever, but because they're carrying metadata (traces, latency budgets, billing ids.) That is to say, encapsulation is just transport. It's pretty much L3 semantics all the way down... In fact, this is more or less the point: L2 is intractable at scale, as broadcast/multicast doesn't work. However, it's hard to find comparisons to anything you're familiar with at Google scale. They have a myriad of proprietary solutions and custom protocols for routing, even though it's all L3 semantics. To learn more:

Andromeda https://research.google/pubs/andromeda-performance-isolation...

Orion https://research.google/pubs/orion-googles-software-defined-...

In my case, Tailscale does not implement K8S CNI.