Others mentioned tailscale, it's cool and all but you don't always need it.
As far as security, that's not even the consideration I had in mind, sure wireguard is secure, but that's not why you should have vxlan inside it, you should do so because that's the purpose of wireguard, to connect networks securely across security/trust boundaries. it doesn't even matter if the other protocol is also wireguard, or ssh or whatever, if it is an option, wireguard is always the outermost protocol, if not then ipsec, openvpn,softether,etc..whatever is your choice of secure overlay network protocol gets to be the tunnel protocol.
Instead you can create multiple Wireguard interfaces and use policy routing / ECMP / BGP / all the layer 3 tricks, that way you can achieve similar things to what vxlan could give you but at layer 3.
There's a performance benefit to doing it this way too, in some testing I found the wireguard interface can be a bottleneck (there's various offload and multiple core support in Linux, but it still has some overhead).
You'd be surprised to know that this is especially popular in cloud! It's just abstracted away (:
Also IME EVPN is mostly deployed/pushed when clueless app developers expect to have arbitrary L2 reachability across any two points in a (cross DC!) fabric [1], or when they want IP addresses that can follow them around the DC or other dumb shit that they just assumed they can do.
[1] "What do you mean I can't just use UDP broadcast as a pub sub in my application? It works in the office, fix your network!" and the like.
Sorry, but that's really reductive and backwards. It's usually pushed by requirements from the lower regions of the stack, operators don't want to let VMs have downtime so they live migrate to other places in the DC. It's not a weird requirement to let those VM's keep the same IP once migrated. I never had a developer ask me for L2 reachability.
Though there are definitely use cases where it is needed, and it is way easier to implement earlier than later.
IPSec over VXLAN is what I recommend if you are doing 10G or above. There is a much higher performance ceiling than WireGuard with IPSec via hardware firewalls. WireGuard is comparatively quite slow performance-wise. Noting Tailscale, since it has been mentioned, has comparatively extremely slow performance.
edit: I'm noticing that a lot of the other replies in this thread are not from network engineers. Among network engineers WireGuard is not very popular due to performance & absence of vendor support. Among software engineers, it is very popular due to ease of use.
The fastest I am aware of is VPP (open-source) & Intel QAT [1], which while it is achieves impressive numbers for large packets (70Gbps @ 512 / 200Gbps @ 1420 on a $20k+ MSRP server), is still not comparable with commercial IPsec offerings [2][3][4] that can achieve 800Gbps+ on a single gateway (and come with the added benefit of relying on a commercial product with support).
[1] https://builders.intel.com/docs/networkbuilders/intel-qat-ac...
[2] https://www.juniper.net/content/dam/www/assets/datasheets/us...
[3] https://www.paloaltonetworks.com/apps/pan/public/downloadRes...
[4] https://www.fortinet.com/content/dam/fortinet/assets/data-sh...
[1] https://www.arista.com/assets/data/pdf/Whitepapers/EVPN-Data...
One wonders what WG perf would look like if it could leverage the same hardware offload.
Isn't this mainly because Tailscale relies on userspace WG (wireguard-go)? I'd imagine the perf ceiling is much higher for kernel WG, which I believe is what Netbird uses.
Tailscale however, although it derives from WireGuard libraries and the protocol, is really not WireGuard at all- so comparing it is a bit apples to oranges. With that said, it is still entirely userspace and its performance is less than stellar.
I'm interested in this because I'm working on a small hobby project to learn eBPF. The idea is to implement a "Tailscale-lite" that eliminates context switches by keeping both Wireguard and L3 and L4 policy handling in kernel space. To me, the bulk of Tailscale's overhead comes from the fact that the dataplane is running between user and kernel space.
> "To me, the bulk of Tailscale's overhead comes from the fact that the dataplane is running between user and kernel space."
Yes and no, it's more complicated. DPDK is the industry standard library for fast packet processing, and it is in entirely user space. The Linux kernel netstack is just not very fast.
I think kernel networking is the only option for Tailscale (or any similar mesh VPN solution). Given this key constraint, the best you can do is do more work in kernel space and reduce context switches.
How deranged would it be to have every nfs client establish a wireguard tunnel and only have nfs traffic go through the tunnel?
Sounds good to me. I have my Wireguard tunnel set up so that only traffic intended for hosts that are in the Wireguard network itself are routed over the Wireguard tunnel.
I mostly use it to ssh into different machines. The Wireguard server runs on a VPS on the Internet, and I can connect to it from anywhere (except from networks that filter Wireguard traffic), and that way ssh into my machines at home while I am away from home. Whereas all other normal traffic to other places is unaffected by and unrelated to the tunnel. So for example if I bring my laptop to a coffee shop and I have Wireguard running and I browse the web with a web browser, all my web browsing traffic still gets sent the same normal way that it would even if I didn’t have the tunnel running.
I rarely use NFS nor SMB, but if I wanted to connect either of those I would be able to that as well over this Wireguard setup I have.
See perhaps NFS over TLS:
* https://datatracker.ietf.org/doc/html/rfc9289
Nowadays I would recommend using NFS4+TLS or Gluster+TLS if you need filesystem semantics. Better still would be a proper S3-style or custom REST API that can handle the particulars of whatever strange problem lead to this architecture.
I use Tinc as a daily driver (for personal things) and have yet to come up with a new equivalent, given that I probably should. Does Vxlan help here?
These days I lean towards WireGuard simply because it's built into Linux, but Tinc would be my second choice.
The problem is no doubt a people problem. I have learned to overcome these people problems by adhering to specific kinds of communication patterns (familiar to Staff Engineers and SVPs).
There is no reason that Wireguard over vxlan over Wireguard can't work, even with another layer (TLS) on top of Wireguard. Nonetheless it is very suboptimal and proprietary implementations of vxlan tend to behave poorly in unexpected conditions.
We should remember that vxlan is next-get vlan.
The type of Wireguard traffic encapsulated within the vxlan that comes to mind first is Kubernetes intra/inter-cluster pod-to-pod traffic. But this Wireguard traffic could be between two legacy style VMs.
If I were the operator told "you need to securely tunnel this vxlan traffic between two sites" I would reach for IPsec instead of Wireguard in an attempt to not lower the MTU of encapsulated packets too much. Wireguard is a layer 4 (udp) protocol intended to encapsulate layer 3 (ipv6 and legacy ip) packets.
If I were the owner of the application I would bake mutual TLS authentication on QUIC with "encrypted hello" (both elliptic and PQ redundant) into the application. The applications would be implemented in Rust, or if not practicable to implement the application in Rust I would write into the Helm chart a sidecar that does such a mutual TLS auth part (in Rust of course).
I would also aggressively "ping" in some manner through the innermost encapsulation layer. If I had tenancy on a classic VM doing Wireguard over the vxlan I would have "ping -i 2 $remote_inside_tunnel_ipv6" running indefinitely.
I considered dropping my root wireguard and setting up just vxlan and flannel, but as I need NAT hole punching I kind of need the wireguard root so that is why i ended up with it.
Going Wireguard inside the vxlan (flannel) in my case, would likely be overkill, unless I wanted my traffic between nodes between regions to be separated from other peers on the network, not sure where that would be useful. It is an easy way of blocking out a peer however, but that could just as well be solved on the "root" wireguard node.
There might be some MTU things that would be messed up going nested wireguard networks.
L2 is a total waste of time.
The lack of support for hardware EVPN is one of the many reasons that Mikrotik is not considered for professional deployments.
People who think one size fits all are not professional.
With that said, I love Mikrotik for what it is: it is very approachable and it fills a niche. I believe it has added a lot of value to the industry and I'm excited to see their products mature.
Take a look at AMS-IX, one of the largest internet exchanges: https://bgp.tools/ixp/AMS-IX
21/1020 (2%) of all peers are Mikrotik. 15 (1.4%) of those are >=1000mbps. 7 (0.6%) of those are 10gbps. None are larger than 10gbps.
If your impression is based on data circa ~2020, you should re-evaluate your priors with the recent packages in mind. See https://mikrotik.com/product/crs812_ddq
Service Provider ISPs cannot use Mikrotik - It is impossible. RouterOS supports none of the features required for a service provider. VRFs are even still unsupported in HW [1]. I am confused why this is even a discussion as anyone with experience working at an ISP/SP would come to the same conclusion.
[1] https://help.mikrotik.com/docs/spaces/ROS/pages/62390319/L3+...
There's a famous use-case from 10 years ago (sic!) of using MikroTik for serving over 400 customers, see https://mum.mikrotik.com/presentations/ID16/presentation_340... proving you could do it on small scale many years ago. Needless to say, A LOT has improved since. MikroTik has become a serious, and affordable means to power a small-to-midsize ISP in the recent years. Of course there are "enterprise" features for some people to get knickers in a twist over, but they are well beyond necessity. It's often that people were taught certain techniques, a certain way to do things (which more often than not includes all this domain over-extension madness and all that it carries with it up to L7!) so they struggle to adapt to alternative architectures.
To say that it's "impossible" to provide ISP services with MikroTik is reaching.
I have a mix of equipment from heavyweight juniper mxs at peeing points to arista dcs/ccs in large sites to £50 mikrotiks in the smallest branch offices.
Right tool for the right job, mikrotik is often but not always the right tool.
VXLAN is L2-like tranport over L3
You can have EoIP over WG with any VLANs you like.
You can have a VXLAN over plain IP, over EoIP, over WG, over IPSec. Only WG and IPSec (with not NULL sec) do providecany semblance ofvencryption in transit
And mandatory X\Y problem.
But it's not necessarily a bad idea. It depends on the circumstances, even when traversing a public network.
I achieve load balancing by running native wireguard on a vps at hetzner, I've got a native wireguard mesh, I believe Talos can do the same, where the peers are manually set up, or via. tailscale etc. I then tell k3s that it should use the wireguard interface for vxlan, and boom my kubernetes mesh is now connected.
flannel-iface: "wg0" # Talos might have something similar.
I do use some node-labels and affinities to make sure the right pods end up in the right spot. For example the metallb annoucer always has to come from the hetzner node. As mentioned in my reply below, it takes about 20ms roundtrip back to my homelab, so my sites can take a bit of time to load, but it works pretty well otherwise, sort of similar to how cloudflare tunnels would work, except not as polished.
My setup is here if it is of help
https://git.kjuulh.io/kjuulh/clank-homelab-flux/src/branch/m...
I used to run my K8S homelab through ZT as well. Latency is extremely bad.
What I wanted is more like meshed L2TPv3, but L2TPv3 is extremely hard to setup nowadays
I see VxLAN mentioned all over the place, but it seems that GENEVE isn't really implemented as much: besides perhaps being a newer protocol, is there a reason(s) why in your opinion? Where do you personally use each?
IPSec-equivalent, VXLAN-equivalent, IPSec-equivalent.
Prevents any compromised layer from knowing too much about the traffic.
Andromeda https://research.google/pubs/andromeda-performance-isolation...
Orion https://research.google/pubs/orion-googles-software-defined-...