Battle of the non-Wireguard private networking solutions: ZeroTier vs. Twingate
My NAS is, unfortunately, behind an ISP-provided CGNAT that makes connecting a reverse proxy directly to its public IP completely impossible. As such, I have gone through several solutions to punch a hole through it to expose select services ( MinIO and Nextcloud, for example). Unfortunately, while I would have loved to be able to use something like Tailscale or NetBird, those are dependent on WireGuard- and curses! The CGNAT also stops WireGuard traffic through DPI (in Tailscale's case, it starts off working fine and then gets cut off after ~60 seconds or so), so I am forced to consider only non-WireGuard, proprietary protocol-based solutions. Out of the I'm sure many offerings out there, I have set up and used three- ZeroTier, Nebula, and Twingate. I would have loved to include Nebula in this test, but I got rid of my config at some point last year and it's a bit more of a hassle to get working compared to the other two. Maybe some time in the future.
Before I get into any testing, here's how my servers are set up:
- Server #1 (my NAS): 12 cores + 64GB RAM, running Unraid 7.0.0 with a 10Gb optical network card (but a 1Gbps internet plan), located in Korea
- Server #2 (HostHatch VPS): 8 (3+5) cores + 32GB RAM, running Ubuntu 20.04, network speeds fluctuate between 100Mbps and 1Gbps, located in Japan
Both machines have network send buffers set to 120MB.
The first test I'll be doing is a simple iperf3 test, with my NAS set up as the server and the VPS set up as the client.
- ZeroTier
- Twingate
Yikes. Not a good look for Twingate! Just in case, I checked whether it was attempting to connect via a relay, but it was connected through a direct peer-to-peer connection. By contrast, the recent updates allowing multithreading have really boosted ZeroTier's speed- prior to this, I was getting less than 100Mbps speeds.
One curious thing of note is that the Twingate connection almost seemed to be "warming up", so to speak, at the end. So I tried running another iperf3 test, but the results were, if anything, worse & more inconsistent than the first, so I've discarded it. In terms of bandwidth, ZeroTier wins handily.
Okay, then: what about latency? That's a bit more complicated.
A simple ping test isn't gonna cut it here, because they end up about the same at 70ms, which is to be expected from a Japan-Korea connection. No, I'm going to look at some real-world measurements. Specifically, Cloudflare Observatory measurements of my selfhosted Nextcloud instance.
Well, this is very odd. The first result is with ZeroTier, and the second one is with Twingate. Despite Twingate doing very badly in raw speed, it somehow manages a significant reduction in latency! I have to imagine this is due to some quirk in packet routing performed, because other than the network interface used to point the reverse proxy to the NAS, all configuration details are the same between the two runs.
Unfortunately, though, a slightly more comfortable website browsing experience is not enough to distract from the abysmal throughput Twingate provides. Hosting websites isn't the only thing I use it for, after all- I do weekly backups of my phone to my MinIO deployment, and the difference in speed saves me hours of having the backup app open draining battery life (this is, of course, completely ignoring other resource transfer factors, from Misskey images stored on MinIO to downloading/uploading files on Nextcloud).
Conclusion: I'll stick with ZeroTier for my CGNAT-bypassing purposes, thanks. At least until one of the WireGuard solutions gets an update to beat DPI...
