Hi. I have been using Zerotier (ZT) v1.6.6 on RouterOS 7.11 for the past few days and i have a speed problem. I use the Hap ac3 as a ZT peer. I think it's better to introduce my configuration first, and then ask for suggestions:
-my ZT network is 172.24.0.0/16 and managed on zerotier central
-my home LAN is 10.0.0.0/16
-the hap ac3 has a LAN ip of 10.0.0.1 and a ZT ip which is 172.24.0.1
-i have a synology NAS sitting behind the hap ac3 with an ip of 10.0.0.10
-i have a static route on the ZT central controller in order to access devices behind the hap ac3. for now, i am only accessing the NAS. the static route is: dst 10.0.0.0/24 via 172.24.0.1 (hap ac3 ZT ip)
-i configured the hap ac3 ZT peer according to this tutorial: https://help.mikrotik.com/docs/display/ROS/ZeroTier and made sure to add the firewall rules.
-my home ISP speed is 1000mbit/s down, 50mbit/s up
-remotely, i can properly ping and access the router and the NAS using their LAN IPs (10.0.0.1 and 10.0.0.10), since the static route was configured on zerotier central and the firewall rules added
to the hap ac3 firewall.
-no further configuration has been done on the home lan.
-accessing the home LAN remotely is done using a speed of approx 40mbps down, 5mbps up

remotely, when I try to transfer files, using WebDav or via http/https from or to the NAS, the speed is way low. I get upload speeds of 300kbps and download speeds of 900kbps, which is nowhere near the speeds i get when i do port forward on the hap ac3 and access my NAS without a tunnel like zerotier. for comparison, port forward method delivers a download speed of 3.5-4Mbps.
The cpu load of hap ac3, when transfering files is 20-30%, avg 25%, but i dont think this explains the slow download speed.
i checked the links between peers(pc-hap ac3) on the zerotier central and using the zerotier-cli on windows, they show a direct(non relayed) connection, and the respective public IPs are also showing up properly. ping times also give away a direct link. Any suggestions on improving my speeds would be appreciated.

Zerotier goes through the ZT network, if your physical location is remote, and there are no ZT root servers nearby, it can be slower.
You can read how it works here: https://docs.zerotier.com/zerotier/manual

A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a root).

My peers make a direct connection, not relayed. Isnt this the fastest link possible? From my understanding, the ZT central is just for letting peers know about the configuration and possible changes. But once the link is established, the peers communicate directly (see 4. in your link). The public IP addresses are accurate and i checked the links (which are direct) using the cli. So what am I missing?

Zerotier goes through the ZT network, if your physical location is remote, and there are no ZT root servers nearby, it can be slower.
You can read how it works here: https://docs.zerotier.com/zerotier/manual

A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a root).

I refuse to believe that this is a relay thing. If i move the zerotier peer from the mikrotik router to a windows desktop in the lan of the router, the speed goes up and maxes out.
This is a zerotier issue on the router and i have yet to find out how to address it.
maybe I have to specifically allow traffic for port number 9993 on the hap ac3? i read that on another forum and i was wondering why it was done..

They connect directly through more local servers. If there are no local servers it may all go through the slower relay...................
Clearly the solution is to move.

I refuse to believe that this is a relay thing. If i move the zerotier peer from the mikrotik router to a windows desktop in the lan of the router, the speed goes up and maxes out.
This is a zerotier issue on the router and i have yet to find out how to address it.

In my tests TailScale was significant faster than ZeroTier Peer to Peer …. Give TailScale a try and see for your self

Zerotier goes through the ZT network, if your physical location is remote, and there are no ZT root servers nearby, it can be slower.
You can read how it works here: https://docs.zerotier.com/zerotier/manual

A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a root).

You quoted that out of context, that is only how the initial packet of communication travels. Whole context here:

1. A wants to send a packet to B, but since it has no direct path it sends it upstream to R (a root).
2. If R has a direct link to B, it forwards the packet there. Otherwise it sends the packet upstream until planetary roots are reached. Planetary roots know about all nodes, so eventually the packet will reach B if B is online.
3. R also sends a message called rendezvous to A containing hints about how it might reach B. Meanwhile the root that forwards the packet to B sends rendezvous informing B how it might reach A.
4. A and B get their rendezvous messages and attempt to send test messages to each other, possibly accomplishing hole punching of any NATs or stateful firewalls that happen to be in the way. If this works a direct link is established and packets no longer need to take the scenic route.

Can you post your sanitized config.

My connection speed hardly is 500kbps now. This cannot be normal. I am aware that wireguard is faster in general, but not by that much. and if zerotier is configured properly, i should be at least getting a decent speed.. okay here is my mikrotik config.

https://pastebin.com/xBuvv0LG

also. heres a picture of the direct links between my windows 10 machine and the Mikrotik peer.

I don't see anything that would cause a speed problem in your config. Zerotier support is still in the works, so maybe it's something on mikrotik's end.

The only thing I have done different is limit the zerotier instance to running on my WAN, and instead of making specific firewall rules for zerotier I just added it to my LAN list
again I don't think either one of these changes will help your speed.

Let me see if i can try and speed test my zerotier tunnel and get back to you, (not that it helps since i am using rb4011)

smyers, how do I connect a subnet on one MT router (acting as a client node), to go out the WANIP of another MT router (acting as a server node) through zerotier,
That is what I have not been able to figure out? Then I will test that vs a wireguard connection I already have doing the same thing.
This sound similar to what the OP is trying to do ?

create route table
/routing table add name=out_zt fib

create an firewall address list with the clients you want to send over zerotier,


then create mangle rule tagging that traffic

/ip firewall mangle add chain=prerouting src-address-list=ZT_LIST action=mark-routing new-routing-mark=OUT_ZT

then create route
/ip route add dst-address=0.0.0.0/0 gateway=[IP of far zt router] routing-table=OUT_ZT

That sounds all MT and NO ZT for setup.
It wont be ip addresses it will be a subnet.
No need to mangle, source address is the subnet but will use Table and Route rule.
But how to get this subnet via zerotier (from client router) to server Router and to the server routers internet.

I know how to manipulate the MT side, just need help on the ZT side!!

Sorry I am not understanding then what your trying to do. There's not really a client server architecture in zerotier. When you connect to zerotier you are essentually plugging a wire into a virtual managed switch.

@smyers119 is there a way to test speed only to my mikrotik and not the NAS ?
wireguard performance is not that better either. i think something's wrong with the router..
did you test your tunnel?

@smyers119 is there a way to test speed only to my mikrotik and not the NAS ?
wireguard performance is not that better either. i think something's wrong with the router..
did you test your tunnel?

My test topology:

PC -->Microtik<-zerotier->opnsense in cloud-->internet

results: (maxed my upload speed)

well, it's definitely not Zerotier then. what's your RouterOS?

@smyers119 is there a way to test speed only to my mikrotik and not the NAS ?
wireguard performance is not that better either. i think something's wrong with the router..
did you test your tunnel?

My test topology:

PC -->Microtik<-zerotier->opnsense in cloud-->internet

results: (maxed my upload speed)
speedtestzt.PNG

well, it's definitely not Zerotier then. what's your RouterOS?

My test topology:

PC -->Microtik<-zerotier->opnsense in cloud-->internet

results: (maxed my upload speed)
speedtestzt.PNG

7.1.1 RB4011

I am thinking I need to go to ZT advanced settings and put in a route.

Destination is 0.0.0.0/0 via ZT IP address of the Server ROUTER.

However that will send any traffic on the ZT virtual LAN from any other node/device NOT JUST the ServerClient device and its specific subnet traffic to the Server Router.

I want ONLY to route ALL the traffic from the Client Router Node to the Server Router Node. If you see what I am saying......

on the device's where you don't want to push routes you can add "allow-managed=0" but note then you need to set ip and any routes manually which would be the preferred way on a router anyway.

on mikrotik

/zerotier interface set 0 allow-managed=no

I am thinking I need to go to ZT advanced settings and put in a route.

Destination is 0.0.0.0/0 via ZT IP address of the Server ROUTER.

However that will send any traffic on the ZT virtual LAN from any other node/device NOT JUST the ServerClient device and its specific subnet traffic to the Server Router.

I want ONLY to route ALL the traffic from the Client Router Node to the Server Router Node. If you see what I am saying......

on the device's where you don't want to push routes you can add "allow-managed=0" but note then you need to set ip and any routes manually which would be the preferred way on a router anyway.

on mikrotik

/zerotier interface set 0 allow-managed=no

I just noticed you could also set this to just not accept default routes, if you don't want the hassle of having it unamanaged:

/zerotier interface set 0 allow-default=no

I have no idea what those settings are doing on the MT.
Remember I have not pushed any traffic yet from any other devices onto the virtual LAN.
So its not a concern at the moment.
I fully expect that the missing gap MUST be done at the zerotier network level not on my MT devices.

For instance lets say I have FIVE MT DEVICES A B C D E
I want subnet X of device A, to go out internet of device E
I want subnet Y of device B, to go out internet of device C
I want subnet Z of device D, to also go out internet of device C.

Where is the zerotier help to make this happen???
There community help is a joke and their FAQ is a joke.
Im starting to lean to tailscale if its simpler.......... this is frustrating.

The routing would be done on the tik's not on zerotier.

so any suggestions on my case?

Zerotier goes through the ZT network, if your physical location is remote, and there are no ZT root servers nearby, it can be slower.

@normis
One reason that TailScale performs much better than ZT is because the USERS TailScale Network is a TRUE MESH --- True Peer to Peer communication

Tailscale’s server is really only needed to help the client devices find each other and get connected. None of the USER'S network traffic passes through the TailScale servers regardless of geography. So Lets say the users is based in Berlin Germany and the TailScale coordination Server is based in Toronto Ontario Canada ... the network path is pre-determined for the user's -- all the Traffic for that German User is local in Germany.

And which is why bandwidth performance is vastly superior IMO based on all my tests so far especially for people who have symmetric bandwidth plans like most fiber networks do.

Zerotier goes through the ZT network, if your physical location is remote, and there are no ZT root servers nearby, it can be slower.

@normis
And which is why bandwidth performance is vastly superior IMO based on all my tests so far especially for people who have symmetric bandwidth plans like most fiber networks do.

If you're looking for raw performance, ZT would be a poor VPN choose. But if you need a Layer-2 bridging, it's one of your only choices. So ZT vs Tailscale is saying TCP is better/worse than UDP – they are just different. Or MLPS vs OSPF would be another apt analogy to ZeroTier vs TailScale. e.g. ZT prefer the reliability of connection like TCP, but similar to MLPS, while Tailscale is more similar to UDP and OSPF. If you want a Mikrotik to show up in Winbox via discovery, you'll need ZT & that's not possible with TailScale. By the same token, if I want to have a more sophisticated auth scheme or simply cloud L3 routing/policy, ZT be poorly suited to those needs.

Anyway. On ZT, the issue is there is no way to know it may be using a root server (or moon or whatnot), or if "directly connected" via the Mikrotik. When I've tried bridging ZT over the internet, it does seem speed is a lot more inconsistent in speeds – sometime get closer to non-VPN speed, other times much slower.

In my case, we don't have stable fiber connections – We typically LTE & Wi-Fi available – plus those connected networks change regularly, plus asymmetric with very variable speed. In my use case, just need enough speed to run low bandwidth stuff like SSH, MQTT, winbox, etc - but as close to 100% uptime regardless of network/path/speed. So we config the remote Mikrotiks to try everything under the sun to make sure some connection out, which now includes ZT. ZeroTier seems quite aggressive at maintaining a link – so far if I can ping sometime from the Mikrotik, ZT has been able to find some pathway out.

That being said, I'm pretty sure it uses the roots/moons/whatnot unnecessarily - or, it reacts slowly to a change possible paths. So it would be nice if MT give a little more guidance on troubleshooting ZeroTier... What I've seen is continue to use a slower LTE route, even though a newer default route to much fast fiber line was added – it did seem "sticky" to way less optimal route, I actually wasn't sure how to troubleshoot thing...

@normis, are there some ZeroTier troubleshooting stats or help page coming? ZT seem to always find SOME link out, but not sure it's always picking an optimal one – that may be the OP's issue. In another posting someone saw ARP going out a weird interface, that I still don't understand and seems unresolved.

Anyway be good to know how does one find the interface a ZT connection should be using? And/or if its "directly connected". That might clarify if it is using a root part here. Connection tracking seems to show quite a few different ZT connections, while you can guess based on traffic, its not quite clear what's going on. ZT's routing table and selection doesn't seem to neatly follow the Packet Flow Diagram so hard to know if what ZeroTier is doing is "right"...

If you're looking for raw performance, ZT would be a poor VPN choose. But if you need a Layer-2 bridging, it's one of your only choices.

@Amm0
Your post was very interesting and I 4 1 very much appreciate the effort you put in to describe the ZT tribulations you've so far experienced.

Please TRY TailScale out and truly find out how a very efficient MESH actually works on a peer to peer basis from a VPN/WireGuard perspective .... I would state its very much like mimicking a MASSIVE Switch ... so no its not layer 2 but very close to it I bet if you actually tried it out you would be objectively impressed. BTW, did you know that TailScale is based out of Toronto Canada while ZeroTier is based out of Irvine, California -- not that it matters much.... BTW if you do take my suggestion and try TailScale out ....there is absolutely nothing to configure on your Tik unless you want to implement TailScale Subnet routers and traffic relay nodes ..... start small and after you get acclimatized -- grow as big as you need to....

I dont mind giving tailscale a shot. Does it run on mikrotiks?

If you're looking for raw performance, ZT would be a poor VPN choose. But if you need a Layer-2 bridging, it's one of your only choices.

@Amm0
Your post was very interesting and I 4 1 very much appreciate the effort you put in to describe the ZT tribulations you've so far experienced.

Please TRY TailScale out and truly find out how a very efficient MESH actually works on a peer to peer basis from a VPN/WireGuard perspective .... I would state its very much like mimicking a MASSIVE Switch ... so no its not layer 2 but very close to it I bet if you actually tried it out you would be objectively impressed. BTW, did you know that TailScale is based out of Toronto Canada while ZeroTier is based out of Irvine, California -- not that it matters much.... BTW if you do take my suggestion and try TailScale out ....there is absolutely nothing to configure on your Tik unless you want to implement TailScale Subnet routers and traffic relay nodes ..... start small and after you get acclimatized -- grow as big as you need to....

I dont mind giving tailscale a shot. Does it run on mikrotiks?

Your Tik is your router. When you install the TailScale client on your Phone, on your NAS, on your windows PC … whatever traffic is behind your Tik goes through your Tik …. There is absolutely nothing that you have to configure on your Tik ….. when you are remote and want to connect to your NAS for example the traffic will go through your Tik via your TailScale Network. TailScale manages everything for you. Give it a try and see for yourself.

I had good performance when i used ZT without it running on the router itself, that is between 2 windows machines. The poor performance was introduced when i moved the peer from the windows machine to the router, since i am out of home and it doesnt make sense to keep the pc on. So i wanted ZT on the router in order to access all the devices behind the router. Also my ds218j nas doesnt support docker and thus no ZT. Same for tailscale. aka i cant deploy it on either the router or nas.

Edit: actually ds218j supports tailscale. I was under the impression i did a search in the past, apparently remember wrong.

I dont mind giving tailscale a shot. Does it run on mikrotiks?

Your Tik is your router. When you install the TailScale client on your Phone, on your NAS, on your windows PC … whatever traffic is behind your Tik goes through your Tik …. There is absolutely nothing that you have to configure on your Tik ….. when you are remote and want to connect to your NAS for example the traffic will go through your Tik via your TailScale Network. TailScale manages everything for you. Give it a try and see for yourself.

Also my ds218j nas doesnt support docker and thus no ZT. Same for tailscale. aka i cant deploy it on either the router or nas.

You do not need docker. TailScale has a client for your Synology NAS … check the package center near the bottom. I’ve installed it on my Synology NAS .
My TailScale network has 2 windows 10 PC’s, my iPhone, my Synology NAS. Remotely I access my NAS via my phone and winows laptop … and when I want to manage my Tik router remotely I use my windows laptop to connect to my windows desktop via windows Remote Desktop. So in my case … my NAS, my desktop PC are behind my Tik router. Everything via TailScale vpn works really well.

BTW, I am only playing with TailScale to learn how stuff works. Normally I just use WireGuard to do everything I need to do and it’s all I need … but if you do not like to configure things especially for non-technical people TailScale is remarkable because it does everything for you under normal circumstances. When more complex issues arise then TailScale Subnet Routers come into play and that requires some effort.

I am trying tailscale atm. So far it seems ok. Any idea why the speeds are inconsistend? Download speed reaches my maximum bandwidth but then drops, then goes up all the time. I even set the metric of the tailscale tunnel in windows as the lowest of all adapters, same happens on my phone. It's definitely not my connection, since I tried mobile data as well.. Could it be a bottleneck on my router? I know it does not run a tailscale tunnel, but isnt it supposed to max out on the bandwidth?
I know for a fact that my synology operates properly, and the e WD Red 4TB drive also is good (110MB/s on LAN), so it could be an issue with my TIK. i will play around, disable the other interface tunnels on the TIK and report..

Also my ds218j nas doesnt support docker and thus no ZT. Same for tailscale. aka i cant deploy it on either the router or nas.

You do not need docker. TailScale has a client for your Synology NAS … check the package center near the bottom. I’ve installed it on my Synology NAS .
My TailScale network has 2 windows 10 PC’s, my iPhone, my Synology NAS. Remotely I access my NAS via my phone and winows laptop … and when I want to manage my Tik router remotely I use my windows laptop to connect to my windows desktop via windows Remote Desktop. So in my case … my NAS, my desktop PC are behind my Tik router. Everything via TailScale vpn works really well.

BTW, I am only playing with TailScale to learn how stuff works. Normally I just use WireGuard to do everything I need to do and it’s all I need … but if you do not like to configure things especially for non-technical people TailScale is remarkable because it does everything for you under normal circumstances. When more complex issues arise then TailScale Subnet Routers come into play and that requires some effort.

Any idea why the speeds are inconsistend? Download speed reaches my maximum bandwidth but then drops, then goes up all the time. I even set the metric of the tailscale tunnel in windows as the lowest of all adapters, same happens on my phone. It's definitely not my connection, since I tried mobile data as well.. Could it be a bottleneck on my router? I know it does not run a tailscale tunnel, but isnt it supposed to max out on the bandwidth?
I know for a fact that my synology operates properly, and the e WD Red 4TB drive also is good (110MB/s on LAN), so it could be an issue with my TIK. i will play around, disable the other interface tunnels on the TIK and report

When testing on your Phone are you in remote or at home ? if testing from home make sure to turn off your phone wireless and use only your cell connection -- if testing from remote location its ok to leave either connection methods on.
When testing on your Windows PC from home are you wired or wireless?

What you describe as >>>> Download speed reaches my maximum bandwidth but then drops, then goes up all the time <<<< is coming from your Tik Router and your ISP gateway ... when testing its best to keep it as simple as possible. Your Tik + your ISP device is providing the Bandwidth ... your Tailscale vpn client is exploiting that bandwidth and it can only use what it receives from the Router + ISP device. If the TailScale Client is an issue there are some troubleshooting steps you can follow: https://tailscale.com/kb/1023/troubleshooting/
Also check out the TailScale support forum at https://forum.tailscale.com/
They are very helpful ... for example https://forum.tailscale.com/t/dramatic- ... onnect/327

I turn on my mobile data with >100mbps 4G Lte, then connect to tailscale and download a file from my remote NAS behind the Mikrotik. the download speed is around 2MB/s (~16mbps), but my ISP speed is 50mbps upload. so i should be getting at least 6MB/s download with my mobile data.. at the same time, zerotier does not run on the Tik. its just the tailscale on the NAS. and wifi on my phone is also turned off. ONLY 4g active. In fact, i checked the TX rate inside the Tik wireguard peer and it reaches no more than 20mbit/s out of the 50 that my ISP provides

same performance is achieved if I connect with my phone the Tik wireguard tunnel and try to download a file from my NAS via this tunnel, but in that case the NAS does not run tailscale or wireguard.
so where is the bottleneck? why is my speed capped at <3MB/s via wireguard/tailscale ? my Coax ISP upload is stable at 50mbps.. docsis 3.1

Maybe my firewall rules? do i have to disable fasttrack or move it higher up? idk it bothers me so much that i cant achieve maximum bandwidth..

Any idea why the speeds are inconsistend? Download speed reaches my maximum bandwidth but then drops, then goes up all the time. I even set the metric of the tailscale tunnel in windows as the lowest of all adapters, same happens on my phone. It's definitely not my connection, since I tried mobile data as well.. Could it be a bottleneck on my router? I know it does not run a tailscale tunnel, but isnt it supposed to max out on the bandwidth?
I know for a fact that my synology operates properly, and the e WD Red 4TB drive also is good (110MB/s on LAN), so it could be an issue with my TIK. i will play around, disable the other interface tunnels on the TIK and report

When testing on your Phone are you in remote or at home ? if testing from home make sure to turn off your phone wireless and use only your cell connection -- if testing from remote location its ok to leave either connection methods on.
When testing on your Windows PC from home are you wired or wireless?

What you describe as >>>> Download speed reaches my maximum bandwidth but then drops, then goes up all the time <<<< is coming from your Tik Router and your ISP gateway ... when testing its best to keep it as simple as possible. Your Tik + your ISP device is providing the Bandwidth ... your Tailscale vpn client is exploiting that bandwidth and it can only use what it receives from the Router + ISP device. If the TailScale Client is an issue there are some troubleshooting steps you can follow: https://tailscale.com/kb/1023/troubleshooting/
Also check out the TailScale support forum at https://forum.tailscale.com/
They are very helpful ... for example https://forum.tailscale.com/t/dramatic- ... onnect/327

….. my Coax ISP upload is stable at 50mbps.. docsis 3.1
…….
Maybe my firewall rules? do i have to disable fasttrack or move it higher up? idk it bothers me so much that i cant achieve maximum bandwidth..

I checked your Tik config I do not see anything there that is hindering you. The TailScale support foks can inspect your TailScale client logs and give you some good feedback as to why your not getting more … I suspect it’s your Connection and the only way to check that is to have those TailScale client logs inspected.. LTE 4G can be erratic since that bandwidth is shared by many depending on the time of day/night …. The very same can be said for cable (DOCSIS) so those comm logs are invaluable …. Have you run the iPerf tests?

no. im not on site(home), but i have figured out the wireguard speed issue. i was not split tunneling, and my devices had all the traffic go through the WG tunnel, youtube etc and the NAS download speeds were slower for this reason, since the TIK had other stuff to do as well. So now i will just do split tunneling to access my home network only

….. my Coax ISP upload is stable at 50mbps.. docsis 3.1
…….
Maybe my firewall rules? do i have to disable fasttrack or move it higher up? idk it bothers me so much that i cant achieve maximum bandwidth..

I checked your Tik config I do not anything there that is hindering you. The TailScale support foks can inspect your TailScale client logs and give you some good feedback as to why your not getting more … I suspect it’s your Connection and the only way to check that is to have those TailScale client logs inspected.. LTE 4G can be erratic since that bandwidth is shared by many depending on the time of day/night …. The very same can be said for cable (DOCSIS) so those comm logs are invaluable …. Have you run the iPerf tests?

no. im not on site(home), but i have figured out the wireguard speed issue. i was not split tunneling, and my devices had all the traffic go through the WG tunnel, youtube etc and the NAS download speeds were slower for this reason, since the TIK had other stuff to do as well. So now i will just do split tunneling to access my home network only

@pitfermi …. EXCELLENT ……

yea, as soon as i changed the 0.0.0.0/0 to 10.0.0.0/16 (lan subnet) and 10.1.0.0/24 (WG subnet) on my client's configs, I get the full bandwidth now. see pic:

no. im not on site(home), but i have figured out the wireguard speed issue. i was not split tunneling, and my devices had all the traffic go through the WG tunnel, youtube etc and the NAS download speeds were slower for this reason, since the TIK had other stuff to do as well. So now i will just do split tunneling to access my home network only

@pitfermi …. EXCELLENT ……

@pitfermi. …. Nice work
I generally do not recommend to have WireGuard working AND TailScale at the same time … one or the Other … not both ….. but many do it

Very interesting this topic about ZeroTier and Tailscale things. In my case the experience with ZeroTier was good, but not perfect:

Some days ZeroTier network is a rocket and some days a small old car. Now about comparison of speed and latency I got so so the same results. I downloaded some files on my personal server at ~8Mbps with ~15 to 20 Mbps upload speed. Then I connected to my Minecraft Server getting ~300 to 500ms of ping.

The part that I hate of ZeroTier is when the computers are correctly connected (and you can see the online status on ZeroTier Central) but the traffic is impossible. All services down and is impossible get any connection, and the only solution sometimes is reinstalling ZeroTier or sometimes with a simple reboot is enought. This problem has been occured to me on all my devices (Windows machines, my RaspberryPi with Kali Linux and my Android mobile).

For now with Tailscale I not got this problem.

For this reason I'm using both now, if ZeroTier fails, I can use Tailscale.

Regards.

Thought to share my experience, although my setup is quite atypical. I am based in Thailand and I am always connected via RDP to a server I have in NJ - USA. There are some issue that are common for Thailand, (small pipe for international bandwidth) which I learned years ago to deal with a VPN which give me an advantage of around 50 ms as average. I let more figured out what could be the reason but I believe my vpn use a more efficient route to reach my server. I am using a RB4011.

I have used for a couple of years ExpressVPN which I would say returned excellent results in term of latency and bandwidth to USA.
I have recently used ZeroTier which unfortunately add a lot of latency to an extent that I would say is barely usable. (correctly installed)
I just added Tailscale and although is just a couple of days I have I would say the performance are at least the same of ExpressVPN which for Asia (Singapore) is by far the best vpn provider.

However one strange thing I noted is the following:

If I don't use any VPN my latency is ~280 ms.
ExpressVPN: ~245 (through Singapore)
ZT = ~265
TS= ~260

I am not sure what is the cause of ZT lag because the latency when I ping my server is more or less the same. But the RDP connection is way more pleasant with TS over ZT.

@krafg

I'm having the same issue: ~20Mbps over Zerotier & HAP ac3
If I turn off Zerotier and just NAT out, then I get ~200Mbps.
And if I enable Zerotier on my Windows desktop, through the same HAP ac3, then I get ~100Mbps

> The cpu load of hap ac3, when transfering files is 20-30%, avg 25%, but i dont think this explains the slow download speed.

It looks like it could be one of the CPU cores maxing out:

Columns: CPU, LOAD, IRQ, DISK
# CPU LOAD IRQ DISK
0 cpu0 45% 25% 0%
1 cpu1 100% 3% 0%
2 cpu2 15% 2% 0%
3 cpu3 1% 0% 0%

I have also added Fasttrack entries under the Firewall, which have made no difference.

Kind regards,
Ryan van Klaveren

@krafg

I'm having the same issue: ~20Mbps over Zerotier & HAP ac3
If I turn off Zerotier and just NAT out, then I get ~200Mbps.
And if I enable Zerotier on my Windows desktop, through the same HAP ac3, then I get ~100Mbps

> The cpu load of hap ac3, when transfering files is 20-30%, avg 25%, but i dont think this explains the slow download speed.

It looks like it could be one of the CPU cores maxing out:

Columns: CPU, LOAD, IRQ, DISK
# CPU LOAD IRQ DISK
0 cpu0 45% 25% 0%
1 cpu1 100% 3% 0%
2 cpu2 15% 2% 0%
3 cpu3 1% 0% 0%

I have also added Fasttrack entries under the Firewall, which have made no difference.

Kind regards,
Ryan van Klaveren

Same here on HAP ac3 and RB3011 we used for testing. One core maxes out at 100% and I guess this is the reason the bandwidth is limited to around 20MBit.
I wonder if this is related to some unsupported hardware encryption?

Did anyone try this on some CCR2116 or maybe CCR2004 or so?

Hey Andreas,

I've tested it with an RB5009 (arm64) and got the full line speed of 100 Mbps and the CPU didn't max out like the smaller units.

Haven't had a chance to test with an RB4011.

Also I believe Zerotier is only available on ARM/ARM64 processors. I tried on a CCR, but it had a Tile processor and didn't have Zerotier available.