MikroTik

ZeroTier on Mikrotik is only available for ARM/ARM64 architecture running RouterOS version 7.1+
This is biggest blocker to a lot useful use cases, since not all Mikrotik's are ARM-based.


You may not have to even read this! ZeroTier aims to be very simple.
So IF you followed Mikrotik's instruction AND know some networking/Mikrotik config already, SHOULD be easy. Now, it's the intersection with other RouterOS features/protocols where there may be more trouble, and why I write this. But seemingly even "OSPF over ZeroTier" can be resolved within 4 forum posts, see: viewtopic.php?t=183182 - so lots of hope it CAN be simple.

I do NOT try to replace Mikrotik documentation, or ZeroTier's either, rather supplement it here.
So this post make whole lot more sense if you'd already tried Mikrotik's ZeroTier instructions FIRST:
https://help.mikrotik.com/docs/display/ROS/ZeroTier

Read this post more as a "ZeroTier service manual for Mikrotik" – rather than a cut-and-paste config cookbook. For MIPSBE/TILE/SMIPS/CHR/X86 users – more "Planning Guide" Although, AFAIK, Mikrotik has NOT said anything about support for other platforms.

Disclaimer: I write at the prompting of @anav in several threads. Most of my knowledge comes from reading ZeroTier's docs, KB, reddit, and other forum posts here, so not an expert on ANY of this. But the details on how this all works on Mikrotik are a little vague and poor cataloged... so I TRIED to distill all the various bit of using ZeroTier on Mikrotik into one post – it's long as a result.... But no warranties or guarantees here.

ZeroTier Not Working?
The pre-flight checklist for RouterOS+ZeroTier:

• Can you ping the internet? (e.g.
  Code: Select all

  /ping [:resolve cloud.mikrotik.com]

  )
• Did you follow Mikrotik instructions to setup ZeroTier? (e.g. https://help.mikrotik.com/docs/display/ROS/ZeroTier)
• Is your firewall allowing both ZeroTier "peers"/tunnels (typically port 9993) & traffic ("zerotierX" interface) in /ip/firewall/filter?
• ZeroTier instance is enabled and running? (e.g. /zerotier/enable [find])
• Is the ZeroTier interface showing "OK"? (e.g. /zerotier/interface/print)
• Are all ZeroTier clients ("members") authorized in ZeroTier Central? (e.g. https://my.zerotier.com)
• Is there other Mikrotik config that might block the particular service you want to use (e.g. /ip/service)?
• Are the ZeroTier flow rules right? (e.g. using ZeroTier's defaults in most cases)
• But if your using non-IP ethertypes in your network...like RoMON, VLAN tagged trunks, etc., did you CHANGE the flow rules to allow the ethertype? (see below for details on flow rules)
• If your ZeroTier interface is a bridged port, is "Allow Bridging" for "Member" at https://my.zerotier.com have a checkmark/enabled? In all other cases, DO NOT CHECK "Allow Bridging"

Wait, so what can I use ZeroTier for?

Mikrotik says:

• Hosting a game server at home (useful for LAN only games) or simply creating a LAN party with your friends;
• Accessing LAN devices behind NAT directly;
• Accessing LAN devices via SSH without opening port to the Internet;
• Using your local Pi-Hole setup from anywhere via the Internet;

I can add a few more Mikrotik-specific ones, covered later here:

1. Using ZeroTier for remote management of RouterOS
2. Site-to-site routing using ZeroTier with static routes
3. Bridging a single LAN/VLAN to desktop/laptop using ZeroTier
4. Bridging an entire VLAN trunk using ZeroTier

Some non-Mikrotik example use cases for ZeroTier can be found here:
https://github.com/zerotier/awesome-zerotier
e.g. You don't necessarily need a Mikrotik to use ZeroTier.

What is ZeroTier, in one word: "deperimeterization"
That's ZeroTier's word choice, not mine – so easy to understand how ZeroTier concepts could get confusing. The ZeroTier client's source code is viewable on GitHub, if interested, but it has a good description of ZeroTier's aims:

ZeroTier is a smart programmable Ethernet switch for planet Earth. It allows all networked devices, VMs, containers, and applications to communicate as if they all reside in the same physical data center or cloud region.

This is accomplished by combining a cryptographically addressed and secure peer to peer network (termed VL1) with an Ethernet emulation layer somewhat similar to VXLAN (termed VL2). Our VL2 Ethernet virtualization layer includes advanced enterprise SDN features like fine grained access control rules for network micro-segmentation and security monitoring.

All ZeroTier traffic is encrypted end-to-end using secret keys that only you control. Most traffic flows peer to peer, though we offer free (but slow) relaying for users who cannot establish peer to peer connections.

The goals and design principles of ZeroTier are inspired by among other things the original Google BeyondCorp paper and the Jericho Forum with its notion of "deperimeterization."

If you want to read more...

ZeroTier's official docs and KB are here:
https://docs.zerotier.com
https://zerotier.atlassian.net/wiki/spaces/SD/overview

ZeroTier also have their own user forum:
https://discuss.zerotier.com/search?q=mikrotik

& Mikrotik-specific ZeroTier discussion is this forum post:
viewtopic.php?t=178063

My use case: "winbox+ssh over LTE"
This really did just work for me from day one – I followed Mikrotik instruction and my Mikrotik showed up in a remote winbox as "neighbor" (via a ZeroTier network). No muss, no fuss. Basically the experience as @gotsprings' comments in another posting:

Wow. For 10 years I have not been able to discover things when in Mikrotik VPNs. Suddenly I am connected to a device behind a router... And I have full view of the network.
[...]
But wow!

What ZeroTier solves is LTE networks are often behind a CGNAT. And with a CGNAT, basically you can enable any VPN protocol in RouterOS since you can't open listen ports. Obviously solvable in a lot of ways. But with ZeroTier you just add few config lines on the Mikrotik & then be able use winbox (or ssh/webfig/etc) from literally anywhere, relatively safely. The only problem is this only works for ARM-based Mikrotik, since ZeroTier is not available for MIPSBE, CHR, TILE, etc. yet

Maybe I'm a cynic... but seeing all your [ARM-based ] remote Mikrotik's just appear in winbox(+Mac+wine+ZeroTier), with only a few lines of config, is nothing short of miraculous. Why I share what I know about "ZeroTier on MikroTik RouterOS". Hopefully others can chime when I'm wrong, or share more details.

Multipath is missing in Mikrotik's ZeroTier!
i.e. potentially using ZeroTier Multipath for "cellular bonding"?
I use a lot of Mikrotik's LTE devices, including wAP ac R, Audience, and RB5009(+USB modem) that are supported by the ZeroTier package. Outside support for all architectures, "Multipath" be my #1 feature request for ZeroTier! It's not support on RouterOS today, ZeroTier's KB describe what it does:
https://zerotier.atlassian.net/wiki/spa ... /Multipath
Today, /zerotier on Mikrotik does not have the needed options to enable ZeroTier Multipath, specifically "defaultBondingPolicy" etc. that are in the ZeroTier's native local.conf. While I can't say how well ZeroTier Multipath Bonding works – I can't test it either. But "cellular bonding" is a common feature on competing LTE routers & RouterOS has NEVER had a good solution to bond (e.g. "aggregate") multiple LTE interfaces. Maybe ZeroTier work for this, dunno yet...

ZeroTier isn't "open source" but can be free
IANAL... For non-commercial use it can be used as if it were through my.zerotier.com controller (or pay for an enterprise plan). The "Free Plan" does offers unlimited networks, just only 1 admin account, and limited to 25 members in total. They do offer paid plans with high limits and other services, see ZeroTier pricing page. ZeroTier is licensed under a MariaDB-type "BSL" license, which is kinda different from a lot of commercial or OSS licenses, you can read it yourself if you want: https://github.com/zerotier/ZeroTierOne ... ICENSE.txt

About ZeroTier Planets, Leafs, and Moons...
Well...ZeroTier's backend isn't discussed here. ZeroTier's docs describe the various roles/topology, if interested. From a Mikrotik POV, these roles are mostly out of your control, so understanding them isn't critical to using ZeroTier. Now /zerotier/peer will identify if leaf, planet, etc, – which ALL can be thought about as "possible routing path" a ZeroTier tunnels MAY take.
A limitation today is a Mikrotik can NOT act as "Moon" (Private Root Server) or as "Self Hosted Controller" today – this is, in part, why you need to use the controller at my.zerotier.com to manage ZeroTier.
In recent v7 RouterOS, a Mikrotik can act as ZeroTier controller. See https://help.mikrotik.com/docs/display/ ... Controller – basically instead of using my.zerotier.com, you can use a Mikrotik acting as a ZT controller to create ZT networks and authorize members. But this seems like a whole different topic – the focus here is using a ZeroTier interface and ZeroTier network & NOT how ZeroTier internally works.

ZeroTier's Own Router Configuration Tips
ZeroTier's KB has an article with a few suggested configuration to make their tunnels work better:
https://zerotier.atlassian.net/wiki/spa ... ation+Tips
With the key takeaways for Mikrotik being:

• Using UPnP on your network can greatly improve performance by allowing ZeroTier endpoints to map external ports and avoid NAT traversal entirely.
• IPv6 is recommended and can greatly improve direct connection reliability if supported on both ends of a direct link.
• No Double NAT. Multiple layers of NAT introduce connection instability due to chaotic interactions between states and behaviors at different levels.
• If you have a firewall, allow traffic to/from UDP port 9993 to allow ZeroTier tunneling
• Should I forward any ports in my router? No. Let ZeroTier and UPnP and IPv6 handle it automatically.
• RouterOS does not support NAT-PMP, but that is part of their recommendations.

IPv6 "cheat code" using ZeroTier tunneling
Consensus is you should use IPv6, if you can. RouterOS v7 now has IPv6 is enabled by default, so if an ISP offers, Mikrotik will use it. And, ZeroTier will prefer IPv6 when establishing potential tunnels. Thus, even if you are blissfully unaware of IPv6, you may get IPv6 by virtue of using a ZeroTier network on a Mikrotik. i.e. even if your internal networks use only IPv4, IPv6 may be used by ZeroTier tunnels. Potentially getting a performance bump from ZeroTier's automatic usage of IPv6 & not having to learn a thing about IPv6 or change your IPv4 network one bit to get IPv6 internet.


ZeroTier vs Wireguard

This is a much discussed topic. To quote @normis:

Wireguard is something else. How will you run wireguard between two networks that have no real IP and where the private IP is changing all the time? This is where Zerotier can help

So I focused on what ZeroTier does – NOT if the right choice for some use case. The central difference between Wireguard and ZeroTier is:

• Wireguard works at with L3/Layer3/IP
• ZeroTier works at L2/Layer2/Ethernet/MACs/Bridges

So if you need just L3/IP routing, there are LOTS of other protocol choices that may be better than ZeroTier... In V6 terms, Mikrotik ZeroTier support does pretty much what EoIP+IPsec would, while Wireguard acts more like IPIP+IPSec. With the underlying philosophy behind them looking a lot like whether you'd use MPLS or OSPF – ZeroTier being akin to MPLS's "Layer 2.5"/tags/overly networks, with Wireguard being in OSPF's strict IP/Layer3 focus (in that analogy).

If you want "geek out", you can read, and compare the original WireGuard paper with ZeroTier's author's goals in his blog titled "Decentralization: I Want To Believe" – the difference between ZeroTier and Wireguard become readily apparent. With Donenfeld describing WireGuard as:

WireGuard is a secure network tunnel, operating at layer3, implemented as a kernel virtual network interface for Linux, which aims to replace both IPsec for most use cases, as well as popular user space and/or TLS-based solutions like OpenVPN, while being more secure, more performant, and easier to use. The virtual tunnel interface is based on a proposed fundamental principle of secure tunnels: an association between a peer public key and a tunnel source IP address.

with Ierymenko's 2014 Design Goals for ZeroTier

I began the technical design of ZeroTier with a series of constraints and goals for the underlying peer-to-peer network that would host the Ethernet virtualization layer. They were and are in my head, so let me try to articulate them now. These are in order of importance, from most to least.

1. Any device must be able, by way of a persistent address, to contact any other connected device in the world at any time. Initialization of connectivity should take no longer than one second on average, ideally as close as possible to underlying network latency. (Non-coincidentally, this is what IP originally did before mobility broke the "static" part and NAT and other kinds of poorly conceived fail broke IP generally.)
2. It must just work. It must be "zero configuration." The underlying design must enable a user experience that does not invite the ghost of Steve Jobs to appear in my dreams and berate me.
3. If the underlying network location of a peer changes, such as by leaving hotel WiFi and joining a 4G access point, connectivity and reachability as seen by any arbitrary device in the world should not lapse for longer than ten seconds. (This is just an aspect of goal two, but it's worth stating separately because it will have implications later in the discussion.)
4. It must work for users obtaining their Internet service via all kinds of real-world networks including misconfigured ones, double-NAT and other horrors, firewalls whose clueless administrators think blocking everything but http and https does anything but inconvenience their users more than their attackers, etc.
5. Communication should be private, encrypted, authenticated, and generally secure. Security should be end to end, with secret keys not requiring central escrow. The network must be robust against "split brain" and other fragmentations of the address space (whether intentionally-induced or not) and Sybil attacks.
6. The overall network should be as decentralized as possible. Peer to peer connectivity should always be preferred, and centralized points of control should be minimized or eliminated.

The fact ZeroTier uses "Ethernet" / Layer2 is actually makes it different to start, and useful property for a lot things (but NOT ALL things) on a Mikrotik. Since ZeroTier allows a "overlay network" on top of whatever network already exist, even regular networks users can also their own create ZeroTier networks, without a Mikrotik admin even being involved – that's part of ZeroTier's goals. So as Mikrotik admin, even if you're not using ZeroTier, it's possible your users are. And why some understand of ZeroTier may be useful even if you're not using it on a Mikrotik.


Back to ZeroTier on Mikrotik...

The first thing to understand is a /zerotier/interface on RouterOS ("zerotier1") = a "Member" of a ZeroTier "Network" created at https://my.zerotier.com. And each ZeroTier network is best thought about as a unique but "global ethernet switch", with Mikrotik's /zerotier/interface being connected to a port on that switch. And on the Mikrotik side, ZeroTier works same as if you added any other layer-2 interface in RouterOS. Which means it ZeroTier can be a port on a RouterOS bridge interface if that's what your want – allowing a range of possibilities, which I'll discuss later. But at the end of the day, the ZeroTier interface, once established, can be used just like an other "ethernet-like" interface on the RouterOS side: bridged or not bridged, up to you.

It's a little like EoIP...
If you're familiar with RouterOS, a good analogy for ZeroTier on Mikrotik is as a "sophisticated" version of EoIP+IPSec – so any use case where EoIP be useful, ZeroTier might be similarly useful. Configuration wise, EoIP and ZeroTier largely identical in their usage in RouteOS, and have similar properties inside /ip/firewall & /interface/bridge – the only difference is in the backend tunnels. In EoIP, tunnels are controlled by explicitly setting the remote end, while in ZeroTier the tunnels are created dynamically following ZeroTier's protocol to find paths to the remote end. And EoIP is one-to-one, while ZeroTier is potentially one-to-many. There are more use case for ZeroTier beyond an EoIP replacement – but EoIP is closest V6 protocol to ZeroTier, at least parts of ZeroTier's possibilities. Basically if you only have just two Mikrotik's connected to a ZeroTier network, it largely no different than having an ethernet cable between them from RouterOS config's POV, just ZeroTier will inject an IP address on it automatically. Now also the MTU be higher with ZeroTier than typical ethernet, & speed wouldn't even be close to line rate – but conceptually the same as an ethernet cable to switch – just slower WITH a few MORE gotchas (and it's these "gotchas" are what I'm try to cover here).


Mikrotik config: /zerotier vs /zerotier/interface
There is a ZeroTier instance under /zerotier ("zt1"), but does not effect configuration much, other than it needs to be enabled: But for the most part it can be ignored - we mainly deal with /zerotier/interface in this post. But it is the instance that automatically creates any needed tunnels used by one or more /zerotier/interface(s). But "the how" is outside the control of RouterOS config since it should follow ZeroTier's VL1/VL2 protocol. While you can control which Mikrotik interfaces a ZeroTier instance is allowed to potentially use via a "/zerotier/set [find] interface=", you can NOT control the specific path ZeroTier will actually use - the Mikrotik admin only controls the Mikrotik interfaces a tunnel will try to use. While the ZeroTier instance is not be covered by Packet Flow Diagram yet?, any ZeroTier interface like "zerotier1" follow ALL the normal packet flow diagram for a "logical in (or out) interface". So ZeroTier interface (e.g. traffic) is controlled the exactly same flow as something like EoIP or anything ethernet-like interface in any firewall rules etc. So managing the Mikrotik firewall around ZeroTier MAY be most difficult part here.

Initial ZeroTier Setup

Creating a ZT instance or network and attaching the various devices to the network is relatively painless.

Note: If you use winbox, you should use v3.32+ since it has ZeroTier support in the GUI. I use CLI commands in examples, but same stuff can be done winbox. In particular, winbox's ZeroTier>Peer view is pretty handy to see what's going on with ZeroTier "tunnels" (e.g. "peers").

I'll assume the basic ZeroTier setup from https://help.mikrotik.com/docs/display/ROS/ZeroTier has been done and at least getting an IP address from ZeroTier.. Try Mikrotik's help doc on ZeroTier before doing anything described here. That means:

• Your Mikrotik is what ZeroTier calls a "Member" of ZeroTier "Network" & need to have a "checkmark" under "Auth?" at https://my.zerotier.com
• /zerotier/interface/print should show "OK"
• Your Mikrotik should have an IP address (/ip/address/print) in the same range as the "Auto-assigned IP address" on the "ZeroTier Network" from my.zerotier.com web console.
• At least to start, do NOT change any the defaults in ZeroTier, other than those Mikrotik's ZeroTier docs recommends. e.g. use the "Easy" mode under "Auto-Assign from Range".
• If the IP range selected under "IPv4 Auto-Assign" within my.zerotier.com is in-use within your network, pick ANOTHER IP range from the list.
• You'll likely need to do something within your firewall to allow ZeroTier. Mikrotik's instructions on ZeroTier have a "optional step" to add a firewall rule (#6), but it isn't optional if you're using the default firewall.
• To see the tunnels created by ZeroTier, you can use:
  Code: Select all

  /zerotier/peer/print

Some additional troubleshooting tips:

• Get an auto-assigned IP address from ZeroTier showing up on the Mikrotik is STEP 1 here to know ZeroTier generally works BEFORE trying to config anything complex.
• For bridging cases, we won't use the defaults settings. But for most Layer3 things, ZeroTier's network defaults really should be okay to start with.
• If you can't get this working, check the firewall rules and/or add ZeroTier the the right /interface/list and/or verify the Mikrotik has a working internet connection/DNS/etc.
• This is easy to forget. But ALL devices joining a ZeroTier network MUST be individually authorized. This is done by checking a box in my.zerotier.com for the particular device ("Member" in ZeroTier terms) under the "Devices" section. Mikrotik's ZeroTier docs explain this, but needs to happen for each new participant in the network.
• Next to each "Member"/device in my.zerotier.com, there is a "wrench" icon. Tapping the "wrench" will show a couple device-specific network settings. By default, both "Allow Ethernet Bridging" and "Do Not Auto-Assign IPs" are UNCHECKED. Leave them this way for all devices initially. But I'll note that both "Allow Ethernet Bridging" and "Do Not Auto-Assign IPs" should be CHECKED ONLY IF if it's a Mikrotik device AND ZeroTier interface has been added as a bridge port – ALL non-Mikrotik other devices should still be UNCHECK even in the bridging case (and Mikrotik's that are NOT bridged, too). There is more configuration needed for bridging, discussed later – so really best to get a non-bridged ZeroTier connection working first!
• If you did already change some settings, you can start again with by creating a new ZeroTier network in my.zerotier.com – you'd need to re-add the network ID on the Mikrotik side to join the newly created network. You'd start at step #4 in the Mikrotik ZeroTier help.
• In the free plan, ZeroTier limits you to 50 "member" devices per network and 1 admin. But you can create unlimited number "Networks", so pretty easy to create multiple networks on my.zerotier.com for testing or have different configurations (say, eventually, have a different ZeroTier Network for each Mikrotik VLAN)...
• Either by design/config/bugs ZeroTier traffic may NOT show up in the /tool/sniffer or /tool/torch. Not tested personally but recall some forum posting about these not always working. So if a sniffer trace doesn't show ZeroTier, it may still be working and the problem could be with the sniffer/etc.
• ZeroTier's controller at my.zerotier.com has "Flow Rules" for a network. In most cases you can leave them alone – except for RoMON or VLAN trunking (and perhaps a few similar things). While you can do nifty stuff like block ports or even "ethernet types" on the ZeroTier side, to control any ZeroTier traffic – this is the "SDN" (software defined networks) part of ZeroTier (and why ZeroTier is pretty different from Wireguard). At 10,000 foot level, ZeroTier's flow rules are like a firewall, but since it's stateless that means no connection tracking & since a ZeroTier network is more like a switch, the Flow Rules operate on ALL ethernet frames passing through the ZeroTier traffic (so they are actually more similar to the RouterOS's Bridge Firewall). But the Flow Rules be whole different topic. Leaving them as defaults in my.zerotier.com network is BEST plan initially.

The rest is not so easy.

Well for the admin uses, I think it's pretty easy. For the bridging cases, the "mysteries" of the Mikrotik "Bridge" interface combine with nuances of ZeroTier – so kinda need to understand both pretty well so, yeah, "not so easy". With V7 being new too...the potential for subtle RouterOS bugs may add another fun element on top. Moving on the actual question about use cases...

Use Case A: Using ZeroTier for remote management of RouterOS

-Admin on iphone to configure an MT router (iphone and mt router on zerotier)

This should actually work if both the iPhone and Mikrotik are joined to the same ZeroTier network as "Members". You can use the ZeroTier assigned IP address to connect via webfig or Mikrotik iPhone app, assuming the iPhone has ZeroTier connected to same network. It really is that simple.

Firewall may be why this wouldn't just work...
If you're using the default firewall, you need to do something with zerotier1 interface to allow ("accept") it. e.g. RouterOS firewall rules (/ip/firewall/filter) may block Winbox, SSH, etc. ports on INPUT from zerotier1 (since, by default, it isn't part of the LAN /interface/list). Basically you need consider the ZeroTier interface (or IP range) and it's effect on your specific firewall rules – otherwise it may be DROP'ed by the default firewall filter rules. Mikrotik's example allows full access to the router from the connected ZeroTier network, while blunt, does work:
But a better approach may be to put the "zerotier1" interface in a specific /interface/list like "LAN" (or "BASE", "MGMT", etc.) so it be allowed via an EXISTING set of firewall rules, and controlled via an "/interface/list". In most default configs this would do the trick too:

And all smartphone/laptops/desktops/etc connected using a ZeroTier client app, that then connect to RouterOS over ZeroTier will appear using their ZeroTier-assigned IP address, via a Mikrotik's ZeroTier interface like "zerotier1" – any these can also be used in firewall rules as needed to accept/drop traffic too. Certainly you can use more restrictive rules within the Mikrotik firewall (e.g. allowing ONLY winbox/ssh/etc TCP ports using "... chain=input in-interface=zerotier1 ...").

/ip/service can also restrict winbox,ssh,https,api,etc
This isn't ZeroTier specific. But since ZeroTier does assign new IP range, you may have to add them as"allowed address" based on
/ip/neighbor "discover-interface-list" should include ZeroTier interface too...
winbox uses the "discovery protocol" from RouterOS – that how the list of RouteOS device is built in winbox's "Neighbors" tab shown at startup. But for this work, the laptop/desktop running Winbox must be connected to same ZeroTier network & the ZeroTier interface must be listed in a /interface/list of the "discover-interface-list" used in /ip/neighbors for "discovery over ZeroTier" to work. If you already added zerotier1 to the "LAN" interface list, the defaults would allow discovery to work over ZeroTier, to check make sure the "discover-interface-list" includes the ZeroTier interface. In some cases the default is "all", to check these use:
You can also change the "discover-interface-list" used, the key part is the /interface/list includes the ZeroTier interface.

iPhone ZeroTier client may be more limited...
Since iPhone come up a lot in my world, it important to note that the ZeroTier iOS client is more limited in the traffic it can pass – some Layer2 things are just not possible in iOS VPNs due to Apple/iOS limitation/restrictions. So stuff like multicast and broadcast may be problematic or not work on iPhone/iPad, what and why is more complex – more something to be aware of when troubleshoot ZeroTier. What this means for remote management is the Mikrotik's iPhone app may not work for discovery to find other Mikrotik routers over ZeroTier – but connecting to RouterOS using a ZeroTier assigned IP address should always work.
With Mac, Linux, and Windows ZeroTier clients, multicast/broadcasts does just work without issue, so these constraints don't come up. But assuming the ZeroTier interface was a "discoverable" in /ip/neighbors, winbox should show them in "Neighbors" tab at bottom of winbox (even using wine on Linux/Mac) on desktop/laptop with a ZeroTier client. Since ZeroTier at Layer2, either the MAC or IP address should work in winbox to connect via ZeroTier.

Mac-telnet also work over ZeroTier
Since ZeroTier operates at Layer2, even WITHOUT the right firewall rules... /tool/mac-telnet etc. should work between routers with ZeroTier. Similar to /ip/service, you'll want to make sure "/tool/mac-server" is configured to allow the ZeroTier interface. By default it's allows "all" (which include "zerotier1"), but may be restricted if someone choose to locked down the router, the defaults look like this (adjust as needed):

RoMON might be useful for routers without ZeroTier direct support
I guess ZeroTier isn't the only protocol on a Mikrotik with opaque tunneling and peer discovery, there's RoMON. NOT necessarily advising this one – RoMON scares me... If you don't know RoMON help says this:

RoMON stands for "Router Management Overlay Network". RoMON works by establishing an independent MAC layer peer discovery and data forwarding network. RoMON packets are encapsulated with EtherType 0x88bf and DST-MAC 01:80:c2:00:88:bf and its network operate independently from L2 or L3 forwarding configuration.

Basically it lets you use winbox, even if the RouterOS config is FUBAR. And does this at Layer2, since ZeroTier also operates at ethernet/Layer2, it should be able "extend" RoMON remotely. I don't want to cover the security risks, but ZeroTier is encrypted, so any RoMON at least be secured remotely. While not all Mikrotik support ZeroTier, all do support RoMON – so if you enabled both ZeroTier and RoMON on at least one router, that router could "RoMON Bridge" across a ZeroTier network to allow remote management, on non-ZeroTier Mikrotik with RoMON enabled.

I'd like to say RoMON "just work" with ZeroTier. But that may NOT be true... We're ignored discussing the flow rules so far. But with RoMON, ZeroTier likely NOT allow RoMON traffic through a ZeroTier network. The reason is the Flow Rules would block it since "ethertype 0x88bf" isn't allowed by the default ZeroTier flow rules.

ZeroTier's "Flow Rules" allow all IPv4, IPv6, and ARP traffic by default...
That sound like everything, and to an Layer3 IP network, it is. But at the ethernet level, there are many different "EtherTypes", which is why RoMON may not work. The "EtherType" comes up in other protocols on a Mikrotik – VLAN, 802.11x, MPLS, Wake-on-LAN, PPPoE, etc these protocol MIGHT have issues similar issue with ZeroTier's default Flow Rules at my.zerotier.com. Wikipedia has a good list of ethertypes:
https://en.wikipedia.org/wiki/EtherType#Values – so RoMON isn't alone here & so this may come up in other context when using ZeroTier.

The ZeroTier default flow include rules like:
While a double negative, what the drop rule WILL do is drop 0x88bf RoMON frames (and 0x8100 VLAN Ethernet frames, etc. too). Lots of way to fix this, but for these "speciality" ethertypes, you can also just have "accept" for them at the top of the rules and be done.

So if just add the following to the TOP of the Flow Rules for the Network at my.zerotier.com. You'd still want the defaults, so LEAVE the rest There is some docs about the Flow Rules next to where you configure on ZeroTier's Network config page. But way the flow rules work (& describe this):

• if action matches the criteria, rule evaluation stops (why we add an "accept ethertype 0x88bf;" at start and be done)
• the default action is deny (why the default includes a final "accept;" line) – so NO rules, means NO traffic
• if you use IP-based (Layer3) matchers, they are ignored for non-IP traffic (e.g. just "passthrough" to next rule)

But that's just a quick summary, check the docs for more details.





Routing and Bridging Options using ZeroTier

Taking a scenario from another forum post by @404Network :

-Site to site subnet connectivity LANA on Router A, to LANB on Router B. (both routers on zerotier)

This revolves around whether you want make the site-to-site connection via a Layer2+3 "bridged", or using IP (Layer3) "routing". Both are possible with ZeroTier, but if you really just need Layer3 subnet routed (e.g. LAN-A to LAN-B), this is where you'd want to make sure Wireguard (or IPSec/MPLS/OSPF/EoIP/etc) isn't a better solution for the site-to-site link. I say this since ZeroTier isn't going to be as FAST as Wireguard, and certainly more complex conceptually. So if a direct Wireguard connection between two Mikrotik is possible, that be preferable to ZeroTier complexity for simple L3 routing between two Mikrotik routers. Especially if both side have a fixed IPs, Wireguard is dirt simple.

But one of key benefits of ZeroTier is it's ability to "tunnel though" through a variety of NATs, with the protocol being smart enough to determine if local path is possible to avoid going to cloud if it's not needed – problem this is complex process and again the Mikrotik firewall/config may end up blocking things. You can view the path it does select however. Using winbox's new ZeroTier UI to look at the "Peer" will show the path being used to troubleshoot the tunnels used by ZeroTier. From the CLI, it's "/zerotier/peer/print" to see the paths. You can also use /ip/firewall/connection on winbox with a filter, typically ZeroTier will use port 9993.

So if break up this up into use cases, there are [at least] three flavors of "site-to-site subnet connectivity" that are relevant to ZeroTier:

• Site-to-site routing using ZeroTier with static routes
• Bridging a single LAN/VLAN to desktop/laptop using ZeroTier
• Bridging an entire VLAN trunk using ZeroTier

Likely better way to proxy the internet than ZeroTier...a "non-use case"
So I'm skipping the cases where you'd use a Mikrotik as default gateway on a ZeroTier network. e.g. using the ZeroTier to proxy internet from a mobile device though the internet connection of a Mikrotik isn't one I'd recommend for ZeroTier. For this case, you'd likely be better off using Wireguard if you could, so going skip describing this in detail. But largely it just adding a 0.0.0.0/0 route on to the network used on my.zerotier.com so it points the Mikrotik's ZeroTier IP address & every ZeroTier client that uses it has to enable "Use Default Gateway" & also config on Mikrotik to do the routing too. So, if you trying to use ZeroTier as VPN concentrator, likely better approaches on RouterOS to do this than ZeroTier. Not say not possible, or even potentially useful – just it won't be my first choice for this problem.

Queue the /queue
My ZeroTier needs for "remote winbox+ssh" can be measured in "kbps", so cannot offer much on QoS topics with ZeroTier. But once your routing/bridging ENTIRE networks – that's a lot more traffic, potential tunneling in "odd ways". And if you use QoS or /queue/tree... today, and apply ZeroTier to your parts of your network, you may have to revisit you queue design to control the new ZeroTier things. Similarly, although very curious, how ZeroTier tunnels and/or traffic combine with V7 queue like fq_codel and CAKE is also something I have not tried. Likely any queues won't effect whether ZeroTier "works", but do think the right /queue's could help with eventual "ZeroTier is slow!" problems. If/How? Dunno.

Use Case B: Site-to-site routing using ZeroTier with static routes


Adapted the RPi-approach from ZeroTier's KB "Route between ZeroTier and Physical Networks"


At a high level, ZeroTier is just a switch. So if two Mikrotik are both "members" of a same ZeroTier network, they are also directly connected from RouterOS's POV. Like any interface with an IP address in RouterOS, a "connected route" is automatically created in /ip/route for the subnet, based on the subnet mask in /ip/address. So the same is true for the ZeroTier interface. If you have two Mikrotik routers and just need IP routing, it's same as you would to enable IPv4 routing on an interface:
the "other Mikrotik's" IP subnet is dst-address in /ip/route, except for ZeroTier the gateway= is the ZeroTier IP address of "other Mikrotik", on the "first Mikrotik".

Basic IP routing stuff, just using "zeroteirX" instead of "etherX" – ZeroTier isn't any different for other IP routing cases...

But walking through this... Taking the LAN A to LAN B case, with two Mikrotik routers, each connected to a different WAN, with some default/QuickSet firewall. Let's assume both Mikrotik were connected to same ZeroTier Network (e.g. used same ZeroTier Network ID on BOTH routers). And the following IP subnet are used:
LAN-A IP address on ROUTER-A = 192.0.2.1/24
LAN-B IP address on ROUTER-B = 198.51.100.1/24
ZeroTier "IPv4 Auto Assigned" IP to ROUTER A = 10.1.1.100/24
ZeroTier "IPv4 Auto Assigned" IP to ROUTER B = 10.1.1.200/24

Since both sides have a shared Layer3/IP subnet created by ZeroTier, a static route can be used on each to route traffic using the ZeroTier created network. Since ZeroTier uses it own algorithms to determine the best path, we don't have to worry TOO much about the WAN side or how the tunnel got made. In the most basic case, all that's need beyond the connection to ZeroTier is a static route on EACH router. (Assuming both routers are were successfully connected to same ZeroTier network.)

So on ROUTER-A,
And reverse on ROUTER-B, Assuming a "QuickSet-like" config, also need the /zerotier/interface (e.g. default "zerotier1") in the LAN interface list:
/ip/firewall/filter could block the IPs or interfaces, if you've deviated too much from the defaults. But how to configure a firewall is a different topic. Also, the zerotier1 interface needs to be running and showing "OK" on both routers (e.g. you'd setup ZeroTier on both routers).

But that's all it should take for a user in LAN A with IP 192.0.2.101 to ping a device at 198.51.100.201 on ROUTER-B/LAN-B and get a response. The traffic be would carried by ZeroTier in a VL1+VL2 tunnel, and en-/de-capsulated into the zerotier1 interface on both sides, but all this happens "automatically" by the ZeroTier package.


More about the ZeroTier tunnels (e.g. /zerotier/peer)...
A spin on this case, is if those same routers were also connected via some local ethernet/VLAN trunk in addition to have a WAN connection. Since Mikrotik's default ZeroTier instance will listen on "all interfaces" on the Mikrotik, it should find a local ethernet path, and used that. So ZeroTier doesn't always use internet/cloud to connect a ZeroTier tunnel between the two routers. It can (and will) if that's the only path available.

The fact the ZeroTier protocol is always looking for a path (and establishes multiple paths to check) can be useful to easily add redundancy/resiliency. ZeroTier will automatically/internally choose a different path for its tunnel if one path fails. Even do stuff some nifty stuff with its tunnels. Like...while you make have only IPv4 traffic, ZeroTier may create a IPv6 tunnel (assuming IPv6 on a WAN) – you'd getting some potential performance advantages from IPv6 internet, while still dealing with more simple IPv4 routing. And if say the IPv6 tunnel stopped working, ZeroTier switch its type tunnel to IPv4 WAN/internet, etc., automatically.

Basically it's ZeroTier job to make sure zerotier1 has some path to join the ZeroTier network together – which is why you don't have to worry too much about how ZeroTier is creating tunnels - since it's abstracted away & presented as the ZeroTier interface to rest of RouterOS. The flip side here is while some people like magic, others know "magic" is generally just complexity in disguise. ZeroTier's "magic" has worked for me, but YMMV.

Note: By default ZeroTier is allow to use any interface to establish its tunnels. If you want to limit the possible path a ZeroTier VL1/VL2 tunnel takes, you need to set the interface= to be a list of interface instead of "all":
Now, if you change this, you may limit ZeroTier ability to find the "best" path. Again the ZeroTier>Peers view in winbox v3.32, or will also show the tunnels/path uses by ZeroTier. If you change interface=, this list may change – obviously no peers is a problem.

Since ZeroTier generally using UDP/9993, you can see ZeroTier tunnels in /ip/firewall/connection. Or, you can use /tool/torch on particular interfaces and filtering for 9993. Basically, the usual RouterOS tools should work the same with ZeroTier - you just need to know what to look for. Since possible ZeroTier may use other ports, using /zerotier/peer should help with that – look at the "path" for each peer will show the IP/port it uses (and other stats).


Use Case C: Bridging a single LAN/VLAN to desktop/laptop using ZeroTier


Loosely adapted from ZeroTier's KB "Bridge your ZeroTier and local network with a RaspberryPi" - except they recommend LPM e.g. /23 for "Managed Router", and /24 "Mikrotik/LAN Subnet", while below suggests changing the /ip/pool size - either work. LPM is certainly easier, UNLESS you use have adjacent IP subnets.


Let's take ROUTER-A from above, and assume we want LAN-A subnet bridged to a remote MacBook..
LAN-A used a 192.0.2.0/24 subnet.
The goal here be the MacBook, even though remote(/not local) to the Mikrotik, also be on the 192.0.2.0/24 subnet.
And specifically, be able to use mDNS/Bonjour across the internet via ZeroTier.

Let's assume ROUTER-A has some default configuration with a LAN being on a /interface/bridge interface.
Specifically /ip/address of 192.0.2.1 for ether2 & a /interface/bridge/port "ether2" is associated with "bridge1" for this example.

Setting this up on the Mikrotik side is pretty simple, assuming the ZeroTier interface is name the default "zerotier1", you "just"* add that as a bridge port: If you use vlan-filtering=yes and the LAN is actually a VLAN, you'd need to do set right things in /interface/bridge/vlan too – e.g. the zerotier1 port is list "untagged" on VLAN ID you want want to bridge over ZeroTier AND pvid= on the bridge port for zerotier1 also matches your VLAN ID. No different than an other ethernet[-like] interface on a VLAN Filtered Bridge.

This is the annoying part...when you want to bridge to any NON-Mikrotik ZeroTier client like Windows/Mac/Linux/smartphones/etc...
Before enabling bridging on the ZeroTier controller at my.zerotier.com, you very likely need to adjust the DHCP Address Pool range (under /ip/pool) for the LAN you want to bridge. I write up the why on this later. But basically ZeroTier doesn't use DHCP to assign an IP, even when bridged. So you need a range of your LAN addresses to use to assign to remote ZeroTier clients, while the Mikrotik DHCP Server will hand out different part to local LAN clients. To see what IP pool range is used: Let's assume I only need a max of 100 IP addresses for local LAN clients, and 50 for "bridged"/remote ZeroTier LAN clients. We can just reduce the pool size on the Mikrotik side, to leave a range we'll use on my.zerotier.com to assign to ZeroTier clients. So for example, let use .100 to .199 for Mikrotik-side LAN clients (we'll later use .200 to .249 on the ZeroTier side). The code for this look like (but you'll need to change to match you needs): There are other approach since to how to deal with too... I discuss underlying "DHCP issue" and why this is needed later in this post. Basically we need some block of IP addresses from the bridged Mikrotik LAN for ZeroTier to use to assign to other ZeroTier client to move on for now...

Now we can enable bridging and set the IP Range on the ZeroTier controller side at https://my.zerotier.com...

Assuming you're starting from the ZeroTier defaults, you need to modify the "Network" configuration at my.zerotier.com as followed:

• Under "Managed Routes" remove any existing ones using trashcan icon, so it says: "No managed routes defined."
• Then, add new "Managed Route" using Mikrotik's LAN network and IP address. In our example, "Destination" is 10.0.2.0/24 and "(via)" is 10.0.2.1, then tap "Submit"
• Under "IPv4 Auto-Assign" select "Advanced", then under "Add IPv4 Address Pools" add the range we allocated from the existing Mikrotik LAN. In our example here, "Range Start" be 10.0.2.200 and "Range End" be 10.0.2.249", then click "Submit".
• Move down the "Members" section on the page and find the Mikrotik's connection in there. If you're unsure, you can compare the Address or MAC address to what's showing under /zerotier/interface/print to find the right one.
• On that one (e.g. the ZeroTier "member" representing the Mikrotik router with the bridged to your LAN), hit the "wrench" icon next to the "Auth?" checkbox.
• Two more checkbox should appear in a purple section that appears. Check both boxes. So "Allow Ethernet Bridging" and "Do Not Auto-Assign IPs" should BOTH have a ticked/checkmark. The change takes effect immediately AND there is NO "Submit" button needed here.
• While looking that "Members", make sure all clients you want to bridge have checkmarks in the "Auth?" column. Importantly, only the Mikrotik with the bridge should be marked as "Allow Ethernet Bridging" – do not allowing bridge on any other Member! And, you'll likely want Auto-Assign IPs, since we just set that up above . So for all OTHER member, don't change the default settings.
• For bridging a single LAN, the default "Flow Rules" are fine. You can change them later if you want to restrict traffic flowing through ZeroTier, but that's an even more advanced topic.
• Similarly, leave the DNS setting alone. DNS is another topic... but see: https://zerotier.atlassian.net/wiki/spa ... Management for details. Key details is if you set DNS at my.zerotier.com, the client must use the "Allow DNS Configuration" option (which is NOT a default setting).

If you followed along, and connect using ZeroTier, you should have your single LAN bridged, and since it's bridged/Layer2, stuff like multicast/broadcast, including Bonjour/mDNS, should just work.

If you use VLANs and vlan-filtering=yes...
You can adapt this approach for VLANs by setting the PVID for zerotier's bridge port and /interface/bridge/vlan to set zerotier port to the right VLAN as UNTAGGED. That's about it...

And you map each VLAN to own seperate ZeroTier Network too...
Both ZeroTier and Mikrotik allow multiple ZeroTier networks/interfaces, so another potential architecture is you'd have a one-to-one map between each VLAN to a unique ZeroTier network. You may want to do this is you want to have different potential members for any remote ZeroTier client.

To do this...basically you repeat the same approach. You start with creating a 2nd (or more) /zerotier/interface, just binding the Network ID for a "New Network" from my.zerotier.com, for each VLAN you want to expose the ZeroTier. Then, you'd add each of the new /zerotier/interface's as bridge port & assigned to right VLAN you wanted to bridge to ZeroTier.

Since the remote ZeroTier network connects to only a single VLAN, it's still be marked as untagged – just with each ZeroTier interface/network being untagged in a different VLAN. The approach allows you to extend the same VLAN topology remotely – without requiring a VLAN trunk across the internet (our last topic). ZeroTier client's on Mac/Windows/etc, that are expecting untagged traffic, still be able to reach any VLAN simply by selecting a different ZeroTier network. ZeroTier client allow using multiple ZeroTier networks at the same time, so this may be pretty useful to get all your VLANs exposed on desktop/laptop/mobile. You can even then "switch VLANs" using ZeroTierOne taskbar icon.

You still need to deal with aligning ZeroTier IPv4 "auto-assign" for each VLAN (e.g. adjust the DHCP pool being one approach from above). e.g. the "DHCP pool splitting" would have to repeated for every VLAN you exposed via ZeroTier – no different than the single LAN case, just multiple times. But let's come back to the DHCP, to explain the why...

How ZeroTier assigned IP/router is critical to understanding bridging...
One of the "subtleties" in ZeroTier is how it manages member (e.g. client) IP address assignment. The key thing know is: the ZeroTier client does NOT run DHCP client by default. Instead IP address and routes are entirely controlled, and "pushed" by ZeroTier by what's set for the specific network settings on my.zerotier.com & their client software does this internally WITHOUT using DHCP.

It be way easier if the ZeroTier client had an option to "Use DHCP for IP addresses" - it DOES NOT have that. When you say "Allow Managed Address" that translate as: copy the IP address that configured on my.zerotier.com for my device under "Members" to this client. If you uncheck that, you'd get no IP address.

ZeroTier itself doesn't care about IP address...
Technically, a ZeroTier member can also have no IP address, or have multiple IPs assigned or each member could be in different IP subnet – what's set as an IP under "Members" in my.zerotier.com for a particular device is what's pushed (along with any routes configured under "Managed Routes" for the same network) to ZeroTier VPN client apps. This is because ZeroTier does NOT care about Layer3 routing or IP addresses, so it's happy to assign what ever you want – even if the rest of the network couldn't use/route it. In fact, all the settings in the my.zerotier.com largely effect how IP addressing/routing are configured on members (e.g. clients) – but ZeroTier does not route by itself or really enforce much about IP/routes, similar to how an ethernet switch doesn't either.

Running your own DHCP client to get IP from ZeroTier-bridged Mikrotik /ip/dhcp-server...
While, on a Mac or Linux, nothing stops you from running your own DHCP client on the ZeroTier interface (since it really is like an ethernet interface). This actually does work. But I don't recommend this approach since it take special configuration on each client. On Mac, you can do this by finding the ZeroTier interface name using "ifconfig" then enabled DHCP on by using:
On Linux, adding a DHCP client on the ZeroTier interface would be specific to the distro you're using, on most:

BUT on iOS (iPhone/iPad with ZeroTier client) it is simply NOT possible to add a DHCP client – so the IP address used by the ZeroTier client on iOS ALWAYS is what's IP is set under "Members".

On Windows and Android, I dunno if it's how/if possible to force DHCP client.

Certainly cases where you do want Mikrotik to provide DHCP over ZeroTier, just the specifics vary by OS so not the focus here. Basically this is why I recommend just splitting the /ip/pool to "leave room" for a range ZeroTier can use: it's just simpler. In the ZeroTier Free plan, there is a limit of 50 members, leaving 200 IPs for the Mikrotik in typical /24. Changing your network to use a larger subnet (say /23 here) be a similar approach to deal with the lack of DHCP via ZeroTier's clients.

Now on Mikrotik, you DO have a DHCP Client readily available – so a site-to-site BRIDGE between two or more Mikrotik's makes the DHCP part much easier. /ip/dhcp-client is under RouterOS's control (unlike on iOS and other ZeroTier clients).

Users (e.g. ZeroTierOne clients) can block auto-assign IP too
We just talked about the my.zerotier.com side of IP address/route assignment. But the ZeroTier client also has four options that also related. I'm using Mikrotik's option names - Mac/Windows/Linux/etc all use slightly different terms – but all work roughly the same across ZeroTier client platforms:

• allow-managed - if true, ZT-managed IPs and routes are assigned
• allow-global - if true, ZT-managed IPs and routes can overlap public IP space
• allow-default - if true, network can override system default route (full tunnel)
• "All DNS Configuration" (not supported on RouterOS) - if true, allows my.zerotier.com some control over the client's DNS settings

These are client-only flags, so there is nothing in the cloud (my.zerotier.com) to change these AFAIK. What that also means is if you add a Managed Route for 0.0.0.0/0...and allow-default=no, then route will NOT be inserted until the user themselves changes that ZeroTier client setting.

Keep in mind RouterOS ZeroTier support is also just another "ZeroTier client". So these setting can be control using:
Use Case D: Bridging an entire VLAN trunk using ZeroTier

e.g. re-create the same VLANs on a remote RouterOS device, and use the same VLANs at two different site, via the internet+ZeroTier.

You can use ZeroTier to bridge an entire VLAN trunk (e.g. multiple tagged VLAN) between ZeroTier members – largely by adding the desire VLANs as tagged on a ZeroTier bridge port. The benefit of using ZeroTier is you'd then have the same VLAN trunk available over the internet to any authorized ZeroTier network client (that also support VLAN trunks, like Mikrotik or Linux).

The only complexity in VLAN tagging/trunking in ZeroTier is you need to change the "Flow Rules" on my.zerotier.com to allow the 0x8100 ethertype. This is critical for VLAN to work over ZeroTier!

I actually think this use case is pretty handy for some folks. @pcunite already has a great post on "Using RouterOS to VLAN your network" so if you've gone down that road, bridging ZeroTier allow you extend any of those topology (on Mikrotik hardware with ZeroTier support, that is...) relatively securely over the internet.

You need be familiar with how "bridge vlan filtering" works before trying this, ZeroTier doesn't make VLANs on a bridge easier...

The default ZeroTier "Flow Rules" DO NOT ALLOW VLANS
So we need to change them a bit. One fix is just adding a line under the "drop" rule in the default flow rules for the ZeroTier network, so it looks like (leaving the rest of the flow rules alone, in particular the final "accept;"):
It's that "ethertype 0x8100" is allowed that's important. Since the flow rules are yet-another-filter-scripting-language, you can write the rules however you'd prefer too. So you could just as easily use "accept ethertype 0x8100;" at the top of the file instead.

TAG the ZeroTier bridge port on any VLAN you want to trunk over a ZeroTier Network
Adapting the single LAN bridging example above, the key differences for "VLAN Tagged Trunks" are:

• You need to be using "bridge VLAN filtering" (or) e.g. vlan-filtering=yes on the bridge interface with the ZeroTier bridge port
• Instead of "untagging" the zerotier1 bridge port (as in the single LAN case), you'd add it as "tagged" interface on any VLAN in any /interface/bridge/vlan you want. Obviously you can tag "zerotier1" on as many VLANs as you need/wanted bridged to the ZeroTier network.
• Since both sides of ZeroTier VLAN trunk are in Mikrotik bridges, BOTH/ALL Mikrotik routers need to be marked to "Allow Bridging" under "Members" on my.zerotier.com
• The /zerotier/interface on each router should be using the same ZeroTier network ID and be authorized and "OK".
• A trunk should have no untagged traffic. Since "trunk" may be confusing, what I mean is ALL packets are VLAN tagged. (Otherwise, it a "Hybrid" port...)
• If it's VLAN trunk, only things that understand VLAN tags, should be members of any ZeroTier network that's using VLANs. You could have other things, I guess, but it wouldn't make sense to say have a Windows ZeroTier client connect to a trunk over ZeroTier (e.g. typically you don't have client devices use an ethernet trunk port).

Basically both routers just need to use the same tagging & VLAN ID for the zerotier1 bridge port, on EACH for router. e.g. the /interface/bridge/vlan configuration need to match on BOTH sides (at least for the VLAN being carried over a ZeroTier network do). Finally, you can then untag an VLAN as needed locally, on either side, onto actual other bridge ports to actually use a bridged VLAN - even over the internet.

No IP Address or DHCP tricks required for VLAN Trunking
ZeroTier doesn't need ANY IP address or routing defined... So you can skip all the changes to /ip/pool or other DHCP tricks for VLAN trunk – since DHCP is inside a VLAN and tunneled to another Mikrotik, it work fine over ZeroTier. So you can ignore the "Managed Routes" and "Address Pool" stuff in my.zerotier.com. You could delete any "Managed Routes" and disable "Auto-Assign from Range" IF there really is no untagged traffic over the ZeroTier network being used for trunking – but you can leave the IP alone (even IPv4 auto-assigned) too since it shouldn't matter if BOTH Mikrotik port require VLAN tags.

I guess you can use both untagged traffic and tagged over same ZeroTier network...e.g. a hybrid port BUT...
This is more complex since you now DO still have to with the IPv4 auto-assign/DHCP stuff that's been described at length above... Since with vlan-filtering=yes, it generally recommended to "tag everything on ingress to bridge interface", it be should be trivial to AVOID having untagged traffic on this "ZeroTier VLAN Trunk" – and that's what you should strive for... But if you really want a hybrid port, you'd need to DO both:

• add a "Flow Rule" to allow ethertype 0x8100 for VLAN (see above)
• since you do have untagged traffic, you need deal with ZeroTier's IP configuration from the single untagged LAN case too (e.g. split a DHCP pool from the single VLAN or take another approach)

Totally doable, but if you ever change the VLAN ID/PVID that's untagged over a "Hybrid ZeroTier VLAN Trunk", then you'd also have to change all the ZeroTier/Mikrotik IP/DHCP stuff to match the new untagged IP subnet everytime you changed the untagged port on the ZeroTier VLAN Trunk.

In all cases involving tagged VLAN over ZeroTier, it's critical the Flow Rules allow 0x8100 ethertype – that is NOT the ZeroTier default & VLANs won't work over ZeroTier unless you add that.

Also only ONE Mikrotik should being running a DHCP Server for any VLAN. Basically one Mikrotik is the "default router" for the VLAN, while other Mikrotik connect via to a "ZeroTier VLAN Trunk" network should be more like the "switch" (e.g. only with tag/untagging in the bridge). Except, if your VLAN also uses VRRP, where you could have multiple Mikrotik acting as a router, but that another topic.


Scripting: Using RouterOS /tool/fetch to call the ZeroTier Central API

So far ZeroTier has been managed using their website, https://my.zerotier.com to do stuff like:

• Create new ZeroTier network
• Set IP "Managed Routes"
• Control "IPv4 Auto-Assign"
• Authorize a "Member" to use the ZeroTier network (e.g. checkbox under Auth?)
• Set "Allow Bridging" on a "Member"

I do NOT have a Mikrotik script to do ANY have that... But it is POSSIBLE to do all those via the ZeroTier Central API, their docs describe the operations in the REST API:
https://docs.zerotier.com/central/v1/

As a REST API, it can be invoked using /tool/fetch on a Mikrotik. You will need an "API Key" from your ZeroTier account page, https://my.zerotier.com/account to use it. Now you'd have to store the apikey some place, hopefully safe - this part is up to you.

Below is short example script that calls into the ZeroTier Central from a MikroTik RouterOS. It only calls ZeroTier's /status API, with an "apikey" – if the key is valid it prints your "display name" from the ZeroTier account. It actually should work if your familiar enough with RouterOS script to run it.

Obviously printing your name isn't that useful, but feel free to extend or adapt this script as you'd like:

And if you got here, hope this was helpful.

v7.1.1+a 19FEB2022 - initial revision
v7.1.1+b 25JUL2023 - minor update - Mikrotik does support acting as ZT controller now

Awesome, It will take me weeks to get through it but it will be well worth it. Great effort!

My router is set up in bridge mode with "use IP firewall" checked and the recommended forward and input rules for ZT set in the firewall. The ZT interface is reachable and passing traffic and the router even pops up in Winbox with the ZT address so this seems to work. However I am unable to reach anything else in the network. I've checked "allow ethernet bridging" and set a route local_lan/24 via router_zt_address just as I would for a regular router. Thoughts?

I was trying to catalog all the interactions. But got me: there is the bridge's "Use IP firewall" setting & that certainly could have side-effects. Should work however. But I don't have an instant answer how that interact with ZeroTier – as I don't regularly use that option, even without ZeroTier.

Since winbox works, that indicates the Mikrotik has a Layer2/Ethernet connection to the ZeroTier network, so your firewall is at least allowing the ZeroTier tunnels. But the IPv4 traffic carried by ZT would be subject to the firewall rules, and seemingly what may be blocking the ZT client from getting to the LAN. The IP addresses/routing in BOTH the Mikrotik and my.zerotier.com has to aligned be the other big thing.

Be good to know what happens when you disable "Use IP Firewall", that might help to isolate the problem is even related to the firewall/bridge. But if you're using "Use IP Firewall", you likely understand how it works than me. A network diagram or better description of your topology might help. e.g. Are they VLANs, multiple bridges, other things in play here. Also troubleshooting from the client more may help. e.g. can the PC use `arp` to find MACs on the Mikrotik bridge – if so, then it routing or firewall configuration somewhere.

"What ZeroTier solves is LTE networks are often behind a CGNAT. And with a CGNAT, basically you can enable any VPN protocol in RouterOS since you can't open listen ports. Obviously solvable in a lot of ways. But with ZeroTier you just add few config lines on the Mikrotik & then be able use winbox"

"you just add few config lines"
witch one ?

don't work very well for me
lot's of timeout (after few seconds) when i'm connect via zerotier with winbox....
hapac2 (ros 7.1.3) out of the box with firewall default + package zerotier and 2 line in top of firewall (accept input and forward in interface zerotier)

thank for your help

Some thoughts...

Outside ZeroTier, you might want to make sure the MTU of the LTE interface is right. Mikrotik seems to default at 1500, but LTE often much lower, typically 1428. Similarly on MTU, you want to make sure ICMP is allowed in via LTE in the firewall filter (this is a typical the default).

If you ping the ZT address of your router from the box you're running winbox, does it show timeouts/drops/etc? If ping looks good, you may want to add ZeroTier to "LAN" interface list.

Or, perhaps trying using the MAC address in winbox to connect, and see if that also drops.

You can also check out the tunnel connection using the CLI, at least most/some of them, using a few commands (fell free to adapt):
What's important is the "recvd" and "send" in the ZeroTier peer out. That shows the last time it reached that peer – annoyingly ROS will show "active" even if it's "stale" and hasn't used a peer in a while. Since most connections use port 9993, you can find those in the firewall connections. In the firewall connection, you should see traffic flowing.

Hello

Change MTU lte
icmp is allowed
zerotier in interface lis LAN

wan 1
zerotier hapac2 = 192.168.196.2
[admin@MikroTik] > /zerotier/peer/print
Columns: INSTANCE, ZT-ADDRESS, LATENCY, ROLE, PATH
# INSTANCE ZT-ADDRESS LATENCY ROLE PATH
0 zt1 62f865ae71 201ms PLANET active,preferred,50.7.252.138/9993,recvd:3s824ms,sent:9s26ms
1 zt1 778cde7190 PLANET active,103.195.103.66/9993,recvd:20m3s639ms,sent:16m1s828ms
2 zt1 cafe04eba9 63ms PLANET active,preferred,84.17.53.155/9993,recvd:24s30ms,sent:9s26ms
3 zt1 cafe9efeb9 201ms PLANET active,preferred,104.194.8.134/9993,recvd:3m227ms,sent:9s26ms
4 zt1 6ab565387a 144ms LEAF active,preferred,35.224.180.120/44917,recvd:49s176ms,sent:9s26ms
5 zt1 3511376444 LEAF active,preferred,88.120.171.15/61075,recvd:3s975ms,sent:4s23ms
6 zt1 1806a73032 162ms LEAF active,preferred,212.58.85.86/28648,recvd:1s43ms,sent:1s109ms
7 zt1 ca736a82c0 121ms LEAF active,preferred,37.165.132.77/39090,recvd:1m5s216ms,sent:9s26ms
8 zt1 36a90fbaee LEAF active,preferred,77.131.35.34/9994,recvd:1m20s607ms,sent:14s33ms



wan 2
zerotier rb4011 = 192.168.196.1

[admin@MikroTik] > /zerotier/peer/print
Columns: INSTANCE, ZT-ADDRESS, LATENCY, ROLE, PATH
# INS ZT-ADDRESS LATEN ROLE PATH
0 zt1 62f865ae71 238ms PLANET active,preferred,50.7.252.138/9993,recvd:32s906ms,sent:2s842ms
1 zt1 778cde7190 197ms PLANET active,preferred,103.195.103.66/9993,recvd:48s97ms,sent:2s842ms
2 zt1 cafe04eba9 97ms PLANET active,preferred,84.17.53.155/9993,recvd:2s692ms,sent:2s842ms
3 zt1 cafe9efeb9 229ms PLANET active,preferred,104.194.8.134/9993,recvd:2s612ms,sent:12s937ms
4 zt1 6ab565387a 167ms LEAF active,preferred,35.224.180.120/44917,recvd:17s868ms,sent:17s868ms
5 zt1 3511376444 105ms LEAF active,preferred,88.120.171.15/58819,recvd:20s217ms,sent:7s892ms
6 zt1 219e5200f5 137ms LEAF active,preferred,37.169.59.236/35028,recvd:7s133ms,sent:2s842ms
7 zt1 ca736a82c0 1ms LEAF active,preferred,192.168.88.6/54074,recvd:792ms,sent:791ms
8 zt1 36a90fbaee 142ms LEAF active,preferred,77.132.56.187/9994,recvd:2s655ms,sent:2s655ms
9 zt1 d2c338e6a9 12ms LEAF active,preferred,192.168.88.10/9994,recvd:17s730ms,sent:17s746ms


ping from 192.168.196.2 to 192.168.196.1 is ok
ping from 192.168.196.1 to 192.168.196.2 ko (

"Debugging" the tunnels isn't easy. Looking at your peer, you can see some rather long times since the last packet. That could be a few reasons. I highlighted them in red/orange below:

wan 1
zerotier hapac2 = 192.168.196.2
[admin@MikroTik] > /zerotier/peer/print
Columns: INSTANCE, ZT-ADDRESS, LATENCY, ROLE, PATH
# INSTANCE ZT-ADDRESS LATENCY ROLE PATH
0 zt1 62f865ae71 201ms PLANET active,preferred,50.7.252.138/9993,recvd:3s824ms,sent:9s26ms
1 zt1 778cde7190 PLANET active,103.195.103.66/9993,recvd:20m3s639ms,sent:16m1s828ms
2 zt1 cafe04eba9 63ms PLANET active,preferred,84.17.53.155/9993,recvd:24s30ms,sent:9s26ms
3 zt1 cafe9efeb9 201ms PLANET active,preferred,104.194.8.134/9993,recvd:3m227ms,sent:9s26ms
4 zt1 6ab565387a 144ms LEAF active,preferred,35.224.180.120/44917,recvd:49s176ms,sent:9s26ms
5 zt1 3511376444 LEAF active,preferred,88.120.171.15/61075,recvd:3s975ms,sent:4s23ms
6 zt1 1806a73032 162ms LEAF active,preferred,212.58.85.86/28648,recvd:1s43ms,sent:1s109ms
7 zt1 ca736a82c0 121ms LEAF active,preferred,37.165.132.77/39090,recvd:1m5s216ms,sent:9s26ms
8 zt1 36a90fbaee LEAF active,preferred,77.131.35.34/9994,recvd:1m20s607ms,sent:14s33ms



wan 2
zerotier rb4011 = 192.168.196.1

[admin@MikroTik] > /zerotier/peer/print
Columns: INSTANCE, ZT-ADDRESS, LATENCY, ROLE, PATH
# INS ZT-ADDRESS LATEN ROLE PATH
0 zt1 62f865ae71 238ms PLANET active,preferred,50.7.252.138/9993,recvd:32s906ms,sent:2s842ms
1 zt1 778cde7190 197ms PLANET active,preferred,103.195.103.66/9993,recvd:48s97ms,sent:2s842ms
2 zt1 cafe04eba9 97ms PLANET active,preferred,84.17.53.155/9993,recvd:2s692ms,sent:2s842ms
3 zt1 cafe9efeb9 229ms PLANET active,preferred,104.194.8.134/9993,recvd:2s612ms,sent:12s937ms
4 zt1 6ab565387a 167ms LEAF active,preferred,35.224.180.120/44917,recvd:17s868ms,sent:17s868ms
5 zt1 3511376444 105ms LEAF active,preferred,88.120.171.15/58819,recvd:20s217ms,sent:7s892ms
6 zt1 219e5200f5 137ms LEAF active,preferred,37.169.59.236/35028,recvd:7s133ms,sent:2s842ms
7 zt1 ca736a82c0 1ms LEAF active,preferred,192.168.88.6/54074,recvd:792ms,sent:791ms
8 zt1 36a90fbaee 142ms LEAF active,preferred,77.132.56.187/9994,recvd:2s655ms,sent:2s655ms
9 zt1 d2c338e6a9 12ms LEAF active,preferred,192.168.88.10/9994,recvd:17s730ms[/color],sent:17s746ms

But after looking at this, I realized you'd post here too: viewtopic.php?p=918457#p918367 – I think you should start a new topic, with details/diagram/etc & what you're seeing where & what you've tried already... Hard to go from "winbox drops over zerotier" to then continue with multiple WANs/routers...

I can offer a script function $ZTPEERS that goes through the the ZeroTier peers to find associated firewall connections & do a /tool/ping-speed to get some data on the quality of the link.

The function goes though the output of /zerotier/peers to find the peer destination IP/port. It then runs /tool/ping-speed for 5 seconds to some additional measurement on the latency (while it shows "speed", ping-speed is NOT accurate since it just based on a few ping times, useful to compare the "relative speed of peers"). It also tries to find associated tracked connection in /ip/firewall/connection – this is useful since it shows you the src-address, and packet/byte counts for each.




Code is below, you can cut-and-paste into terminal to load the $ZTPEERS function, then call it using just $ZTPEERS in the Terminal window. If you want to save it, create a scheduled script to run it at startup to load the function on reboot.

Thank you very much Amm0 but i think it just don't work between 2 LTE connection.

Work great between LTE <-> static IP but i d'ont have a stable connection between 2 LTE connection...

I have a zerotier network with 3 members - Mikrotik router, one iPhone and one MacBook Pro. I am trying to remote access the Mikrotik router. The iPhone uses an LTE internet connection and receives an IPv6 address from the ISP, the MacBook Pro is on hotel wifi and has an IPv4 address. The MacBook can connect to the Mikrotik router via HTTP or SSH but the iPhone can only connect via HTTP. If I try SSH or the Mikrotik iPhone app it times out. I can only think it has something to do with IPv6.

Upgrading to 7.2.3 did not fix this issue.

Hi, thank you for the very informative write-up. I successfully implemented Use case C and am able to get on my home LAN when I am on the road. Since my MikroTik/ZT instance is "behind" my pfsense router, I don't have any firewall rules setup on the MikroTik. Do people who use this configuration also have firewall rules on the MikroTik? If so, what rules are recommended?

Thank you!

Just updated to routeros 7.2.3 and trying out Zerotier

1. If i disable the Zerotier interface and enable it again (from GUI), it does not get an IP.
2. If i delete and manually create the zerotier, again it does not get an ip.

It only works if i create the interface via cli - zerotier/interface/add network=1d71939404912b40 instance=zt1

Any thoughts?

It only orks when i create the interface via cli - zerotier/interface/add network=1d71939404912b40 instance=zt1

Any thoughts?

I think the first attempts to configure via GUI are available as of 7.3b33

Ok. i will try again on next update

Multipath *does work* on Zerotier on RouterOS and it is enabled by default. It seems to be using balance-aware. Each WAN connection needs a default route and policies to use them with Zerotier. I am using bonded LTE connections on ZT ROS to make this post.

Each WAN connection needs a default route and policies to use them with Zerotier. I am using bonded LTE connections on ZT ROS to make this post.

can you elaborate a bit on configuring the policy to use two WAN (ISPs) connections with the zerotier? I was planning a simple failover scenario (using gateway ping method) but curious if I could use both ISPs at the same time with multipath.

Each WAN connection needs a default route and policies to use them with Zerotier. I am using bonded LTE connections on ZT ROS to make this post.

can you elaborate a bit on configuring the policy to use two WAN (ISPs) connections with the zerotier? I was planning a simple failover scenario (using gateway ping method) but curious if I could use both ISPs at the same time with multipath.

I would like to know about this as well? I'm testing ZT on several hAP's with a wired WAN, wireless and als LTE dongle as WAN connections. Although it does looks like it's working - I'm just guessing here that the default route/distance in the MT are being use by ZT to determine the best available WAN route. Is there anything else you can share with us on how you came to this conclusion? Would be much appreciated!

How to make Zerotier an Entry Node ( Reverse Exit Node)

We need to access a remote Linux zerotier machine, from our office room without installing zerotier on BYOD's.
The zerotier setup may be exactly reverse concept of managed route (Exit node). BYOD : Windows Laptops Without Zerotier Installed.

Without installing Zerotier on BYOD's, they have to access Remote zerotier Linux machine.
We dont want to allow Remote machine to be accessible from outside of our office Room. So we can not install zerotoer on BYOD's.
I am able to ping between 172.29.11.11 and 172.29.22.22 from both sides.

Please suggest the configuration procedure/Steps, including any firewall/iptables setup.

Thanking you.

Hi all, thanks @Amm0 for this interesting and detailed guide!

There is a point not very clear to me;
Supporting either UPnP or NAT-PMP on your network can greatly improve performance by allowing ZeroTier endpoints to map external ports and avoid NAT traversal entirely.

this means that, in case of public IP available, UPnP should be enabled for interface ZeroTier?

NAT-PMP is still not supported by RouterOS?

Endpoint-Independent NAT could help?
Thanks!

Let me reiterate one point first: If you can enable IPv6 on your WAN, ZT will preferably use IPv6. This avoids any hole punching on the Mikrotik for IPv4 ZT VL1 tunnels. Obviously not always possible, so to answer the questions...

There is a point not very clear to me;
Supporting either UPnP or NAT-PMP on your network can greatly improve performance by allowing ZeroTier endpoints to map external ports and avoid NAT traversal entirely.

this means that, in case of public IP available, UPnP should be enabled for interface ZeroTier?

The point was about if you have another router between the Mikrotik with ZT and the internet. Say an ISP router in-between you can't remove, and it's that route that may support uPnP or NAT-PMP – and on that non-Mikrotik router, you might want to make sure either one of those protocols is enabled to help ZeroTier enable a more direct path though the intermediate router. e.g. sometimes other routers have some UI/checkbox to enable UPnP and/or NAT-PMP support, which be good to enable to aid ZT path discovery.

If the Mikrotik has a direct route to the internet (even if behind a NAT), no additional steps are needed (beyond the firewall filter and/or NAT rules discussed in docs and above).

Code: Select all

/ip upnp interfaces add interface=WAN type=external
/ip upnp interfaces add interface=ZT type=internal

That should be unnecessary if the goal is allow Mikrotik ZT tunnel to make a tunnel to the ZT network.

What the above rules allow... and if the Mikrotik is the default gateway for a ZT network e.g. the ZT network at my.zerotier.com has a 0.0.0.0/0 route to the Mikrotik's ZT IP address. Is if say you have some desktop/PC/laptop/smartphone that is a member of the same ZT network...and the client has "Allow Default Router Override" set so the Mikrotik is this desktop route to the internet... Then the above UPnP rule will allow that desktop, connect to same ZT network as Mikrotik, to punch holes in the Mikroitk firewall. UPnP is typically done by games and SIP clients, so that allow it happen from members of the ZeroTier network, who were using the Mikrotik as the default gateway to internet.

A different use for uPnP being enable on the Mikrotik is if the Mikrotik LAN if it has users that also have ZeroTier clients.
Since these non-Mikrotik ZT client on the local LAN may use additional/different ZT networks than the Mikrotik, those may have different paths needed. If uPnP is enabled for the LAN interface list, then those desktop/smartphone ZeroTier client will attempt UPnP to form their own paths to various ZT networks. Since UPnP is deterministic on the assigned port it likely slightly quicker to establish tunnels since it doesn't have to test using various port. Basically UPnP will short-circuit more complex hole punching techniques. As a bonus, if LAN ZT client's tunnels will show up as dynamic entires in the firewall with comments – so the ZT tunnels from LAN become more "visible" to the admin. Otherwise, an admin finding VL1 tunnels in a network is non-determistic (see the complex script in #2 above to find peers in firewall on the Mikrotik, and it's impossible to do for LAN client's ZT tunnels).

NAT-PMP is still not supported by RouterOS?

No. But the ZeroTier client on Mikrotik may try it when establishes paths to next-hops beyond the current Mikrotik running ZT. Similar to above, it just an option ZeroTier's ZL1 tunnels may try when establishing paths.

Endpoint-Independent NAT could help?

Code: Select all

/ip firewall nat add action=endpoint-independent-nat chain=srcnat out-interface=ZT protocol=udp
/ip firewall nat add action=endpoint-independent-nat chain=dstnat in-interface=ZT protocol=udp

Hmm. I'm of the thought that if action=endpoint-independent-nat is needed for something, someone else made some poor design decision someplace. So not thought about its interaction with ZT. But ZT has its own complex logic to deal with NAT hole punching, so thinking it's best not mess with their logic by remapping things. It really shouldn't be needed since ZT will likely do something reasonable without ANY kind of port forwarding etc.

Now since Mikrotik didn't support being a ZT controller when I wrote this article, I cannot say how rules involving "action=endpoint-independent-nat" would interact with ZT controller side... But since ideally ZT use one port (default 9993), just making sure that port was available on a controller should be all that's need. So worse case, you might need some dst-nat upstream in more complex topology. But the how ZT controller work be another article...

Excellent as always, @Amm0!
- I was referring to an MT router with a direct route to the internet (no NAT, Public IP assigned), assigning ZT interface to LAN interface-list was enough for me
- Unfortunately no IPv6 (still...)
- I can confirm that UPnP enabled on ZT interface is creating a dynamic roule in NAT, when using a device with Allow Default route enabled

Hi all, trying to learn more about ZT, read couple times the very well written guide by @Ammo (thanks again!), what I would like to achieve is in the middle between use case C and D, bridge single VLAN to another MikroTik device, to be able to reach an IoT device connected to Router B (behind NAT) from main router A.
This is what I did;

- both devices with a same configuration and same VLANs
- ZT as bridge port with VLAN ID
- ZT untagged in bridge VLANS
- added correct route in ZT portal
- selected Bridged on both router and not assigning any IP
- disabled DHCP on router B

Following all the instructions from use case C and, with ZT client on Windows, I'm able to ping the IoT device connected to Router B that got IP from DHCP on Router A.
What's strange and probably due to a misconfiguration is, doing a speedtest on router B is very slow and is using public IP from Router A.
Allow-Managed=yes|no makes no difference.
Without ZT added to the bridge on Router B I was not able to ping device IoT from router A.

Spent couple days without success, should be nice to have the same behavior that I find on Windows client on Router B.

Make sense? Any suggestion?
Many thanks

The "middle" starts getting complex ...

If I get the basic setup... you have some VLANs on one router...and different set of VLANs on another router... For simplicity, let assuming "MAIN", "GUEST", "IOT" on each. And MAIN on Router A wants to access IOT on Router B.

My first question is do you want the IOT on Router A to be the same network Router B? OR, do you want to keep Router A and Router B's IOT VLAN as different networks?

If it the later... and you just one of those VLAN to be same on both side, Option C right – IOT VLAN is same on both side (bridged) & any device can "see" any other device across the WAN. The access from your MAIN VLAN is really up to the firewall rules... but assuming MAIN is allowed to access local IOT VLAN, and ZT bridge that network to other side's IOT VLAN (using Option C). But essentially you end up one "IOT" VLAN that same network on both sides.

If you want the former (e.g. different IOT LANs on both sides, just Router A can access Router B IOT VLAN only)... In this case, you essentially might want to create a new VLAN interface called "IOT-B" and add it to Router A & untagged the ZeroTier to that IOT-B VLAN (e.g. set PVID). On Router B just need to set the PVID= on the ZeroTier bridge port to the IOT VLAN on Router B. This essentially add a new VLAN to Router A. Still same rough steps as Option C...

"Option D" describes a VLAN "trunk", but that means that non-Mikrotik ZeroTier clients cannot connect in this mode – the client don't untag VLANs, so that's approach isn't quite right.

But you may have gotten the "Option E" ...

"I guess you can use both untagged traffic and tagged over same ZeroTier network...e.g. a hybrid port BUT...
This is more complex since you now DO still have to with the IPv4 auto-assign/DHCP stuff that's been described at length above..."

It's actually not that complex... but it is context specific & the more you understand how Bridge VLANs work, the more sense it makes...

Anyway, if you can provide your configs that's help. And better explanation of what VLANs should map where...

OK so, an example of config (I don't have the export with me now but it's pretty much based on MT KB and pcunite's guide :
Only difference between two Routers is Router A is using PPPoE with public IP and Router B is using DHCP-Client with NATted IP for WAN.

My first question is do you want the IOT on Router A to be the same network Router B? OR, do you want to keep Router A and Router B's IOT VLAN as different networks?

Don't want to create confusion but, at the moment they share the same network for semplicity, Router A has devices connected only on LAN and GUEST, Router B is using IoT.

If it the later... and you just one of those VLAN to be same on both side, Option C right – IOT VLAN is same on both side (bridged) & any device can "see" any other device across the WAN. The access from your MAIN VLAN is really up to the firewall rules... but assuming MAIN is allowed to access local IOT VLAN, and ZT bridge that network to other side's IOT VLAN (using Option C). But essentially you end up one "IOT" VLAN that same network on both sides.

This is what I did, following your instructions with the only difference that ROUTER-A setup is also replicated to Router B, bridging ZeroTier as untagged port.
It's working, what I can't understand is why all the traffic from Router B is "routed?" to Router A (like a VPN tunnel), where I mostly intrested accessing IoT devices connected to Router B from LAN on Router A.

"Option D" describes a VLAN "trunk", but that means that non-Mikrotik ZeroTier clients cannot connect in this mode – the client don't untag VLANs, so that's approach isn't quite right.

As soon as everything works I don't mind having tagged or untagged traffic as ZeroTier clients are not needed (tested also "Option D" and finding the same behaviour of "Option C"), just want to connect those MikroTik Routers

Hope everything is clear, thanks @Ammo!

I got lazy writing this towards the end is also part of the problem here – I was going to take the @pcunite's VLAN as an example, but it was already pretty long.

I think your problem is that you're using "Allow Managed" and/or the bridged VLAN's IP address are duplicates on both side...once bridged. But rather than guess perhaps a more simple "checklist" for ZeroTier bridging be useful for how it works with vlan-filtering=yes (i.e. "@pcunite-style") Bridge VLAN scheme...


On the Mikrotik side,

- make sure "Allow Managed" addres is unchecked/="no" on both routers – this is critical... Otherwise, it will create a "connected route" for any ZT assigned address (same as RouterOS does for any other interface when an IP address is added to interface). You don't want this since RouterOS already has a routing table & the bridged VLAN already has an IP for each router (e.g. the one assigned the IOT VLAN) – so you don't need ZT adding a 2nd IP to same interface (which is what happens with "Allow Managed Address" is enabled/checked)

- make sure both routers have a unique IP address assigned to the VLAN on each side (e.g. 10.0.0.2/24 on RouterA and 10.0.0.1/24 on RouterB) & the associated DHCP server network for IOT should use that particular router's IOT VLAN's IP as the gateway (e.g. 10.0.0.1 or 10.0.0.2). Otherwise, there be IP address conflict once ZT is connecting both side & default gateway given to clients may be wrong

- bridge port for zerotier1 interface uses the "right" frame-types= and pvid= So for bridge one VLAN, frame-types= is "admit-only-untagged..." and pvid= match what you want as the network shared via non-Mikrotik ZeroTier clients. To bridge multiple VLANs...whether you use frame-types=admit-all or frame-types=admit-only-vlan-tagged depend on if you want non-Mikrotik clients, i.e. need a hybrid port (frame-types=admit-all) if desktop/smartphone will connect & you want have VLAN tagged traffic over ZT network. Or, if ZT is used ONLY for trunking VLANs across a WAN, then frame-types=admit-only-vlan-tagged is what you'd want — in either hybird or truck case...just like any other hybrid/trunk bridge port for ethernet...you'd need to define the zerotier1 port as tagged= in /interface/bridge/vlan for the tagged VLAN(s) & adjust the flow rules as described in Option D in my main writeup

- only one side should have dhcp-server enabled since with ZT bridging. Otherwise, you'd end up with multiple DHCP servers run on same LAN...not necessity going to break anything, but potentially confusing/error-prone — e.g. DHCP server configuration have to be identical & you'd want to ensure "Conflict Detection" is enabled in /ip/dhcp-server on both.

- dhcp address pool (e.g. /ip/pool) for ZT-bridged VLAN should be adjusted so that there is some range for ZT clients to use (that be assigned by ZT to desktop/smartphone). In theory, it doesn't matter if DHCP server use the "Conflict Detection" but better just to prevent ZT from using same IP address in the same subnet – the main writeup at top describes how to adjust the pool in more detail

- beyond the scope here... but VRRP could used so that if the WAN is lost, each RouterOS's IOT VLAN could operate independently during WAN outage (since dhcp-server listen on the VRRP, and use VRRP address as the gateway).


On the ZeroTier controller side (my.zerotier.com),

- in the routes, remove any existing ones & add one for the the bridge LAN subnet by using just 10.0.0.0/24 with no gateway – this essentially defines the scope for non-Mikrotik clients. For bridging, you don't need any other routes. Although you can add them if you wanted to allow ZeroTier desktop/smartphone clients access other networks, as they need routes defined in ZT controller/portal) – again you do NOT want "Allow Managed" on RouterOS since it [should] knows these routes already.

- whether it's an access port or hybrid port, the auto-assign IP address range should be same subnet as what's bridged – but critical it uses the same subnet as the VLAN that's being bridged (e.g. say 10.0.0.201-10.0.0.249 but whatever as long as different/non-overlapping range with the /ip/pool). Since a true VLAN trunk (e.g. nothing untagged) will not allow non-VLAN aware ZT clients, the auto-assignment should set to something that's NOT any subnet you used, to avoid routing issues – but it doesn't matter since there be only tagged traffic & ZT will passes those along in Layer-2, but it has NO VLAN helpers or anything to "untag" any of them

- both Mikrotik should have the bridging enabled under the "Members" section using the gear (as again described more fully at top of thread) – nothing else should have that defined

- if you have no tagged[/i] VLANs passing over ZT (e.g. frame-types=admit-only-untagged-and-priority on the bridge port, and correct pvid= set), then you can ignore the "Flow Rules". But for Hybrid or Trunk ports (no untagged traffic), you'll need to make sure you allow the VLAN "ethertype" in the flow rules – this is not the default & will block any tagged VLANs from flowing through the ZeroTier network.

Followed your instructions a small step forward...
In my specific case VLAN's IP address were duplicates on both side...(I feel so dumb!)

1st "Bridged Test":
[on Routers]
✓ Created ZeroTier interface on both Routers (A - B) with Allow Managed unchecked
/zerotier interface add allow-default=no allow-global=no allow-managed=no instance=zt1 name=ZeroTier network=""
✓ Created a Bridge with all ports as my last post assigning PVID / VLAN-ID and frame-types=admit-only-untagged-and-priority-tagged
✓ Unique IP address assigned to the VLAN on each side
on Router B /ip address set IoT address=10.0.0.2/24 and ...dhcp-server gateway=10.0.0.2
✓ I ended up enabling DHCP Snooping /interface bridge set Bridge dhcp-snooping=yes (losing Bridge Hardware Offload) instead of having DHCP anabled only on Router B (where the IoT devices are connected, in my case), this is probably not correct but was the solution to avoid all traffic passing trough the other Router.
✓ "Conflict Detection" was enabled by default in /ip/dhcp-server, DHCP Address Pool adjusted as suggested, just in case...
[on ZeroTier portal]
✓ Checked "Bridged" on both Routers ID under "Members"
✓ Removed any "Route"
✓ Disable assigning IP
X I'm able to ping IoT device from Router A connected to Router B via ZeroTier network with bridged interface only from the Router console... probably due to picky firewall rules, this must be figured out, at lest is something, better than nothing

2nd "Unbridged Test":
[on Routers]
✓ Created ZeroTier interface on both Routers (A - B)
/zerotier interface add allow-default=no allow-global=no allow-managed=yes instance=zt1 name=ZeroTier network=""
✓ Added ZeroTier as LAN in interface list
[on ZeroTier portal]
✓ Created a new Newtork, defining Route [ 10.0.10.0/24 via (LAN) =Empty ] and Auto-Assign IPv4 Range [ 10.0.10.3 - 10.0.10.10 ]
✓ Assigned "Managed IPs" for both router just for convenience ( Router A=10.0.10.1 | Router B=10.0.10.2 )
X Still able to ping IoT device connected to Router B from Router A, only from console, using ZT PC clients is working as expected (previous ZT interface created for test was disabled)

I'll continue trying to solve the problem to be able to reach device on Router B from PC connected to router A via ZeroTier network, what are the advantages from using "Use Case C" (my 1st Test) over "Use Case A" (my 2nd Test)?

Thanks again.

On 1st "Bridged Test"...

If both routers can ping the other route's VLAN 30 IP from the console, ZeroTier should be working.
Assuming it bridged correctly (seem like it is), then should just follow firewall rules for VLAN 30...
So not sure it's the firewall – benefit of bridging... it should follow the VLAN of it's assigned pvid=, so NEW firewall rules should be needed.

Re DHCP, you may have to the zerotier1 bridge port as "Trusted" for DHCP to pass, that might be simpler.

If you posted a sanitized version of both router's config, that help here. And perhaps output of /routing/route/print.


On 2nd "Nonbridged Test"... (aka "Routed" approach)

This one I'm not sure be is useful IF one is coming from "@pcunite VLAN" setup.... Since it involves using IPv4 routing ("subnets"/"prefixes"), vs thinking in VLANs and Layer-2 "interfaces"... And this approach also involves creating new firewall rules, too...

The basic theory is using a SINGLE ZeroTier network that's connected all other routers. That gets ALL router on the same IP subnet – e.g. so the routers can "ping" each other. But that doesn't help any other traffic like VLAN – since you need have /ip/routes to that use the fact all routers are connected via ZT... One way is do it statically — that mean using "/ip route add" with the other router's ZT address as the gateway= and dst-address= the IP range/prefix associated with remote VLAN – for any subnet you wanted.

Instead, you actually define routes to some/all your VLANs in my.zerotier.com. Setting the routes there gets more tricky. But these ZT defined routes will be added by checking the "Managed Address" (which includes any routes at my.zerotier.com). Useful, but by default problematic... since you have to set the "Route Distance" property of zt1 instance to at least 2 (or higher). If you don't set route-distance=2 on "zt1" instance... problems can happen. More specifically, very easy to creat ECMP/load balance route to a VLAN when route-distance=1 and your routing an existing VLAN...

And while ZeroTier will allow multicast (e.g. mDNS/SSDP/etc) over its network (unlike ZeroTier), /ip/route isn't going get deal with multicast without using IGMP-Proxy/PIM/etc. (IF multicast is needed) e.g. LAN traffic is NOT "bridged", but "routed" via ZeroTier's switch to another router on the "ZeroTier Cloud Switch" (e.g. what's defined at my.zerotier.com).

My thought is when you start with VLANs...ZeroTier just extends these – so "non bridged" case changes the model you're using in a routed approach, when the firewall/etc is designed around VLAN and interface-list.

Appreciated your assistance, thanks @Amm0!
Spent days trying to figure out what's wrong, had mixed and inconsistent results.
Today I noticed that the ARP table was not populated with IP and MAC from device at the other side (Router B) - Bridge Test, adding it helped to succeed with Ping instead of "unreachable".
It may need some sleep and start from scratch following each step carefully.

Your guide and all institutions to Bridge (as Untagged VLAN or Trunk) are pretty clear, just thinking in case of a second remote device (Router C) will be added in the future, bridging the same IoT VLAN, having just one subnet (10.0.0.1 R-A | .2 R-B | .3 R-C) /24 is suggested or a smaller one (e.g. /26 + adding Routes) will help avoiding conflicts?

Zerotier bridging works great.

But as soon as i join two MK-routers within the same local lan to the same bridged ZT Network,
i loose connection to both!
When I DENY one of those routers on the ZT-controller, ping comes back as usual.
There must be somthing extra needed...

What i wanted to achieve ist if my main routers Internet goes down, the (second) LTE router should allow ZT access.
It works if i tic the "Authorize" in the ZT-controller then, but id would be nice to be up all the time

Zerotier bridging works great.

But as soon as i join two MK-routers within the same local lan to the same bridged ZT Network,
i loose connection to both!
When I DENY one of those routers on the ZT-controller, ping comes back as usual.
There must be somthing extra needed...

What i wanted to achieve ist if my main routers Internet goes down, the (second) LTE router should allow ZT access.
It works if i tic the "Authorize" in the ZT-controller then, but id would be nice to be up all the time

That can work. On both Mikrotik routers bridged to ZeroTier, do you have "Managed Address" unchecked on the ZertoTier? And at my.zerotier.com BOTH router are marked as "Bridged" in the options next to the router in "Members" section (exposed by wrench icon)?

Another possibility is the bridge has loop detection...so possible RSTP might be tripping...and it generally harmless to disable. To do this, you go your Bridge interface on each router, and select "None" as the "Protocol Type" in the STP tab. Alternatively, you can try using a higher path cost (say path-cost=20) for the zerotier bridge ports in each router (in Bridge > Ports > STP).

One question, are the routers physically connected via ethernet? That changes the story a bit...since it's be even more likely STP is coming into play.

One thing you can do is look at the bridge and see the forwarding state ("role") of the port. The zerotier1's bridge port role should not be "disabled", when you have both ZT interface up... if it is STP is kicking in....

Several points are not entirely clear. 1. About the compatibility of Mikrotik and LTE versions. If Mikrotik is not an Arm architecture, but MMIPS, but an LTE modem is inserted into it, will such a combination work?
2. If I understand correctly, without the central ZeroTier cloud it is impossible to create your own network. For example, if I created such a network by registering on their website, and then their website stopped working, or for some reason the Internet fell apart into pieces in several countries, will the ZeroTier network I created continue to work without the main cloud?
3. Without a special application, the device will not be visible on the ZeroTier network. Does this mean my IoT devices will remain unavailable?

1- no. ARM/ARM64 devices only. That point is quite clear.

2. It may, it may not. Depends if a path can be found using the parts still left operational
But you can always host your own zerotier cloud controller ...

3. Yes and no. ONE part on that network needs to be zerotier-capable. From there on you can hop to other devices within that same network segment.
I use as example an RB5009 with my customer to ROMON into devices on 28 locations all connected via EOIP over MPLS to that RB5009.
Only that RB5009 has Zerotier installed (and Wireguard, I like a backup plan).
With that same customer I have an AC3 LTE, also using Zerotier (it's not part of that MPLS network), so I can always reach it, even when VDSL line is down (also there, wireguard backup).

2. It may, it may not. Depends if a path can be found using the parts still left operational
But you can always host your own zerotier cloud controller ...

This is true & the feature come out AFTER I wrote his article. The story is bit more complex — since there are still root servers — but it's pretty well described in Mikrotik's docs:
https://help.mikrotik.com/docs/display/ ... Controller

3. Yes and no. ONE part on that network needs to be zerotier-capable. From there on you can hop to other devices within that same network segment.
I use as example an RB5009 with my customer to ROMON into devices on 28 locations all connected via EOIP over MPLS to that RB5009.
Only that RB5009 has Zerotier installed (and Wireguard, I like a backup plan).
With that same customer I have an AC3 LTE, also using Zerotier (it's not part of that MPLS network), so I can always reach it, even when VDSL line is down (also there, wireguard backup).

It's the term "without special attention" that confuses the question.

If ZT interface is bridged to desired VLAN/etc, then IoT/etc devices can span physical sites and operate the same.

The main caveat (and what's described in article) is the some attention MAY be need in the "Flow Rules" — but ONLY IF your devices use non IP/IPv6 protocols. For example, RoMON does use a different ether-type, so by default ZT controller would block...but small change to flow rules would allow RoMON, then it work just fine.

Also, it's also fair to mention that latency sensitive things may not be ideal to run across ZeroTier Network, so that be another area of "special attention". And ZeroTier likely add more latency than some point-to-point VPNs (e.g. WireGuard). Essentially ZT trades latency (using potentially slower indirect path) to work across more diverse/restricted networks (e.g. CGNAT's).

Hi,

I am running ZeroTier on a MikroTik ARM router. The router is behind a T-Mobile 5G gateway with CGNAT - no IP6 provided. I added a basic ZT Network with a managed route added for my internal network 192.168.0.X via the MT ROuter address on the ZT network. I added the ZT interface to the interface list (LAN) as recommended in the documentation. I then turned on a cell phone hotspot and connected my laptop. I am running a windows ZT Client. I can ping but I am getting many dropped packets. In fact sometimes I have to try the ping 5 or 6 time before it works and then it will work for the next 5 tries. If I try to use Remote Desktop to a server on my internal network - it start to work but then times out very quickly.
My T-Mobile connection is very solid and it is not a latency issue. My speedtest pings are about 32ms.

Any thoughts. If you have any ideas please provide detailed instructions - I am not super skilled on MT or ZT.

Thanks

The router is behind a T-Mobile 5G gateway with CGNAT - no IP6 provided. I added a basic ZT Network with a managed route added for my internal network 192.168.0.X via the MT ROuter address on the ZT network.
[...]
Any thoughts. If you have any ideas please provide detailed instructions - I am not super skilled on MT or ZT.

You might want to try change the "route distance" on the "zt1" instance under zerotier > instance tab to "10". Otherwise it's possible 192.168.0.0/24 route might be load balanced if the ZeroTier added one's distance (default is 1) match the distance of that internal/Mikrotik version route.

But you have a double NAT situation, worse still with MULTIPLE ZeroTier client trying to use the default ports....that gets problematic... If there is any way to get IPv6 enabled on you mobile account, that be best solution – may not be possible, but that be best. Is there an options for a "DMZ" on the T-Mobile gateway, or can you map ports on the T-Mobile gateway?

Hi - changing the instance tab to 10 seems to have made it more reliable. It is problematic the first try or two and then just works solid. What does that do?
Regarding T-Mobile - I don't think they will provide an IPv6 and there is no control over the router they provide to create a DMZ. I will call today to find out.

Thanks

Hi - changing the instance tab to 10 seems to have made it more reliable.

If you look at /ip/route, you'll notice that the 192.168.0.0/24 is likely repeated (one for bridge/etherX, one from "zerotier1"). When there are multiple routes for same subnet, the LOWEST distance wins. BUT if there are two subnet, at same distance...then routing is split 50%/50%...so changing the "zt1" to distance of 10 means it has a lower priority.

But it could just be port mapping through the CGNAT isn't conflicting & not the route distance. Time will tell.

Since ZeroTier can use IPv6 to establish the tunnels to the ZeroTier planets/root servers... you wouldn't have the same potential for port mapping issues. But the underlying ZeroTier tunnel can still use IPv4 inside, nothing else has to change IF you can get T-Mobile to enable IPv6.

Hello Amm0,

Great Post, very detailed and a goldmine of information.

I have a scenario where I need to create a Site to site Layer 2 tunnel, either between two mikrotiks, or a mikrotik and Linux server.
for now I tried only the MT-Linux version, but it does not really work well.

I think I got my configs right, as I can ping, get UDP broadcast traffic, and it would work as I expect but ....

By default the connection without any traffic has maybe a few packets, just a little idle traffic of 100 kilobytes, then suddenly out of nowhere, boom. Tx jumps to 150Mbps, while Rx is mostly 0. I can see this on the LAN Bridge and the ZT interface, ...
This goes on for a few seconds, and it drops back to idle, 4.4 kbps Tx, 3-4 p/s Tx, mostly 0 bps Rx.
Then back at 150Mbps after 10 seconds, and so on, in completely random time intervals.

This happens if there is only the WAN port connected, and Zt is active, once the Zt interface is deactivated the effect stops. ...

I wonder if this might be related to some version mismatch, as the Mikrotik is reported to be ZeroTier version 1.10.3 while every other OS reports ZeroTier version 1.12.2.

It looks like MT is trying to push something through the tunnel, like crazy. On the other end of the line, this is not received.

Any ideas, or what to check, what to look at, I would appreciate it. Thanks.

I have tried with the MT-MT version, where both sides use 1.10.3, and got the same ...

@amm0, would it be possible to ask you a question about my relatively simple setup? If so, I would love to reach out. Thanks!

Very nice work @Amm0

But the following is also nice to know
Tailscale vs. ZeroTier: Side-by-Side Comparison

@amm0, would it be possible to ask you a question about my relatively simple setup? If so, I would love to reach out. Thanks!

Feel free to post your needs/question(s) and any relevant config.

But the following is also nice to know
Tailscale vs. ZeroTier: Side-by-Side Comparison

I think that's article is a pretty good summary. I'd agree with most of it, specifically

Tailscale is different than WireGuard in many ways, but it’s a better comparison to ZeroTier than WireGuard due to the way that it’s set up and configured, as well as its functionality.

I kinda focus on the interaction with RouterOS features in this original article here - than a comparison of VPN/protocols. But do agree Tailscale is more similar to ZeroTier since both have some centralized control of clients/members/peers & routing, vs "pure WireGuard" that really just a peer-to-peer protocol within Linux kernel that OTHER things can make sure for a complete VPN solution (i.e. with Tailscale and Mikrotik's new Back-to-Home be examples of providing some higher-level control/policy to WireGuard.

But the high-level decision between WG/tailscale/BTH/etc vs ZeroTier is if "ethernet-like" Layer2 link is needed. The need for Layer2/ethernet-like often comes up with devices that use multicast or other discovery protocols. Conversely, pure/unicast IP networks would likely be better severed with some WireGuard-based protocol – since WG/tailscale/BTH is more efficient than ZeroTier, especially if all traffic is Layer3/IP. And Tailscale add all the "policy management" stuff that's often needed in corporate/enterprise/education markets - that ain't in ZeroTier (or RouterOS). While ZeroTier "policy" is more limited to it's "flow rules", which is akin to doing policy management using Mikrotik's bridge/firewall filters – so Tailscale makes a lot more sense when you need user-level access control than ZeroTier. And Tailscale add user-authentication between just ZeroTier's simplistic (and ID-based) membership concept.

Additionally, Mikrotik's ZeroTier support has not expanded significantlly since I wrote this article*. So some of ZeroTier's newer features/configuration are not available on Mikrotik. Specifically low-bandwidth mode, peering control, and load balancing options. And CHR/X86 lacking ZeroTier remains a significant limitation since now some other solution is still needed for a VPN/SD-WAN/etc. WG is available on ALL Mikrotik platforms, and there are ways of integrating plain WG links into Tailscale - so that be another advantage from a Mikrotik POV.

I still maintain as an out-of-band management protocol for Mikrotik's (at least ones that's support ZT) is hard to beat – especially with RoMON being enabled. Other use cases, does require more thought into picking between ZT and Tailscale or something else - starting with if there any Layer2 or multicast needs since that's not easily fixed AFTER a VPN/SD-WAN deployment.

Additionally, in the Mikrotik context...older routers might still be better off with IPSec-based things since some older routers with limited CPUs do support hardware encryption – while both ZeroTier and WG are going increase CPU load since they cannot make use of hardware encryption in older routers (that support IPSec in hardware). So that be another important consideration if dealing with older hardware.

*in fairness, Mikrotik did add the option of running you're own controller since my original post, but I'm less sure of the use cases.