MikroTik

Hi

I own two sites that I want to keep connected just like as if it was a single LAN (all devices in site1 can communicate with all devices in site2, maybe even share a single DHCP server and a single DNS server, though this is not necessary in case it's too much trouble).
I have a few ideas on how to achieve this, but it would mean a lot of testing and probably a lot of deceptions. The thing is, I don't have the global vision on what would be the best method to achieve what I want. That is why I'm asking before I start working:

What is the best way to connect two mikrotik routers through the internet in order to build a single LAN (or as close as possible to a single LAN)?

Sidenotes:
1) Both mikrotiks sit behind ISP routers, so, both WAN addresses are private addresses. This can not be circumvented. Both ISP routers allows DMZ, one allows bridge mode.
2) The traffic at one site should not be all routed through the other. Only the LAN(s) traffic.

Thanks in advance for your insight!

Zerotier

Zerotier

I have not been down that road before. Never even installed the package. I might take a look, but I would prefer something more confortable for an "old tech" guy.....

If both were behind NAT and otherwise inaccessible due to private IPs, I think you need another server with public IP as a gateway. What kind of traffic passes through the LAN between clients? If its not remarkably high-throughput maybe you can get away with cheap VPS in your region. That way you can use "old-school" solution like OpenVPN, etc or if you're in the v7 branch, Wireguard/OpenVPN UDP.

Zerotier is (almost) low-configuration, low-management solution so try to look into that first.

When your MikroTiks are behind ISP routers that allow "DMZ" (and where that really works properly), you can enable that feature and use any of the available VPN technologies available in the router.
For LAN-LAN connections I would suggest setting up a GRE tunnel with IPsec encryption (just add a GRE tunnel interface and enter the IPsec secret to auto-configure IPsec), that will be hardware-accelerated on most MikroTik routers (contrary to those other protocols).
Then you add a /30 IP network to the GRE tunnel endpoints (e.g. 10.0.0.1/30 on one end and 10.0.0.2/30 on the other end) and you add a static route towards the other end's LAN range "via" the other end's GRE endpoint address.
This way it will work without fiddling with NAT exceptions and it will be fully transparent for any traffic routed between the sites.

Add one to the mix
Wireguard. A lot faster then all the rest.

Wireguard would have been my first response but based on the OPs post "as if it was a single LAN (all devices in site1 can communicate with all devices in site2," I thought he was looking for a more EOIP over wireguard solution as in able to conduct traffic at layer 2 like broadcasting not just layer3.

...based on the OPs post "as if it was a single LAN (all devices in site1 can communicate with all devices in site2," ...

Different subnets can also reside in one single LAN.
But yes, if discovery, broadcast, etc is needed, EOIP needs to be added on top.
Otherwise wireguard on itself would be enough.

The (rudimentary) testing I did with AX Lite for various VPN protocols, showed zerotier underperforms immensely comparing to WG.

Add one to the mix
Wireguard. A lot faster then all the rest.

100% better solution is WireGuard just as @holvoetn stated plus WireGuard Security is second to none without sacrificing performance ...

Thanks to all answers.
Been checking zerotier and I think it's not for me. Signing up? Closed source software to install on my devices? Relaying packets through some site? Warnings of slowness? The f__k?
I'm leaning on wireguard (been using it as RW for my laptop and my phone), maybe I'll try EOIP over it later to better tune it as a single LAN, but anyway, I can live with 2 LANs...
Will get back to you to tell you of my success or to ask for further help in setting it up.

For Wireguard... Someone has to have a public address.

We have the office, bosses home and warehouse.

Office is behind carrier grade NAT from a Wisp.
Home is behind starlink.
Warehouse is the one with a public IP.

Warehouse is set up as the server... The other sites connect to it.

Road warriors also connect to the warehouse.
Once connected... Any site is reachable via IP. All layer 3.

For LAN-LAN connections I would suggest setting up a GRE tunnel with IPsec encryption (just add a GRE tunnel interface and enter the IPsec secret to auto-configure IPsec), that will be hardware-accelerated on most MikroTik routers (contrary to those other protocols).

I also use a GRE + IPSec for a L2 tunnel.
Wireguard + IPoE is more CPU intensive, this is what I see using two hEX.
Maybe without IPoE the Wireguard tunnel is lighter, I have to check...

For Wireguard... Someone has to have a public address.

We have the office, bosses home and warehouse.

Office is behind carrier grade NAT from a Wisp.
Home is behind starlink.
Warehouse is the one with a public IP.

Warehouse is set up as the server... The other sites connect to it.

Road warriors also connect to the warehouse.
Once connected... Any site is reachable via IP. All layer 3.

Well, what if the Warehouse was natted? Couldn't it be done? I think it could.....
Anyway, one of my ISP routers allows mode bridge. That, as far as I understand, means I can have the mikrotik behind that ISP router with a public address on its WAN. I've tried it and it shows a public WAN address.
The other allows DMZ. Not really as a DMZ server, as far as I understand. It "kind of" works as a bridge. The mikrotik has a natted WAN private address, but the all the ISP router (no firewall, no NAT port rules, no nothing) is doing is driving all traffic to the mikrotik WAN port. It's just replacing 1 public/1 private IP. There's nothing else connected to the ISP router, I'm using it as a fiber/ethernet converter. I've been long using my RW wireguard server precisely on this mikrotik (RB5009) without caring about the fact that it stands behind the ISP router and has a private IP on its WAN interface. I just forget about that and everything works fine, just like I had the RB5009 with the public IP on its WAN interface.

And now the problems have began. In the second site (new site) I connected an hexS router that I had lying around. I managed to setup the ISP router in bridge mode and I obtained a public address on the mikrotik's WAN address. Since I was going away on vacation, I first wanted to make sure I could connect to this second site as easily as I connect to the first one (RB5009), so I could work on the site-to-site link during the vacation dead time. Therefore I decided to setup a RW (for my laptop) wireguard interface on the hexS, thinking I would deal with the site-to-site wireguard interface later. I just could not do it. After making sure I had not done any mistakes for 1000 times, I searched the internet and came up with this post which mirrors my problem:

viewtopic.php?t=183234

The issue (same as mine, including logs) was not solved.
So, my new question is: Do I have to buy another RB5009 for the second site? Is there an insurmountable problem with wireguard and hexS?
I ran out of time to even setup an ovpn to the hexS. So, I'm away of both sites until September, I can't run any tests on site 2 now.

humbfig

If the warehouse router was DMZ'd behind a GATEWAY (Modem+Router+WiFi), THAT HAS A PUBLIC IP ADDRESS...

Yes that would work.

As long as one end always has a public IP, normal WG is fine for Layer 3. To make it one LAN, you'd need to use GRE, EoIP (without IPSec) or VXLAN interface too to carry the Layer-2 ethernet traffic between the sites.

Now if both sides have private/NAT address and without some DMZ option enabled...(and assuming the ARM-based router) ....that's when you'd need to use ZeroTier for the tunnel.... OR the brand-new BTH WG feature (also need a GRE/EoIP/VXLAN L2 tunnel too). The later BTH feature will tunnel encrypted WG traffic via servers at Mikrotik. But depending on your location, BTH may actually be even slower than ZeroTier. Why it's important to get a public IP if you can...as y'all have both more and faster options...than these ones.

And, for completeness...if BOTH sides had a public IP... just EoIP with IPSec enabled is dirt simple and efficient (e.g. you set the pre-shared key, put remote public IP as dest on both, bridge EoIP to desired LAN). But EoIP with IPSec requires BOTH side have public IP. (While EoIP using WG doesn't need public IP, since EoIP uses the WG address as the tunnel dest IP)

humbfig

If the warehouse router was DMZ'd behind a GATEWAY (Modem+Router+WiFi), THAT HAS A PUBLIC IP ADDRESS...

Yes that would work.

I see now what you meant. Never crossed my mind that an ISP would assign you a private IP.....
In my Country people call bad names to ISP's that don't provide bridge mode on their routers. I guess we don't know how lucky we are.....

As long as one end always has a public IP, normal WG is fine for Layer 3. To make it one LAN, you'd need to use GRE, EoIP (without IPSec) or VXLAN interface too to carry the Layer-2 ethernet traffic between the sites.

Now if both sides have private/NAT address and without some DMZ option enabled...(and assuming the ARM-based router) ....that's when you'd need to use ZeroTier for the tunnel.... OR the brand-new BTH WG feature (also need a GRE/EoIP/VXLAN L2 tunnel too). The later BTH feature will tunnel encrypted WG traffic via servers at Mikrotik. But depending on your location, BTH may actually be even slower than ZeroTier. Why it's important to get a public IP if you can...as y'all have both more and faster options...than these ones.

And, for completeness...if BOTH sides had a public IP... just EoIP with IPSec enabled is dirt simple and efficient (e.g. you set the pre-shared key, put remote public IP as dest on both, bridge EoIP to desired LAN). But EoIP with IPSec requires BOTH side have public IP. (While EoIP using WG doesn't need public IP, since EoIP uses the WG address as the tunnel dest IP)

I always have a public IP everywhere. The worst it can get to me is when I have to do NAT on the ISP router.
I knew I could do it with IPSec. And it would fit since both routers (RB5009 and hexS) have hardware for it. But I had a bad experience a few years ago with my IKEv2 RW. It worked for a few months, then it stopped working and I spent too much time trying to figure it out. Gave up and settled for an ovpn, and later to wireguard, which is quite good. So, I wanted to do it using WG, at least for a start (just layer 3). But the hexS must have a problem with WG.... I did find it strange when I first connected it (it was laying around without use for 2 years) showing v6.49 as the latest version....

Hear you about IPSec, but for a Layer-2 tunnel the nice part about EoIP is that the IPSec stuff is really just a checkbox and setting a pre-shared key. Since you'd already need another protocol with WG to get ethernet. But WG + GRE is another option if you want keep wireguard but bridge a layer-2 LAN. But EoIP for layer is dirt simple, it deals with all the IPSec config for you.

If just layer-3 IP routing is all that's need, well then it's just WG

And on the hEX is the channel set to "upgrade", that's how you get from V6 to V7. Once at V7, you can switch the /system/package channel back to stable or testing.

Hear you about IPSec, but for a Layer-2 tunnel the nice part about EoIP is that the IPSec stuff is really just a checkbox and setting a pre-shared key. Since you'd already need another protocol with WG to get ethernet. But WG + GRE is another option if you want keep wireguard but bridge a layer-2 LAN. But EoIP for layer is dirt simple, it deals with all the IPSec config for you.

If just layer-3 IP routing is all that's need, well then it's just WG

And on the hEX is the channel set to "upgrade", that's how you get from V6 to V7. Once at V7, you can switch the /system/package channel back to stable or testing.

Heard you all. Read a few things. I will try the route GRE + IPSEC, since it seems to fit better what I want and also my hardware. I'm ready for the IPSEC headache, though you say it will be easy.
So, since I'm locked out of the hexS till September, now it's time to enjoy vacationing.
Thanks to all!

I will try the route GRE + IPSEC

Well, I'll buy the aspirin. Do think the secret is, well, ipsec-secret= set (either EoIP or WG) if Layer2 tunnel is what's needed and you have public IP at both ends. If only Layer3/IP, no argument with WG there

I will try the route GRE + IPSEC

Well, I'll buy the aspirin. Do think the secret is, well, ipsec-secret= set (either EoIP or WG) if Layer2 tunnel is what's needed and you have public IP at both ends. If only Layer3/IP, no argument with WG there

I might be confused. I thought GRE and EoIP were the same thing.....
I have an argument against layer3 WG. hexS doesn't seem to work.....

EoIP interface is layer-2. EoIP use the GRE protocol internally, but inside the GRE packet is an Ethernet frame.

Mikrotik also has a different "GRE interface", but that is only Layer3/IP, so similar to L3 WireGuard in that it's an IP packet inside the tunnel.

Both EoIP and GRE support the ipsec-secret= which should automatically setup the IPSec stuff. But you do likely want to try EoIP, not GRE...

(and IPSec should be hardware offloaded on hEX)

EoIP interface is layer-2. EoIP use the GRE protocol internally, but inside the GRE packet is an Ethernet frame.

Mikrotik also has a different "GRE interface", but that is only Layer3/IP, so similar to L3 WireGuard in that it's an IP packet inside the tunnel.

Both EoIP and GRE support the ipsec-secret= which should automatically setup the IPSec stuff. But you do likely want to try EoIP, not GRE...

(and IPSec should be hardware offloaded on hEX)

yep, thanks. I got it.
Anyway, I'm on vacation until September and cannot contact the hexS. Meanwhile I've got bigger problems. My RB5009 suddenly stopped accepting incoming traffic from the WAN. I can contact it because I setup a zerotier. But that's another thread.....

Hi

Finally back from vacation, I tried the EoIP tunnel and could not make it work. As far as I understand I need 2 public IP's on both routers and one of the routers has a private address (DMZ) because the ISP router does not allow for bridge mode.
I setup a l2tp connection instead, which works mostly fine. Devices connected to R1 can talk to devices connected to R2. The problem is that my RW wireguard to R1 can not reach the R2 through the l2tp interface.
The R1 wireguard interface does not forward to the l2tp interface.
My setup is this:

R1
WAN DMZ 192.168.1.69
LAN 192.168.27.0
l2tp tunnel 172.16.1.2
wireguard 192.168.28.1
routes: dst 192.168.43.0 gw 172.22.1.1

R2
WAN public IP
LAN 192.168.43.0
l2tp tunnel 172.22.1.1
routes: dst 192.168.27.0 gw 172.22.1.2

I tried the EoIP tunnel and could not make it work. As far as I understand I need 2 public IP's on both routers and one of the routers has a private address (DMZ) because the ISP router does not allow for bridge mode.

How I think it should be done:
-You need to setup wireguard (only 1 side needs to have a real public IP)
-Set an IP address on both ends of that connection
-Use those 2 addresses to setup EOIP
-Add EOIP to bridge on both ends

I tried the EoIP tunnel and could not make it work. As far as I understand I need 2 public IP's on both routers and one of the routers has a private address (DMZ) because the ISP router does not allow for bridge mode.

How I think it should be done:
-You need to setup wireguard (only 1 side needs to have a real public IP)
-Set an IP address on both ends of that connection
-Use those 2 addresses to setup EOIP
-Add EOIP to bridge on both ends

Yup, if don't have two public IPs, then @holvoetn has it right. You'd want to disable IPSec in EoIP, and use the far-end WG addresses in the EoIP src/dest fields.

I only mentioned EoIP+IPSec since the hEX CPU does show its age with WG – but that's not possible unless you can use public IP at both ends. Since WG is fine with NAT on ONE end, essentially you replace IPSec with WG, but rest of EoIP bridging is same.

How I think it should be done:
-You need to setup wireguard (only 1 side needs to have a real public IP)
-Set an IP address on both ends of that connection
-Use those 2 addresses to setup EOIP
-Add EOIP to bridge on both ends

Yup, if don't have two public IPs, then @holvoetn has it right. You'd want to disable IPSec in EoIP, and use the far-end WG addresses in the EoIP src/dest fields.

I only mentioned EoIP+IPSec since the hEX CPU does show its age with WG – but that's not possible unless you can use public IP at both ends. Since WG is fine with NAT on ONE end, essentially you replace IPSec with WG, but rest of EoIP bridging is same.

Well, but that doesn't really answer the question. Why doesn't the wireguard interface in R1 forward to the l2tp interface?
And anyway, I began all this by trying to setup a wireguard between R1 and R2. For that, I started to setup a normal RW wireguard in R2 (hexS) so I could access both R's while I was out of both sites. I could not get the wireguard to work in the hexS. Later I found other posts in this forum related to the very same problem. I concluded that the hexS has a problem with wireguard.....

Well, but that doesn't really answer the question. Why doesn't the wireguard interface in R1 forward to the l2tp interface?

That's probably true. Trying to steer you away from L2TP + WG for Layer2 needs to EoIP + WG if you have only one end with public IP. L2TP probably can work, but it's terrible complex between IPSec policies and the firewall, so I don't have any quick answer. Essentially L2TP make a couple trips through the firewall, so needed rules start getting very complex...
https://help.mikrotik.com/docs/display/ ... ecPolicies

Now how well WG works on MIPSBE, I dunno know. But it should work... I'm guessing configuration/firewall sometime get in the way of things working. It just not simple to understand the packet flow diagram – specially with VPN because they take all take some twist-and-turns through firewall – & IPSec used by L2TP adds even more.

On a device without hw offloading, WG is always faster then IPSEC ( and definitely L2TP/IPSEC).

Heck, on some devices which do support hw offloading, wg is still faster.
See my testing on hap ax lite.

On a device without hw offloading, WG is always faster then IPSEC ( and definitely L2TP/IPSEC).

I'm not 100% sure you're right, maybe? Just IPSec encryption is/should-be offloaded on a hEX, while CPU is needed for WG encryption. IPSec encryption "offloading" is different than switch-chip offloading. So results depend on how loaded the CPU for WG performance.

viewtopic.php?t=187892

Key difference is the hEX has IPSec offloading while RB2011 doesn't... Without IPSec encryption offloading, I'd totally imagine WG would be faster...

You'll note the hEX-S list IPSec performance while the RB2011 used in other thread does not list IPSec specs:
https://mikrotik.com/product/hex_s#fndtn-testresults
https://mikrotik.com/product/RB2011UiAS ... estresults
https://mikrotik.com/product/hap_ax2#fndtn-testresults

Now, the thread notes:

When using WireGuard, the speed in the channel is slightly higher. So as is the CPU load.

This make sense. But issue is a hAPax or RB4011 ARM is way more powerful than hEX-S MIPS...so less CPU available for things on hEX...

FWIW It's the combining BOTH WG and L2TP that sounds problematic here...

Why both ?

Wg is already an encrypted vpn.

Why both ?

Wg is already an encrypted vpn.

Exactly. I think OP want's Layer2 bridging. So IMO, on a hEX/hEX-S, since EoIP+IPSec isn't possible here, and OP has WG so EoIP+WG seem like best fit. But OP running into issue with WG, so tried L2TP I think.

Anyway, if the OP posted some diagram and/or sanitized config, might help.

Why both ?

Wg is already an encrypted vpn.

Exactly. I think OP want's Layer2 bridging. So IMO, on a hEX/hEX-S, since EoIP+IPSec isn't possible here, and OP has WG so EoIP+WG seem like best fit. But OP running into issue with WG, so tried L2TP I think.

Anyway, if the OP posted some diagram and/or sanitized config, might help.

I'm not sure what a sanitized config would look like.....

R1 is connected to R2 through l2tp. All devices in both sites can access all devices in both sites.
That is "solved", even though it's working without ipsec because I did it in a hurry when I was at site 2 and now I don't want to risk loosing the connection to site 2 (I'm usually at site 1). I'll add ipsec when I physically return to site 2.
WG is a completely different problem. I've used for a long time a RW WG on my laptop to connect to R1. But when I'm connected to R1 through WG, I can't connect to site 2. That is the problem!!!
I also have a permanent ovpn connection on a NAS in site 1 to connect to site 3. When I'm using my RW WG I can connect to site 3. All I did was add a route to the MT. Why can't the MT manage the route to site 2 when I'm using the WG?

So, no ipsec (for now).
l2tp is only between R1 and R2.
wireguard is only between my laptop and R1.

WG between both sites has to work, provided config is ok.
But we never saw that ?

Export config of both routers with wireguard configured.

terminal
/export file=anynameyouwish
Remove serial number, public wanip, private keys, ...
Post config of both devices separate between code quotes (easier to read).

WG between both sites has to work, provided config is ok.
But we never saw that ?

Export config of both routers with wireguard configured.

terminal
/export file=anynameyouwish
Remove serial number, public wanip, private keys, ...
Post config of both devices separate between code quotes (easier to read).

You might be right. But my ultimate goal is a L2 tunnel. And I even have ipsec offload on both routers. That is why I was aiming for EoIP. Both sites have public IP's, but R1 is behind a ISP router that doesn't do bridge mode.
Besides, I tried WG on R2 (hexS) for my RW laptop (like I have on R1) so I could build the L2 between R1 and R2 while I am RoadWarrioring, and I couldn't even get the R2 WG to work. So, WG is really only for my laptop, nothing else.

There is no problem having S2S using Wireguard.
I have multiple setups like that.
Put EOIP on top and you have L2.

Oh well, your problem to solve it then.