Dual Wan Port Fowarding

henrique89 · Thu Jan 26, 2017 4:32 pm

Hey folks.
I've been searching about my problem here, but unfortunately any topic could help me.
So, what I need is:
Actually I've 2 wan link and I need to do port forwarding through both links.
Wan1: 1.1.1.1
Wan2: 2.2.2.2
Lan: 8.8.8.0/24
Server: 8.8.8.8
E.g. Acess my server 8.8.8.8 through Wan1 and Wan2 on ssh port 22.
Listed below my mangle and nat rules
NAT:
1 chain=dstnat action=dst-nat to-addresses=8.8.8.8 to-ports=22 protocol=tcp in-interface=eth_Wan2 dst-port=522 log=no log-prefix=""

Mangle:
1 chain=prerouting action=mark-connection new-connection-mark=wan1_conn passthrough=yes connection-state=new in-interface=eth_Wan1 connection-mark=no-mark log=no log-prefix=""
2 chain=prerouting action=mark-routing new-routing-mark=WAN1 passthrough=no in-interface=LAN connection-mark=wan1_conn log=no log-prefix=""
3 chain=prerouting action=mark-connection new-connection-mark=wan2_conn passthrough=yes connection-state=new in-interface=eth_Wan2 connection-mark=no-mark log=no log-prefix=""
4 chain=prerouting action=mark-routing new-routing-mark=WAN2 passthrough=yes in-interface=eth7 connection-mark=wan2_conn log=no log-prefix=""

Am I doing something wrong?

Sob · Thu Jan 26, 2017 7:04 pm

Yes. If you want it to work from both WANs, then you can't have in-interface=eth_Wan2 in dstnat rule. If you don't want to have duplicate rules, use dst-address-type=local dst-address=!<address on LAN interface>.

henrique89 · Thu Jan 26, 2017 7:31 pm

Sob, thanks for reply, but still not working.

Let me explain a little more about it.
My primary internet link is Wan1, if I change dst-interface of this NAT, from Wan2 to Wan1, it works perfectly through wan1, and through wan2 doesn't work.
And if I change my internet connection to go out through Wan2, this NAT works perfectly as well, and of course, it'll not work through wan1.

What I want is make port forwarding through both links, independently which is the primary.

Sob · Thu Jan 26, 2017 8:28 pm

I'm not sure if we understand each other. If your dstnat rule has in-interface=eth_Wan1, then it will only work from WAN1, that's what this condition is for. And if it has in-interface=eth_Wan2, it will only work from WAN2. If you remove it, then it will work from everywhere, which might be too much. So you should add at least dst-address-type=local condition (= any address owned by router). You might want to also exclude LAN address, but since it's non-standard port, it's not strictly required. So your dstnat rule should look like this:

/ip firewall nat
add chain=dstnat action=dst-nat to-addresses=8.8.8.8 to-ports=22 protocol=tcp dst-address-type=local dst-port=522

Edit: One more thing, your route marking rules have one in-interface=LAN and the other one in-interface=eth7. One of that is wrong, it should be internal interface where server is connected for both.

henrique89 · Thu Jan 26, 2017 9:11 pm

Yes I know, if I put dst-interface at NAT it will works only from that specific interface.
I've already made this changes, it works from wan1 but not from wan2.
Remembering, wan1 is my primary link, if I set wan2 as primary, than the nat works through wan2 but not through wan1
My Nat rule:
chain=dstnat action=dst-nat to-addresses=8.8.8.8 to-ports=22 protocol=tcp dst-address-type=local dst-port=522 log=no log-prefix=""

About rule mangle nuber 4, it supposed to be LAN and not eth7, my bad when I was typing.

Sob · Thu Jan 26, 2017 10:09 pm

What about routes? In addition to ones in main routing table, you also have following routes, right?

/ip route
add dst-address=0.0.0.0/0 gateway=<gateway for WAN1> routing-mark=WAN1
add dst-address=0.0.0.0/0 gateway=<gateway for WAN2> routing-mark=WAN2

henrique89 · Thu Jan 26, 2017 10:43 pm

Yes, it is
dst-address=0.0.0.0/0 gateway=wan2_gw gateway-status=wan2_gw reachable via eth_wan2 check-gateway=ping distance=1 scope=30
target-scope=10 routing-mark=WAN2
dst-address=0.0.0.0/0 gateway=wan1_gw gateway-status=wan1_gw reachable via eth_wan1 check-gateway=ping distance=1 scope=30
target-scope=10 routing-mark=WAN1

che · Fri Jan 27, 2017 10:15 am

This is a common mistake with implementing mangle rules, and oddly enough correct solutions are hard to find. In short, you are missing mangle rules in order to make this work properly, but sadly I don't have time to write them now. If noone jumps in I'll post them tonight.

henrique89 · Fri Jan 27, 2017 1:08 pm

This is a common mistake with implementing mangle rules, and oddly enough correct solutions are hard to find. In short, you are missing mangle rules in order to make this work properly, but sadly I don't have time to write them now. If noone jumps in I'll post them tonight.

Thanks Che, I'm looking forward to reading your reply!!

che · Sun Jan 29, 2017 12:04 am

In order for you to inderstand this logic I need to state my optimizing argument: since you are doing simple failover WAN, to prevent excess CPU usage I will advise you to mangle only connections that go via secondary gateway, because even without any mangle rules all connections will go through WAN1 anyway. If you want more complicated solution (load balancing), we would need to add more rules and strain to the router. Depending on your line speeds, this could mean difference between 20% and 100% CPU usage.

/ip firewall mangle
add chain=prerouting in-interface=eth_Wan2
add action=mark-connection chain=forward connection-state=new in-interface=eth_Wan2 new-connection-mark=WAN2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=WAN2 new-routing-mark=WAN2 passthrough=no

These are all the rules you need to make your DST-NAT rules incoming via secondary inactive gateway work, so basically you were missing only forward mangle. Note that my first mangle rule has to be #1 on your mangle list, and default action is accept!

Hope you like this approach as much as I do.

henrique89 · Tue Jan 31, 2017 12:18 pm

Hey Che.
Man still not working, here are the rules.
I could see the packets incoming at NAT and Mangle0 rule, but the connection still failing with timeout.
What else could be?

Thanks for the effort

0 chain=prerouting action=accept in-interface=eth_Wan2 log=yes log-prefix=""
1 chain=forward action=mark-connection new-connection-mark=wan2_conn passthrough=no connection-state=new in-interface=eth_Wan2 log=no log-prefix=""
2 chain=prerouting action=mark-routing new-routing-mark=WAN2 passthrough=yes connection-mark=wan2_conn log=no log-prefix=""

che · Tue Jan 31, 2017 8:18 pm

You literally need only 2 DST-NAT rules (for both WAN links, or only 1 rule if you know how to make address/interface lists), those 3 mangle rules and 1 policy routing rule for DST-NAT to work via both gateways. If you don't see counter going up on any magle rules but 0, you haven't have done proper NAT.

henrique89 · Tue Jan 31, 2017 10:12 pm

che,
Here is my NAT rule. Is there something wrong?

chain=dstnat action=dst-nat to-addresses=8.8.8.8 to-ports=22 protocol=tcp dst-port=522 log=no log-prefix=""

As I told you before, I could see the packets arriving at nat rule.

che · Tue Jan 31, 2017 11:50 pm

/ip firewall nat
add action=dst-nat chain=dstnat dst-address=WAN1ADDRESS dst-port=522 protocol=tcp to-addresses=8.8.8.8 to-ports=22
add action=dst-nat chain=dstnat dst-address=WAN2ADDRESS dst-port=522 protocol=tcp to-addresses=8.8.8.8 to-ports=22

Make sure that you are allowing connections to ports you need in firewall, and also make sure your rules have your server's IP instead of 8.8.8.8 (I hope this was fictional address you used to cover your local IP for whatever the reason).

henrique89 · Wed Feb 01, 2017 1:58 am

I've already tested this configuration, specifying the IP address from each wan interface, and still not working.

Sob · Wed Feb 01, 2017 2:38 am

There's nothing obviously wrong (or we all missed it...) in what you posted so far. Maybe it's something you didn't post yet. You can always try exporting and posting your whole config, and if it's not kilometer long, someone might spot something interesting in there.

mattstephenson · Wed Feb 01, 2017 2:44 am

I have the same problem, and cannot despite countless rules in mangle get it to function.

I see ICMP packets arrive on the second interface, but are returned on the first interface - despite mangle rules forcing all ICMP via the second interface.

Both my gateways are PPPoE with the same provider and therefore same gateway - I do not know if this is what causes the confusion?

Sob · Thu Feb 02, 2017 10:26 pm

One idea, can't it be fasttrack? I don't normally use it, so I'm not sure, but it makes packets skip some steps, maybe that includes routing? Check that, I'm too lazy to look it up right now myself.

mattstephenson · Fri Feb 03, 2017 1:27 am

One idea, can't it be fasttrack? I don't normally use it, so I'm not sure, but it makes packets skip some steps, maybe that includes routing? Check that, I'm too lazy to look it up right now myself.

Thank you, but I do not use fast track rules in the firewall, so I do not think that is the issue.

Sob · Fri Feb 03, 2017 1:51 am

In your case, you didn't post many details about your config, but if you have two gateways with same address, you should try routes with gateway=<address>%<interface> (e.g. gateway=1.2.3.4%ether1).

mattstephenson · Fri Feb 03, 2017 2:03 am

In your case, you didn't post many details about your config, but if you have two gateways with same address, you should try routes with gateway=<address>%<interface> (e.g. gateway=1.2.3.4%ether1).

Thank you - unfortunately the gateway is assigned by DHCP from the PPPoE and after speaking to the provider, the gateway changes periodically, so reluctant to hard code.

Maybe the only option is to have two MikroTik routers, each with a single PPPoE connection and default gateway. Then create an additional default route from the first router to the second with a higher "distance"?

Sob · Fri Feb 03, 2017 2:22 am

You can try to play with routing filters (distance 7 serves as unique selector):

/ip dhcp-client
add default-route-distance=7 interface=test2
/routing filter
add chain=dynamic-in distance=7 set-distance=1 set-in-nexthop-direct=test3

mattstephenson · Fri Feb 03, 2017 3:35 am

You can try to play with routing filters (distance 7 serves as unique selector):
Code: Select all
/ip dhcp-client
add default-route-distance=7 interface=test2
/routing filter
add chain=dynamic-in distance=7 set-distance=1 set-in-nexthop-direct=test3

Thank you, I will try this in the next maintenance window. So I replace "test2" with "pppoe-out1", what is "test3" referencing?

I have included some more information below, when both WANs are PPPoE with the same provider and connect and obtain their static IP and routing information.

If both are enabled, only the primary interface appears functional, input or dst-nat to the secondary WAN does not work. Ping from the router itself to 8.8.8.8 works only IF the src-address is set to the secondary interface IP.

I do not need load balancing, only dst-nat to function inbound for both IPs, so I can spread inbound services across the two connections. Also if the connections could failover based on their distance - that would be great too, but I think that part works already - as I set the distance on the pppoe connection.

/interface print
Flags: D - dynamic, X - disabled, R - running, S - slave
 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU MAC-ADDRESS
 0  R  ;;; WAN1
       ether1                              ether            1500  1598       8156
 1  R  ;;; WAN2
       ether2                              ether            1500  1598       8156
 2  R  ;;; LAN Trunk
       ether3                              ether            1500  1598       8156
 3     ether4                              ether            1500  1598       8156
 4     ;;; Config
       ether5                              ether            1500  1598       8156
 5     ether6                              ether            1500  1598       8156
 6     ether7                              ether            1500  1598       8156
 7     ether8                              ether            1500  1598       8156
 8     ether9                              ether            1500  1598       8156
 9     ether10                             ether            1500  1598       8156
10     sfp1                                ether            1500  1600       8158
11  R  ;;; WAN1
       pppoe-out1                          pppoe-out        1492
12  R  ;;; WAN2
       pppoe-out2                          pppoe-out        1492
13  R  ;;; Main
       vlan2                               vlan             1500  1594
14  R  ;;; Staff
       vlan3                               vlan             1500  1594
15  R  ;;; Guest
       vlan4                               vlan             1500  1594
16  R  ;;; Hosted
       vlan5                               vlan             1500  1594

/ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          62.3.80.17                1
 1  DS  0.0.0.0/0                          62.3.80.17                2
 2 ADC  62.3.80.17/32      88.xxx.xxx.151  pppoe-out1                0
 3 ADC  62.3.80.17/32      88.xxx.xxx.182  pppoe-out2                0
 4 ADC  192.168.1.0/24     192.168.1.1     vlan3                     0
 5 ADC  192.168.2.0/24     192.168.2.1     vlan4                     0
 6 ADC  192.168.50.0/24    192.168.50.1    vlan2                     0
 7 ADC  192.168.70.0/24    192.168.70.1    vlan5                     0
 8 A S  192.168.71.0/24                    192.168.70.10             1
 9  DC  192.168.88.0/24    192.168.88.1    ether5                  255

Sob · Fri Feb 03, 2017 8:46 pm

"test2" and "test3" are my interfaces. Unfortunately, this is most likely wrong. I still think that something can be done using route filters, but I'll have to do few experiments to see if I can come up with working solution.

You can help by sharing your PPPoE config, specifically I'm looking for any hint why you have two routes for 62.3.80.17/32, but I get only one with two interfaces (I'm a little new to PPPoE):

 4 ADC  62.3.80.17/32      88.0.0.182      pppoe-out2                0
                                           pppoe-out1

mattstephenson · Fri Feb 03, 2017 8:51 pm

"test2" and "test3" are my interfaces. Unfortunately, this is most likely wrong. I still think that something can be done using route filters, but I'll have to do few experiments to see if I can come up with working solution.

You can help by sharing your PPPoE config, specifically I'm looking for any hint why you have two routes for 62.3.80.17/32, but I get only one with two interfaces (I'm a little new to PPPoE):
Code: Select all
 4 ADC  62.3.80.17/32      88.0.0.182      pppoe-out2                0
                                           pppoe-out1

Hi again, you caught me out that time, correct - when they match it shows as one line, I however doctored the "route print" to show with the same gateway, sometimes it comes up with different gateways for each, but often they are the same - just not that day.

If the gateways are the same or different it still will not work - despite all the mangles I have tried.

Sob · Fri Feb 03, 2017 9:38 pm

I don't see a way how to do what I wanted to with route filters (it doesn't necessarily mean that there isn't one, I might have overlooked something). But if I understand it correctly, PPPoE should not care about gateways very much and should work with routes that have interface as gateway. I tried it with connections to router and it works:

/ip route
add distance=1 gateway=pppoe-out1 routing-mark=wan1
add distance=1 gateway=pppoe-out2 routing-mark=wan2
/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out1 \
    new-connection-mark=wan1 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out2 \
    new-connection-mark=wan2 passthrough=yes
add action=mark-routing chain=output connection-mark=wan1 new-routing-mark=wan1 \
    passthrough=yes
add action=mark-routing chain=output connection-mark=wan2 new-routing-mark=wan2 \
    passthrough=yes

To use it with internal services (dstnat), add either just these rules:

/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=<LAN> \
    new-routing-mark=wan1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=<LAN> \
    new-routing-mark=wan2 passthrough=yes

Or you can drop in-interface=<LAN> to make it work with all interfaces and also add this:

/ip route rule
add action=lookup-only-in-table dst-address=192.168.0.0/16 table=main

mattstephenson · Fri Feb 03, 2017 10:12 pm

I don't see a way how to do what I wanted to with route filters (it doesn't necessarily mean that there isn't one, I might have overlooked something). But if I understand it correctly, PPPoE should not care about gateways very much and should work with routes that have interface as gateway. I tried it with connections to router and it works:
Code: Select all
/ip route
add distance=1 gateway=pppoe-out1 routing-mark=wan1
add distance=1 gateway=pppoe-out2 routing-mark=wan2
/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out1 \
    new-connection-mark=wan1 passthrough=yes
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out2 \
    new-connection-mark=wan2 passthrough=yes
add action=mark-routing chain=output connection-mark=wan1 new-routing-mark=wan1 \
    passthrough=yes
add action=mark-routing chain=output connection-mark=wan2 new-routing-mark=wan2 \
    passthrough=yes
To use it with internal services (dstnat), add either just these rules:
Code: Select all
/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=<LAN> \
    new-routing-mark=wan1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=<LAN> \
    new-routing-mark=wan2 passthrough=yes
Or you can drop in-interface=<LAN> to make it work with all interfaces and also add this:
Code: Select all
/ip route rule
add action=lookup-only-in-table dst-address=192.168.0.0/16 table=main

Hi

I've already done this 100 times over, just to get a PING to the router from outside, it works only when one interface is enabled. I even forced all ICMP traffic to reply through the interface I was pinging, but when using the TORCH I could see it was replying out of the other interface. The only way I could get it to stick to the interface was make a route rule based on destination address.

/interface pppoe-client
add add-default-route=yes comment=WAN1 default-route-distance=1 disabled=no interface=ether1 keepalive-timeout=disabled max-mru=1492 max-mtu=1492 name=\
    pppoe-out1 password=xxx user=acc1
add add-default-route=yes comment=WAN2 default-route-distance=2 disabled=no interface=ether2 keepalive-timeout=disabled max-mru=1492 max-mtu=1492 name=\
    pppoe-out2 password=xxx user=acc2

Sob · Fri Feb 03, 2017 10:28 pm

It may be time to post your config (routes, firewall rules, ...), because marking routing is something that usually works without any problem. You may have some tiny mistake somewhere...

mattstephenson · Fri Feb 03, 2017 10:44 pm

It may be time to post your config (routes, firewall rules, ...), because marking routing is something that usually works without any problem. You may have some tiny mistake somewhere...

Below is the configuration, currently pppoe-out1 is disconnected to allow function on pppoe-out2 as normal.

This is the clean config without any mangles or route marking.

/ip route
add distance=1 dst-address=192.168.71.0/24 gateway=192.168.70.10

/ip firewall address-list
add address=224.0.0.0/4 list=Bogon
add address=203.0.113.0/24 list=Bogon
add address=198.51.100.0/24 list=Bogon
add address=198.18.0.0/15 list=Bogon
add address=192.88.99.0/24 list=Bogon
add address=192.0.2.0/24 list=Bogon
add address=192.168.0.0/16 list=Bogon
add address=172.16.0.0/12 list=Bogon
add address=169.254.0.0/16 list=Bogon
add address=127.0.0.0/16 list=Bogon
add address=10.0.0.0/8 list=Bogon
add address=0.0.0.0/8 list=Bogon
/ip firewall filter
add action=drop chain=input comment=Invalid connection-state=invalid
add action=accept chain=input comment=Config ipsec-policy=in,ipsec
# pppoe-out1 not ready
add action=drop chain=input comment="Bogon WAN1" in-interface=pppoe-out1 src-address-list=Bogon
add action=drop chain=input comment="Bogon WAN2" in-interface=pppoe-out2 src-address-list=Bogon
add action=accept chain=input comment=Site-to-Site in-interface=pppoe-out2 protocol=ipsec-esp
add action=accept chain=input comment="Established, Related" connection-state=established,related
add action=accept chain=input comment=DNS dst-port=53 in-interface=all-vlan protocol=udp
add action=accept chain=input comment=DNS dst-port=53 in-interface=all-vlan protocol=tcp
add action=accept chain=input comment=Config in-interface=vlan2
add action=accept chain=input comment=Config in-interface=vlan5
add action=accept chain=input comment=Config in-interface=ether5
add action=accept chain=input comment=ICMP in-interface=all-vlan protocol=icmp
add action=drop chain=input comment=Drop
add action=drop chain=forward comment=Invalid connection-state=invalid
add action=accept chain=forward comment=Site-to-Site ipsec-policy=in,ipsec
add action=accept chain=forward comment="Main VLAN to Hosted VLAN" in-interface=vlan2 out-interface=vlan5
add action=accept chain=forward comment="Hosted VLAN to Main VLAN" in-interface=vlan5 out-interface=vlan2
# pppoe-out1 not ready
add action=drop chain=forward comment="Bogon WAN1" in-interface=pppoe-out1 src-address-list=Bogon
add action=drop chain=forward comment="Bogon WAN2" in-interface=pppoe-out2 src-address-list=Bogon
add action=accept chain=forward comment="Established, Related" connection-state=established,related
# pppoe-out1 not ready
add action=accept chain=forward comment="VLAN to WAN1" in-interface=all-vlan out-interface=pppoe-out1
add action=accept chain=forward comment="VLAN to WAN2" in-interface=all-vlan out-interface=pppoe-out2
# pppoe-out1 not ready
add action=accept chain=forward comment="Config to WAN1" in-interface=ether5 out-interface=pppoe-out1
add action=accept chain=forward comment="Config to WAN2" in-interface=ether5 out-interface=pppoe-out2
add action=accept chain=forward comment="HTTPS: srv-remote" dst-address=192.168.50.12 dst-port=443 protocol=tcp
add action=accept chain=forward comment="HTTP: srv-lamp" dst-address=192.168.50.14 dst-port=80 protocol=tcp
add action=accept chain=forward comment="OpenVPN: srv-vpn" dst-address=192.168.70.10 dst-port=1194 protocol=udp
add action=accept chain=forward comment="SSH: srv-host1" dst-address=192.168.50.5 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-back" dst-address=192.168.50.30 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-arch" dst-address=192.168.50.31 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-lamp" dst-address=192.168.50.14 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-vpn" dst-address=192.168.70.10 dst-port=22 protocol=tcp
add action=accept chain=forward comment="SSH: srv-qb" dst-address=192.168.50.13 dst-port=22 protocol=tcp src-address=82.145.45.154
add action=accept chain=forward comment="Camera: camera" dst-address=192.168.50.81 dst-port=8000 protocol=tcp
add action=drop chain=forward comment=Drop
/ip firewall nat
add action=masquerade chain=srcnat comment=Loopback dst-address=192.168.50.0/24 src-address=192.168.50.0/24
add action=masquerade chain=srcnat comment="Loopback (Hosted)" dst-address=192.168.70.0/24 src-address=192.168.70.0/24
add action=accept chain=srcnat comment=Site-to-Site dst-address=192.168.0.0/16 src-address=192.168.50.0/24
add action=accept chain=srcnat comment="Site-to-Site (Hosted)" dst-address=192.168.0.0/16 src-address=192.168.70.0/24
add action=accept chain=srcnat comment="Site-to-Site (VPN)" dst-address=192.168.0.0/16 src-address=192.168.71.0/24
# pppoe-out1 not ready
add action=masquerade chain=srcnat comment=WAN1 out-interface=pppoe-out1
add action=masquerade chain=srcnat comment=WAN2 out-interface=pppoe-out2
add action=dst-nat chain=dstnat comment="HTTPS: srv-remote" dst-address=88.xx.xx.182 dst-port=443 protocol=tcp to-addresses=192.168.50.12
add action=dst-nat chain=dstnat comment="HTTP: srv-lamp" dst-address=88.xx.xx.182 dst-port=80 protocol=tcp to-addresses=192.168.50.14
add action=dst-nat chain=dstnat comment="OpenVPN: srv-vpn" dst-address=88.xx.xx.182 dst-port=1194 protocol=udp to-addresses=192.168.70.10
add action=dst-nat chain=dstnat comment="SSH: srv-host1 (WAN1)" dst-address=88.xx.xx.151 dst-port=22 protocol=tcp to-addresses=192.168.50.5
add action=dst-nat chain=dstnat comment="SSH: srv-host1 (WAN2)" dst-address=88.xx.xx.182 dst-port=22 protocol=tcp to-addresses=192.168.50.5
add action=dst-nat chain=dstnat comment="SSH: srv-back (WAN1)" dst-address=88.xx.xx.151 dst-port=222 protocol=tcp to-addresses=192.168.50.30 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-back (WAN2)" dst-address=88.xx.xx.182 dst-port=222 protocol=tcp to-addresses=192.168.50.30 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-arch (WAN1)" dst-address=88.xx.xx.151 dst-port=223 protocol=tcp to-addresses=192.168.50.31 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-arch (WAN2)" dst-address=88.xx.xx.182 dst-port=223 protocol=tcp to-addresses=192.168.50.31 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-lamp" dst-address=88.xx.xx.182 dst-port=224 protocol=tcp to-addresses=192.168.50.14 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-vpn" dst-address=88.xx.xx.182 dst-port=225 protocol=tcp to-addresses=192.168.70.10 to-ports=22
add action=dst-nat chain=dstnat comment="SSH: srv-qb" dst-address=88.xx.xx.182 dst-port=230 protocol=tcp to-addresses=192.168.50.13 to-ports=22
add action=dst-nat chain=dstnat comment="Camera: camera" dst-address=88.xx.xx.182 dst-port=8000 protocol=tcp to-addresses=192.168.50.81

Sob · Sat Feb 04, 2017 12:01 am

And if you add what I wrote (routes and mangle rules), then what exactly happens? When you ping the router from internet (both addresses), do you see increasing counters on those rules?

mattstephenson · Sat Feb 04, 2017 12:17 am

I cannot do it right now, as I am not on site, and I do not want to risk locking myself out.

When I did it before the counters do increase yes - it does hit the rules in the mangle.

Sob · Sat Feb 04, 2017 1:39 am

It must work, it's just three simple steps:
- Incoming connection from pppoe-outX gets marked with connection mark wanX
- Outgoing packets with connection mark wanX get routing mark wanX
- The only route from routing table wanX gets used

What could go wrong there? You can easily verify first two steps. There are counters and if you have doubts, you can enable logging for rules and make sure they really catch the right packets. And once the packet gets routing mark, it can't choose wrong route, because there's only one in given routing table, which points to right interface. I don't have any explanation, except perhaps a typo in routing table name.

mattstephenson · Sat Feb 04, 2017 1:41 am

It must work, it's just three simple steps:
- Incoming connection from pppoe-outX gets marked with connection mark wanX
- Outgoing packets with connection mark wanX get routing mark wanX
- The only route from routing table wanX gets used

What could go wrong there? You can easily verify first two steps. There are counters and if you have doubts, you can enable logging for rules and make sure they really catch the right packets. And once the packet gets routing mark, it can't choose wrong route, because there's only one in given routing table, which points to right interface. I don't have any explanation, except perhaps a typo in routing table name.

I know exactly what you mean, I have tried from scratch several times (by reverting to clean backup) but still no dice, I will try again soon when I am onsite - it is infuriating, and your help is greatly appreciated. Thank you

mattstephenson · Sat Feb 04, 2017 1:49 am

Living dangerously - I've done it remotely - same result

mattstephenson · Sat Feb 04, 2017 1:50 am

 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          pppoe-out1                1
 1 A S  0.0.0.0/0                          pppoe-out2                1
 2 ADS  0.0.0.0/0                          62.3.80.17                1
 3  DS  0.0.0.0/0                          62.3.80.17                2
 4 ADC  62.3.80.17/32      88.xx.xx.182    pppoe-out2                0
                                           pppoe-out1
 5 ADC  192.168.1.0/24     192.168.1.1     vlan3                     0
 6 ADC  192.168.2.0/24     192.168.2.1     vlan4                     0
 7 ADC  192.168.50.0/24    192.168.50.1    vlan2                     0
 8 ADC  192.168.70.0/24    192.168.70.1    vlan5                     0
 9 A S  192.168.71.0/24                    192.168.70.10             1
10  DC  192.168.88.0/24    192.168.88.1    ether5                  255

/ip route
add distance=1 gateway=pppoe-out1 routing-mark=wan1-route
add distance=1 gateway=pppoe-out2 routing-mark=wan2-route
add distance=1 dst-address=192.168.71.0/24 gateway=192.168.70.10

Sob · Sat Feb 04, 2017 1:59 am

And your exact mangle rules are? Just as a last check, before you go find a shaman or similar non-technical help?

mattstephenson · Sat Feb 04, 2017 2:02 am

/ip firewall mangle
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out1 new-connection-mark=wan1-conn passthrough=yes
add action=mark-connection chain=prerouting connection-state=new in-interface=pppoe-out2 new-connection-mark=wan2-conn passthrough=yes
add action=mark-routing chain=output connection-mark=wan1-conn new-routing-mark=wan1-route passthrough=yes
add action=mark-routing chain=output connection-mark=wan2-conn new-routing-mark=wan2-route passthrough=yes

 /ping 8.8.8.8 interface=pppoe-out1
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 8.8.8.8                                    56  61 36ms
    1 8.8.8.8                                    56  61 18ms
    2 8.8.8.8                                    56  61 18ms
    sent=3 received=3 packet-loss=0% min-rtt=18ms avg-rtt=24ms max-rtt=36ms

/ping 8.8.8.8 interface=pppoe-out2
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 8.8.8.8                                                 timeout
    1 8.8.8.8                                                 timeout
    2 8.8.8.8                                                 timeout
    sent=3 received=0 packet-loss=100%

mattstephenson · Sat Feb 04, 2017 2:11 am

/ip route nexthop
 0 address=62.3.80.17 gw-state=reachable forwarding-nexthop="" interface="" scope=10 check-gateway=none
 1 address=192.168.70.10 gw-state=reachable forwarding-nexthop="" interface="" scope=10 check-gateway=none

Sob · Sat Feb 04, 2017 2:27 am

Try a different test, ping both public addresses from outside. Because when you initiate connection from router to internet, it won't get marked, because first packet coming back won't match connection-state=new.

mattstephenson · Sat Feb 04, 2017 2:29 am

Try a different test, ping both public addresses from outside. Because when you initiate connection from router to internet, it won't get marked, because first packet coming back won't match connection-state=new.

I tried that first.. same as before - only the lowest distance interface responds.

Sob · Sat Feb 04, 2017 2:32 am

Then what about:

/ping 8.8.8.8 routing-table=wan1-route
/ping 8.8.8.8 routing-table=wan2-route

mattstephenson · Sat Feb 04, 2017 2:38 am

[admin@router] > /ping 8.8.8.8 routing-table=wan1-route
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 8.8.8.8                                    56  61 18ms
    1 8.8.8.8                                    56  61 18ms
    sent=2 received=2 packet-loss=0% min-rtt=18ms avg-rtt=18ms max-rtt=18ms

[admin@router] > /ping 8.8.8.8 routing-table=wan2-route
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 8.8.8.8                                                 timeout
    1 8.8.8.8                                                 timeout
    sent=2 received=0 packet-loss=100%

Sob · Sat Feb 04, 2017 2:45 am

What about some extra logging?

/ip firewall mangle
add action=accept chain=postrouting dst-address=8.8.8.8 log=yes out-interface=pppoe-out1 protocol=icmp
add action=accept chain=postrouting dst-address=8.8.8.8 log=yes out-interface=pppoe-out2 protocol=icmp

This is what I get for those two pings, I see packets going the right way with right source address:

01:43:00 firewall,info postrouting: in:(none) out:pppoe-out1, proto ICMP (type 8, code 0), 88.0.0.151->8.8.8.8, NAT 88.0.0.151->8.8.8.8, len 56
01:43:05 firewall,info postrouting: in:(none) out:pppoe-out2, proto ICMP (type 8, code 0), 88.0.0.182->8.8.8.8, NAT 88.0.0.182->8.8.8.8, len 56

mattstephenson · Sat Feb 04, 2017 2:48 am

00:47:18 firewall,info postrouting: in:(none) out:pppoe-out1, proto ICMP (type 8, code 0), 88.xxx.xxx.151->8.8.8.8, NAT 88.xxx.xxx.151->8.8.8.8, len 56
00:47:19 firewall,info postrouting: in:(none) out:pppoe-out1, proto ICMP (type 8, code 0), 88.xxx.xxx.151->8.8.8.8, NAT 88.xxx.xxx.151->8.8.8.8, len 56
00:47:20 firewall,info postrouting: in:(none) out:pppoe-out1, proto ICMP (type 8, code 0), 88.xxx.xxx.151->8.8.8.8, NAT 88.xxx.xxx.151->8.8.8.8, len 56
00:47:23 firewall,info postrouting: in:(none) out:pppoe-out2, proto ICMP (type 8, code 0), 88.xxx.xxx.182->8.8.8.8, NAT 88.xxx.xxx.182->8.8.8.8, len 56
00:47:24 firewall,info postrouting: in:(none) out:pppoe-out2, proto ICMP (type 8, code 0), 88.xxx.xxx.182->8.8.8.8, NAT 88.xxx.xxx.182->8.8.8.8, len 56
00:47:25 firewall,info postrouting: in:(none) out:pppoe-out2, proto ICMP (type 8, code 0), 88.xxx.xxx.182->8.8.8.8, NAT 88.xxx.xxx.182->8.8.8.8, len 56

Sob · Sat Feb 04, 2017 2:55 am

And if you add one more:

/ip firewall mangle
add action=log chain=prerouting protocol=icmp src-address=8.8.8.8

Then you see only replies to 88.x.x.151 on pppoe-out1?

mattstephenson · Sat Feb 04, 2017 2:58 am

[admin@router] /log> /ping 8.8.8.8 routing-table=wan1-route
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 8.8.8.8                                    56  61 19ms
    1 8.8.8.8                                    56  61 18ms
    sent=2 received=2 packet-loss=0% min-rtt=18ms avg-rtt=18ms max-rtt=19ms

[admin@router] /log> /ping 8.8.8.8 routing-table=wan2-route
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 8.8.8.8                                                 timeout
    1 8.8.8.8                                                 timeout
    sent=2 received=0 packet-loss=100%

00:56:28 firewall,info postrouting: in:(none) out:pppoe-out1, proto ICMP (type 8, code 0), 88.xxx.xxx.151->8.8.8.8, NAT 88.xxx.xxx.151->8.8.8.8, len 56
00:56:28 firewall,info prerouting: in:pppoe-out1 out:(none), src-mac 10:e8:78:a7:e6:02, proto ICMP (type 0, code 0), 8.8.8.8->88.xxx.xxx.151, NAT 8.8.8.8->88.xxx.xxx.151, len 56
00:56:29 firewall,info postrouting: in:(none) out:pppoe-out1, proto ICMP (type 8, code 0), 88.xxx.xxx.151->8.8.8.8, NAT 88.xxx.xxx.151->8.8.8.8, len 56
00:56:29 firewall,info prerouting: in:pppoe-out1 out:(none), src-mac 10:e8:78:a7:e6:02, proto ICMP (type 0, code 0), 8.8.8.8->88.xxx.xxx.151, NAT 8.8.8.8->88.xxx.xxx.151, len 56

00:56:32 firewall,info postrouting: in:(none) out:pppoe-out2, proto ICMP (type 8, code 0), 88.xxx.xxx.182->8.8.8.8, NAT 88.xxx.xxx.182->8.8.8.8, len 56
00:56:32 firewall,info prerouting: in:pppoe-out2 out:(none), src-mac 10:e8:78:a7:f3:79, proto ICMP (type 0, code 0), 8.8.8.8->88.xxx.xxx.182, NAT 8.8.8.8->88.xxx.xxx.182, len 56
00:56:33 firewall,info postrouting: in:(none) out:pppoe-out2, proto ICMP (type 8, code 0), 88.xxx.xxx.182->8.8.8.8, NAT 88.xxx.xxx.182->8.8.8.8, len 56
00:56:33 firewall,info prerouting: in:pppoe-out2 out:(none), src-mac 10:e8:78:a7:f3:79, proto ICMP (type 0, code 0), 8.8.8.8->88.xxx.xxx.182, NAT 8.8.8.8->88.xxx.xxx.182, len 56

Sob · Sat Feb 04, 2017 3:07 am

But it's right there... tell me I'm not the only one who sees it.

Try to enable logging for these rules, to see if they for some unknown reason catch those replies:

/ip firewall filter
add action=drop chain=input comment=Invalid connection-state=invalid
add action=drop chain=input comment=Drop

mattstephenson · Sat Feb 04, 2017 3:14 am

I already removed the "Invalid" rule just to be sure it was not that, nothing logs on the "drop" rule.

I also enabled logging on ICMP input rule...

01:11:44 firewall,info input: in:pppoe-out1 out:(none), src-mac 10:e8:78:a7:e6:02, proto ICMP (type 8, code 0), 78.xx.xx.11->88.xx.xx.151, NAT 78.xx.xx.11->88.xx.xx.151, len 84
01:11:45 firewall,info input: in:pppoe-out1 out:(none), src-mac 10:e8:78:a7:e6:02, proto ICMP (type 8, code 0), 78.xx.xx.11->88.xx.xx.151, NAT 78.xx.xx.11->88.xx.xx.151, len 84
01:11:46 firewall,info input: in:pppoe-out1 out:(none), src-mac 10:e8:78:a7:e6:02, proto ICMP (type 8, code 0), 78.xx.xx.11->88.xx.xx.151, NAT 78.xx.xx.11->88.xx.xx.151, len 84
01:11:47 firewall,info input: in:pppoe-out1 out:(none), src-mac 10:e8:78:a7:e6:02, proto ICMP (type 8, code 0), 78.xx.xx.11->88.xx.xx.151, NAT 78.xx.xx.11->88.xx.xx.151, len 84

Sob · Sat Feb 04, 2017 3:22 am

Hmmm... we see the packet in prerouting, but it doesn't get to input. Where else it can go? Put this at the top:

/ip firewall filter
add action=log chain=forward protocol=icmp

mattstephenson · Sat Feb 04, 2017 3:24 am

Hmmm... we see the packet in prerouting, but it doesn't get to input. Where else it can go? Put this at the top:
Code: Select all
/ip firewall filter
add action=log chain=forward protocol=icmp

Done.. nothing logs

Sob · Sat Feb 04, 2017 3:42 am

Actually, I meant mangle, not filter, my mistake. But there shouldn't be any difference, it would go to both, if it came this way. You may try it anyway, but I don't think there can be any difference.

I'm running out of ideas...

lost-packets.png

mattstephenson · Sat Feb 04, 2017 3:49 am

It only logs when ping goes through pppoe-out1 interface, pinging pppoe-out1 public does nothing in log.

I think it is down to the 'default route' and how the pppoe gateways work.

Sob · Sat Feb 04, 2017 4:01 am

But outgoing packets are ok and incoming ones too (a little cleaned up version of previous log):

postrouting: out:pppoe-out1, proto ICMP (type 8, code 0), 88.xxx.xxx.151->8.8.8.8, NAT 88.xxx.xxx.151->8.8.8.8
prerouting: in:pppoe-out1, proto ICMP (type 0, code 0), 8.8.8.8->88.xxx.xxx.151, NAT 8.8.8.8->88.xxx.xxx.151

postrouting: out:pppoe-out2, proto ICMP (type 8, code 0), 88.xxx.xxx.182->8.8.8.8, NAT 88.xxx.xxx.182->8.8.8.8
prerouting: in:pppoe-out2, proto ICMP (type 0, code 0), 8.8.8.8->88.xxx.xxx.182, NAT 8.8.8.8->88.xxx.xxx.182

The last one gets lost somewhere in your router. But how and why?

It's a little late here (well, a lot, actually), so I'm officially giving up for today. But I really want to know an explanation for this.

mattstephenson · Sat Feb 04, 2017 4:03 am

Thank you very much for your persistence and time spent on helping me - I am very appreciative.

Sleep well, and let me know if you think of anything else to try.

Many thanks Matt

mattstephenson · Sat Feb 04, 2017 4:51 am

I've only gone and worked it out!!

I remembered that when I set the router up originally I read to enable "RP Filter" to 'strict'. I have since changed this to 'loose' and it works!!!!!!

I can't believe it has taken so long to figure it out - but at least we are further forward and now have pings both ways...

Next is to sort the dst-nat, etc but that can wait until tomorrow

Sob · Sat Feb 04, 2017 7:32 pm

Congratulations! Now to find and punish whoever forgot to include this important part in config export...

mattstephenson · Sat Feb 04, 2017 7:34 pm

Unfortunately I was not breaking out the champagne... i got over confident and locked myself out with a mangle prerouting rule. So a 70 mile drive today to correct.

I did all of the changes and had it all working fine... except.. IPsec site-to-site traffic was very problematic.. would ping, but then any real transfer would fail. Also if you disable/enable one of the PPPoE connections traffic was not respecting 'distance' rules, and would often stick to the last interface brought up.

I have backed up my progress, but for now reverted to the old single connection

Hope you slept well.. i managed 4 hours from the worry!

Sob · Sat Feb 04, 2017 9:04 pm

These changes with routing marks can't break anything by themselves, if you only mark new incoming connections.

The problem is that when you have both connections active and this happens:

 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 2 ADS  0.0.0.0/0                          62.3.80.17                1
 3  DS  0.0.0.0/0                          62.3.80.17                2
 4 ADC  62.3.80.17/32      88.xx.xx.182    pppoe-out2                0
                                           pppoe-out1

You get automatic load balancing. Distances won't help you. Sure, you have two default routes and the one with lower distance wins. But they both have the same default gateway, which is reachable using both connections. That's what I was originally trying with routing filters, to change default gateways from 62.3.80.17 to 62.3.80.17%pppoe-outX, to force them to use only one interface.

But as PPPoE does not seem to require default gateway as IP address, you may try to set add-default-route=no for PPPoE clients and then add manual default routes:

/ip route
add distance=1 gateway=pppoe-out1
add distance=2 gateway=pppoe-out2

This way you'd get two distinct routes and no unwanted load balancing would occur.

There's one problem, it's not the best thing to try remotely. Not impossible, but one small mistake and you're driving again.

mattstephenson · Sat Feb 04, 2017 9:57 pm

Hi

I tried static routes before, but it still seemed to combine them.

I do not really need outgoing from both WANs, just inbound.. could I have NO default route and just the routing-mark route for WAN2?

Also I could not get it to always send IPSec on WAN2.. but to be honest I have given up on that now!

Sob · Sat Feb 04, 2017 10:11 pm

When you tried static routes before, did you disable adding default routes in PPPoE clients? If you disable it just for WAN2, you'll have only one default route, but still using the same default gateway reachable using both connections.

mattstephenson · Sat Feb 04, 2017 10:21 pm

When you tried static routes before, did you disable adding default routes in PPPoE clients? If you disable it just for WAN2, you'll have only one default route, but still using the same default gateway reachable using both connections.

Previously I disabled it on both PPPoE connections and added them manually for both. Is that wrong?

Sob · Sat Feb 04, 2017 11:45 pm

It should give you these routes:

/ip route print detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip,
b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=pppoe-out1 gateway-status=pppoe-out1 reachable
        distance=1 scope=30 target-scope=10 routing-mark=wan1-route 

 1 A S  dst-address=0.0.0.0/0 gateway=pppoe-out2 gateway-status=pppoe-out2 reachable
        distance=1 scope=30 target-scope=10 routing-mark=wan2-route 

 2 A S  dst-address=0.0.0.0/0 gateway=pppoe-out1 gateway-status=pppoe-out1 reachable
        distance=1 scope=30 target-scope=10 

 3   S  dst-address=0.0.0.0/0 gateway=pppoe-out2 gateway-status=pppoe-out2 reachable
        distance=2 scope=30 target-scope=10 

 4 ADC  dst-address=62.3.80.17/32 pref-src=88.0.0.182 gateway=pppoe-out2,pppoe-out1 
        gateway-status=pppoe-out2 reachable,pppoe-out1 reachable distance=0 scope=10

And it should work. First two are for marked incoming connections, those won't get used for anything else automatically. Next two are in main table, tied to individual interfaces and with help of distance, there's no doubt which one should be used (or you can skip adding the other one completely). Last route to 62.3.80.17/32 has both interfaces, but it's alone, nothing else depends on it.

mattstephenson · Sun Feb 05, 2017 12:10 am

Sounds good - and as you said before I will do it when on site

On Tuesday evening when everyone leaves for the day.

So in this scenario if WAN1 goes down, it will use WAN2?

Also is there a possible way to chain IPSEC tunnels to just WAN1 - so it never tries to use WAN2 under any circumstance?

Sob · Sun Feb 05, 2017 2:31 am

Yes, WAN2 should work as backup connection.

To force IPSec to WAN1, you can mark routing for it in output chain. Best by target address, if it's static. If not, you can do it for udp port 500 and protocol 50, but that's probably not good enough in there was NAT involved.

mattstephenson · Sun Feb 05, 2017 2:37 am

They are static addresses at both ends. When I did an output rule before it did not seem to fix it, the link would just refuse to establish.

I will try again Tuesday along with all the other things you have said, I have made a list

mattstephenson · Tue Feb 07, 2017 9:58 pm

I have some good news, the static routes and the other tricks seems to have made it a lot better.

IPsec, I have made the mangle rules as you said but they still do not like to work reliably. I have IPsec linked to WAN1, if I disable the default route for WAN2 and reboot the router it works perfectly.

So for now I have inbound routing on both WANs, but no failover on outbound... I am still very pleased with the progress

Any suggestions on the IPsec?

Thanks Matt

Sob · Thu Feb 09, 2017 1:14 am

Lets say IPSec is something I know less about than other things. My experience is limited to static site-to-site tunnels, with static addresses and single WAN at both sides.

But you wrote that you have also static addresses, so if you mark routing for outgoing packets to other end of tunnel, to use WAN1, it should work. You may set local address in /ip ipsec peer, to make sure that the other one can't be chosen accidentally.

Failover should work for normal traffic. If WAN1 goes down, it should make default route with distance 1 inactive and the other one with distance 2 should take over. It won't work for IPSec traffic, if it requires specific address.

mattstephenson · Thu Feb 09, 2017 1:35 am

Hi

Hope you are well

Yes I have all the rules in the mangle in place correctly.

The IPsec SA is setup correctly with the correct source/destination IPs, if WAN2 has a default route in place it will not connect the routes... once they are established if I enable the route, I think they stay up, but they will not establish in the first place - so for now is inbound only on WAN2.

I realise there will not be failover on the IPsec.. which is fine, I just want the failover for general internet traffic.

I did a Mangle Log entry for all traffic to the SA destination IP to find out where the IPSEC problem could lie, and it is using the correct interface and IPs - I found another forum post about it, it seems to be a long standing bug.. maybe not many people use the IPSEC in combination with dual WAN, and also lots of people seem to not like IPSEC in MikroTik - and recommend using GRE instead with IPSEC encryption.

Sob · Thu Feb 09, 2017 2:26 am

I found another forum post about it, it seems to be a long standing bug..

Link?

... recommend using GRE instead with IPSEC encryption.

I'm guilty myself. It's not that I'm completely against it, but the problem with IPSec in tunnel mode is that you need to specify policies for all subnets at both sides that should use the tunnel and keep them updated. But if you create tunnel using GRE and encrypt it using IPSec in transport mode, then it's "set it & forget it". You can then change subnets as you wish and it's just normal routing. It can save a lot of headaches.

mattstephenson · Thu Feb 09, 2017 2:29 am

At the risk of making you want to throw the book at me, is it complicated to change?

I agree with what you say, as we have multiple subnets at one site and I had to make multiple policies to get it work, but it does seem to work very well and auto-reconnects if ever there is an outage at one end. Can the same be said for the alternative you suggest?

Thanks Matt

Sob · Thu Feb 09, 2017 3:59 am

If current config works ok for you, I'd keep it. But it's not that much different.

Now you have policy with source address <Subnet1>, destination <Subnet2>, SA addresses <Router1> and <Router2>, all protocols and tunnel=yes. And that's it, except having similar policies for other subnets.

The other way is to make GRE (or IPIP) tunnel between <Router1> and <Router2>. Give it some IP addresses and add routes to remote subnets (same way as if it was ethernet connection between two routers). Then add just one policy where both SA and src/dst addresses are <Router1> and <Router2>, protocol 47 (or 4) and tunnel=no.

When you think about it, it's not much simpler when it comes to amount of config. You save on policies, but you have to add routes instead. So it's probably most appealing to people who are new to IPSec (ok, that's not necessarily true in all cases), because they have to do the scary part only once.

Another good thing is that it gives you normal interface to work with. If you have tunnel mode IPSec, traffic from remote subnet looks like it comes from WAN interface and while you can make sure that it came from IPSec, some people might prefer if they see it coming from gre-officeA interface instead. As I understand it, IPSec can have something called VTI (Virtual Tunnel Interface), which behaves the same way, but RouterOS doesn't support it (yet?).

About reliability, it's still the same IPSec, so there shouldn't be any difference.

mattstephenson · Thu Feb 09, 2017 12:09 pm

thank you for the explanation, I think you are right though - at this stage not to rock the boat!!

plus if it uses the same IPsec protocol to do the outer layer of the connection it would probably still cause the same problem that I am facing with the dual wans so would not hugely benefit me.

Final question...

I have two VLANs (different subnets) that need to talk to one another - would i just create two firewall filter forward accept rules VLAN2 to VLAN5 and VLAN5 to VLAN2?

Sob · Sat Feb 11, 2017 4:58 am

Yes, if you have two VLANs as interfaces, accept rules in forward chain is the way to let them talk to each other (if they have this router as gateway).

mattstephenson · Sat Feb 11, 2017 5:00 am

Yes, if you have two VLANs as interfaces, accept rules in forward chain is the way to let them talk to each other (if they have this router as gateway).

Yes they do, but I need TWO rules, one for each direction in FORWARD chain?

Sob · Sat Feb 11, 2017 5:05 am

Depends. If you want both to be equal, i.e. both able to initiate connection to other subnet, you need both. But if you'd want e.g. VLAN2 talk to VLAN5, but not the other way around, you need just one and other direction (replies) would be allowed by "accept established & related" rule.

mattstephenson · Sat Feb 11, 2017 5:06 am

Depends. If you want both to be equal, i.e. both able to initiate connection to other subnet, you need both. But if you'd want e.g. VLAN2 talk to VLAN5, but not the other way around, you need just one and other direction (replies) would be allowed by "accept established & related" rule.

I understand - perfect thank you very much.

I would like to send you a PM but I cannot see how to do this.

Sob · Sat Feb 11, 2017 5:25 am

It looks like PMs are not available here. It's probably a good thing when you think about it. Imagine all those people spamming others with "I saw a post of yours from ten years ago about a problem that I think is remotely similar to mine, come to my thread and solve it for me!" But maybe it's just my not completely optimistic nature and it would not actually happen.

mattstephenson · Sat Feb 11, 2017 5:44 am

It looks like PMs are not available here. It's probably a good thing when you think about it. Imagine all those people spamming others with "I saw a post of yours from ten years ago about a problem that I think is remotely similar to mine, come to my thread and solve it for me!" But maybe it's just my not completely optimistic nature and it would not actually happen.

I understand completely, I have just made a throwawaymail.com address: godrotofri@housat.com - if you email me, I can email you back what I actually wanted to say.