Using Mangle Is Breaking Remote Access

Ufdah · Sat Apr 30, 2022 5:49 am

I have a post here viewtopic.php?t=185577 that explains how I used mangle to track multiple WAN connections and always route inound WAN traffic back out to the appropriate gateway. However implementing this has seemed to break my ability to access the router with WinBox. It seems that everything that goes through the router to a device and back in the LAN gets routed out the WAN correctly, but when the destination is the WAN IP address for WinBox access it fails to return the traffic. Here's a diagram of my system. I'm trying to use WinBox from the B Corp server through Cell 2 and open 192.168.200.2:8291.

When I add a static /IP Routing rule for 0.0.0.0/0 back out that interface (WAN2) it works, but breaks WAN1 traffic. Without that rule both WAN1 and WAN2 work as desired, but WinBox doesn't work. What am I missing?

Ufdah · Sat Apr 30, 2022 5:59 am

Config if anyone wants to dig in with me:

# jan/02/1970 01:50:02 by RouterOS 7.2.1
# software id = V57V-SU19
#
# model = RB750r2
# serial number =
/interface bridge
add admin-mac= auto-mac=no comment=defconf name=bridge
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp ranges=192.168.0.240-192.168.0.249
/ip dhcp-server
add address-pool=dhcp interface=bridge name=defconf
/routing table
add disabled=no fib name=out_WAN2
add disabled=no fib name=out_WAN1
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
/ip firewall connection tracking
set enabled=yes
/ip neighbor discovery-settings
set discover-interface-list=WAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/ip address
add address=192.168.0.1/24 comment=defconf interface=bridge network=\
    192.168.0.0
add address=192.168.13.101/24 interface=ether1 network=192.168.13.0
add address=192.168.200.2/24 interface=ether2 network=192.168.200.0
add address=192.168.200.101/24 interface=ether2 network=192.168.200.0
add address=192.168.13.1/24 interface=ether1 network=192.168.13.0
add address=192.168.13.102/24 interface=ether1 network=192.168.13.0
add address=192.168.13.103/24 interface=ether1 network=192.168.13.0
add address=192.168.200.102/24 interface=ether2 network=192.168.200.0
add address=192.168.200.103/24 interface=ether2 network=192.168.200.0
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1
/ip dhcp-server network
add address=192.168.0.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
    192.168.0.1 netmask=24
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.0.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment=WinBox dst-port=8291 protocol=tcp
add action=accept chain=input comment=WinBox dst-port=80 protocol=tcp
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=ether1 new-connection-mark=WAN1_Conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=ether2 new-connection-mark=WAN2_Conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_Conn \
    new-routing-mark=out_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_Conn \
    new-routing-mark=out_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1_Conn \
    in-interface-list=LAN new-routing-mark=out_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_Conn \
    in-interface-list=LAN new-routing-mark=out_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=no-mark disabled=yes \
    dst-address=172.17.32.0/21 in-interface-list=LAN new-routing-mark=\
    out_WAN2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
add action=dst-nat chain=dstnat dst-address=192.168.13.101 dst-port=4000 \
    in-interface-list=WAN protocol=tcp to-addresses=192.168.0.101 to-ports=\
    4000
add action=dst-nat chain=dstnat dst-address=192.168.13.103 dst-port=4000 \
    in-interface-list=WAN protocol=tcp to-addresses=192.168.0.103 to-ports=\
    4000
add action=dst-nat chain=dstnat dst-address=192.168.13.102 dst-port=4000 \
    in-interface-list=WAN protocol=tcp to-addresses=192.168.0.102 to-ports=\
    4000
add action=dst-nat chain=dstnat dst-address=192.168.200.101 dst-port=4000 \
    in-interface-list=all protocol=tcp to-addresses=192.168.0.101
add action=dst-nat chain=dstnat dst-address=192.168.200.103 dst-port=4000 \
    in-interface-list=WAN protocol=tcp to-addresses=192.168.0.103
add action=dst-nat chain=dstnat dst-address=192.168.200.102 dst-port=4000 \
    in-interface-list=WAN protocol=tcp to-addresses=192.168.0.102
/ip route
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=\
    192.168.200.1 pref-src=0.0.0.0 routing-table=out_WAN2 scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=\
    192.168.13.31 pref-src=0.0.0.0 routing-table=out_WAN1 scope=30 \
    suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set winbox address=0.0.0.0/0
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=\
    33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

Sob · Sat Apr 30, 2022 12:58 pm

You do need some route(s) in main routing table, otherwise the initial routing decision fails, because it uses main routing table. You do mark routing for those packets, but it happens only after that. Add e.g. this:

/ip route
add dst-address=0.0.0.0/0 gateway=192.168.13.31 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.200.1 distance=2

Router will use it for own connections to outside (e.g. to access DNS server, check for updates, etc), and also for initial routing decision. But your mangle rules will override it for marked connections.

Zacharias · Sat Apr 30, 2022 8:51 pm

@sob i don't agree with you.
Yes there are cases where not having a Main Routing Table can cause situations where the packets can't be routed and are dropped or are routed in a wrong way.
But not in this case.

The OP describes that can't reach from B corp the Router 200.1.
The packet looking the Packet flow on for example input chain https://help.mikrotik.com/docs/display/ ... erOS-Input
the packet first goes to the prerouting chain, it will be marked accordingly and only after that will go to the Routing Decision where it is already marked and will have no problems to be routed as it should.
The same applies to the forward chain, first is the prerouting chain, and only after that the routing decision follows.

Not having a main routing table would cause problems e.g. for packets initiated from the Router, in IPsec cases etc.

For the OP, if you trace route to 200.1, do the packets go to the correct way or not ? I can tell they are routed to the Internet right ?
You should accept your directly connected networks.
Read here https://wiki.mikrotik.com/wiki/Manual:P ... _Balancing

anav · Sat Apr 30, 2022 8:57 pm

The router is smarter than the 3 morons above me. ;-PP
Its has recognized that opening up the port for winbox to anything other than the LAN is hazardous to its health.
This is a preservation move by the router and you will not find a solution anywhere, as its neurons are not programmable!

/ip firewall filter
add action=accept chain=input comment=WinBox dst-port=8291 protocol=tcp

of course if you thought that was bad it can get worse.
add action=accept chain=input comment=WinBox dst-port=80 protocol=tcp

The entire IP address schema baffles me, as not sure why what has been chosen has been chosen, I would think setting statiC DHCPs would suffice???
In any case its probably one of those cases that is perfectly fine and I just dont understand the logic.
Seems to be a pattern of setting the IP address for the subnet and then individually assigning other addresses on the port and then the bridge running over ether2 as well.......sigh.

/ip address
add address=192.168.0.1/24 comment=defconf interface=bridge network=\
192.168.0.0
add address=192.168.13.101/24 interface=ether1 network=192.168.13.0
add address=192.168.200.2/24 interface=ether2 network=192.168.200.0
add address=192.168.200.101/24 interface=ether2 network=192.168.200.0
add address=192.168.13.1/24 interface=ether1 network=192.168.13.0
add address=192.168.13.102/24 interface=ether1 network=192.168.13.0
add address=192.168.13.103/24 interface=ether1 network=192.168.13.0
add address=192.168.200.102/24 interface=ether2 network=192.168.200.0
add address=192.168.200.103/24 interface=ether2 network=192.168.200.0

My bad, there is only one LAN address, the bridge the rest are WAN1 and WAN2, each with multiple IPs of some sort.............

Zacharias · Sat Apr 30, 2022 9:04 pm

@anav you are right on that.
However i thought the OP was trying to access 200.1 as you can see from my previous answer and not 200.2.
Anyways, the OP indicated that it works on some cases, so i am confused now, maybe the tests where made with firewall disabled ? who knows...

Sob · Sat Apr 30, 2022 9:43 pm

@Zacharias: I've been wrong before, it can happen (don't tell @anav), but I think that I'll last a little longer before I inevitably repeat it.

OP wants to connect to 192.168.200.2:8291, that's this router, so you should be rather looking at output chain as the same page.

Another possible solution is routing rules like:

/routing rule
add src-address=192.168.200.2/32 action=lookup table=out_WAN2

Zacharias · Sun May 01, 2022 12:35 pm

@sob you are right, sorry.
I just missed that the reply will have nowhere to go because of the missing main Routing Table and that the Routing decision in Output chain is earlier than the Routing Adjustment.

Ufdah · Sun May 01, 2022 7:48 pm

Thank you all for your responses. I really appreciate the input. I haven’t been able to try @Sob’s route rule and won’t be able to until I get back to work Monday. I’m hoping this is the piece that I need.

As I mentioned at the end of the OP, whenever I add a main table routing rule it seems to break the marked connections. Is that the way the packet/rule flow is supposed to work or do I have something wrong there?

Ufdah · Sun May 01, 2022 8:07 pm

The entire IP address schema baffles me, as not sure why what has been chosen has been chosen, I would think setting statiC DHCPs would suffice???
In any case its probably one of those cases that is perfectly fine and I just dont understand the logic.
Seems to be a pattern of setting the IP address for the subnet and then individually assigning other addresses on the port and then the bridge running over ether2 as well.......sigh.

Prior to adding this router, A Corp was the only one connected to the three RTU’s. The RTU’s are dumb measurement devices in a a pipeline system that only have static IP’s and don’t broadcast data, they only respond to poll requests from the server. The entire network from the servers to the RTU’s is isolated from the internet. (Private cellular APN with no routes in or out to the world.) The three RTU’s were on the Cell 1 LAN at 192.168.13.101/102/103. However my company, B Corp just purchased a potion of A Corps assets and needs access to the same measurement data. The result is that we are adding our own Cell routers but can’t just add them to the 13.0/24 subnet. I have to add a intermediate gateway with routing rules to allow the RTU’s to communicate with two different servers. Because of this I’m using port forwarding in the WAN connections of the new router, hence the multiple WAN IP addresses. Hopefully that makes sense. I don’t know any way around this setup and it gives us the best network isolation between A and B Corp that I know how to provide, while still sharing end devices.

Ufdah · Mon May 02, 2022 5:33 pm

Another possible solution is routing rules like:
Code: Select all
/routing rule
add src-address=192.168.200.2/32 action=lookup table=out_WAN2

THIS. Thank you @Sob! I added this to the /Routing Rules and everything is working as I expect it to! Mangle is tracking packets and router generated traffic goes where I want it to. Thank you!

Using Mangle Is Breaking Remote Access

Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Re: Using Mangle Is Breaking Remote Access

Who is online