UDP connection tracking not working (OpenVPN)

arnaldo · Tue Feb 15, 2022 2:12 am

I'm having a problem with connection tracking and UDP, specifically with OpenVPN (ROSv7).

While the actual configuration is way more complex (PCC load balancing and recursive routing), I have simplyfied the scenario for testing and for posting here.

- There are 2 ISPs: ISP_A and ISP_B
- Configuration is for failover using different distances: 1 for ISP_A and 2 for ISP_B
- Both IP addresses are resolvable on the Internet side
- Mangle/prerouting is used to connection-mark each inbound connection, to allow returning through the proper external interface
- For fowarding connections, Mangle/prerouting is used to set route-mark based on the connection mark (at the start and at the end of the prerouting chain)
- For output connections, Mangle/output is used to convert the connection-mark to route-mark.
- There are route tables with the proper routes to force traffic to a specific ISP, to ensure proper connections.

All works well with TCP connections (both for forwarding and for output). The only other protocol that traverses or leaves the firewall are SIP (properly set in services and working fine for years) and OpenVPN.

My problem is with OpenVPN, inbound: if the connection is addressed the the IP address of ISP_A (that is the active "default" route) it works. But if the connection is addressed to the IP address of ISP_B, that is the backup route, even though the initial UDP packet is being properly connection marked in Mangle/prerouting.

In other words: UDP OpenVPN connections to the server using ISP_A work, and will NOT work usign ISP_B as the target. TCP OpenVPN work for both IP addresses. Log shows the connection but will not show the authentication (I guess it means that the initial package is recognized but after that it stops). Clients claim the conntecion timed-out.

Any ideas?

sindy · Tue Feb 15, 2022 9:14 am

When you use /tool sniffer quick port=1194 ip-address=ip.of.the.client, can you see the OVPN client requests incoming via ISP B to be indeed responded via ISP B?

arnaldo · Thu Feb 17, 2022 8:05 pm

Packages addressed to the IP address of ISP_B are being returned thru ISP_A, as if connection tracking was not working. But only with UDP packages. TCP works fine.

sindy · Thu Feb 17, 2022 8:07 pm

In that case, follow the advice in my automatic signature below. Something must be wrong in the config (or, less likely, in RouterOS 7).

arnaldo · Thu Feb 17, 2022 11:30 pm

Ok. Here it is (just the relevant parts)

# feb/17/2022 17:56:51 by RouterOS 7.1.2
# software id = MMRW-IEXQ
# model = RB5009UG+S+
/interface ethernet
set [ find default-name=ether1 ] comment="Ethernet Port #1"
set [ find default-name=ether5 ] comment="Ethernet Port #5 - LAN Interface #1"
set [ find default-name=ether6 ] comment="Ethernet Port #6 - LAN Interface #2"
set [ find default-name=ether7 ] comment="Ethernet Port #7 - LAN Interface #3"
set [ find default-name=ether8 ] comment="Ethernet Port #8 - LAN Interface #4"
set [ find default-name=ether2 ] comment="Ethernet Port #2" name=ISP_B 
set [ find default-name=ether3 ] comment="Ethernet Port #3 - WAN Reserved A" name=reserved_a
set [ find default-name=ether4 ] comment="Ethernet Port #4 - WAN Reserved B" name=reserved_b
set [ find default-name=sfp-sfpplus1 ] comment="10GB Ethernet SFP+ Port #1"
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add comment="Real Interfaces that can have Internet connectivitry (WANs that are not VPN tunnels)" name="True WAN"
/ip pool
add name=openvpn-pool ranges=10.10.13.0/24
/ppp profile
add change-tcp-mss=yes comment="OpenVPN Profile" dns-server=10.10.10.1 local-address=10.10.12.253 name=openvpn-profile only-one=no remote-address=openvpn-pool use-compression=no use-encryption=required use-ipv6=no use-mpls=no use-upnp=yes
/routing table
add comment="Routing table that balances traffic thru ISP_A" disabled=no fib name=to_ISP_A
add comment="Routing table that balances traffic thru ISP_B" disabled=no fib name=to_ISP_B
add comment="Routing table that forces traffic thru ISP_A" disabled=no fib name=force_ISP_A
add comment="Routing table that forces traffic thru ISP_B" disabled=no fib name=force_ISP_B
/interface bridge port
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=ether6
add bridge=bridge comment=defconf interface=ether7
add bridge=bridge comment=defconf interface=ether8
add bridge=bridge comment=defconf interface=sfp-sfpplus1
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ISP_B list=WAN
add interface=ISP_A list=WAN
add interface=ether1 list="True WAN"
add interface=ISP_B list="True WAN"
add interface=ISP_A list="True WAN"
add interface=wireguard1 list=LAN
/interface ovpn-server server
set auth=sha1 certificate=server-br cipher=blowfish128,aes256 default-profile=openvpn-profile enabled=yes netmask=22 protocol=udp
/ip address
add address=10.10.10.254/22 interface=bridge network=10.10.8.0
add cname=zeon.viegasdelima.com name=pbx.viegasdelima.com type=CNAME

/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="Fasttrack - for packets in the main routing table (unmarked and established/related)" connection-state=established,related disabled=yes hw-offload=yes packet-mark=no-mark
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked 
add action=accept chain=input comment="WireGuard inbound traffic" dst-port=13231 protocol=udp
add action=accept chain=input comment="OpenVPN inbound traffic" dst-port=1194 protocol=udp
add action=accept chain=input comment="WireGuard/OpenVPN clients are considered \"inside\"" src-address=10.10.12.0/22
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="Allow from VPN clients" src-address=10.10.12.0/22
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list="True WAN"

/ip firewall mangle
add action=jump chain=prerouting comment="route-mark: pass #1: if already connection marked, call route-mark and get out from there" connection-mark=!no-mark in-interface-list=LAN jump-target=route-mark
add action=mark-connection chain=prerouting comment="Connection mark new inbound connections - ISP_A" connection-mark=no-mark in-interface=pppoe-ISP_A new-connection-mark=to-ISP_A passthrough=no
add action=mark-connection chain=prerouting comment="Connection mark new inbound connections - ISP_B" connection-mark=no-mark in-interface=ISP_B new-connection-mark=to-ISP_B passthrough=no
add action=jump chain=prerouting comment="If not connection marked, use PCC to select outbound link" connection-mark=no-mark in-interface-list=LAN jump-target=pcc
add action=jump chain=prerouting comment="route-mark: pass #2: if newly connection marked, select route" connection-mark=!no-mark in-interface-list=LAN jump-target=route-mark

add action=jump chain=pcc comment="Send TCP traffic (except HTTPS) to address+port classifier" dst-port=!443 jump-target=pcc-port protocol=tcp
add action=jump chain=pcc comment="Send UDP traffic to address+port classifier" jump-target=pcc-port protocol=udp
add action=jump chain=pcc comment="Classify remaining outbound packets - HTTPS and non TCP/UDP" connection-mark=no-mark jump-target=pcc-addr

add action=mark-connection chain=pcc-addr comment="Classify outbound packets per addresses-pair (6:5 ratio)" new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses:11/0
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses:11/1
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses:11/2
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses:11/3
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses:11/4
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses:11/5
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses:11/6
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses:11/7
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses:11/8
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses:11/9
add action=mark-connection chain=pcc-addr new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses:11/10
add action=mark-connection chain=pcc-port comment="Classify UDP and non HTTPS TCP outbound packets (6:5 ratio)" new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses-and-ports:11/0
add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses-and-ports:11/1 add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses-and-ports:11/2
add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses-and-ports:11/3 add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses-and-ports:11/4
add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_A passthrough=yes per-connection-classifier=both-addresses-and-ports:11/5 add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses-and-ports:11/6
add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses-and-ports:11/7 add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses-and-ports:11/8
add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses-and-ports:11/9 add action=mark-connection chain=pcc-port new-connection-mark=to-ISP_B passthrough=yes per-connection-classifier=both-addresses-and-ports:11/10

add action=mark-routing chain=route-mark comment="Route mark outbound packets connection marked for ISP_B (ISP_B)" connection-mark=to-ISP_B new-routing-mark=to_ISP_B passthrough=no
add action=mark-routing chain=route-mark comment="Route mark outbound packets connection marked for pppoe-ISP_A (ISP_A)" connection-mark=to-ISP_A new-routing-mark=to_ISP_A passthrough=no
add action=mark-routing chain=route-mark comment="Force route mark outbound packets connection marked for pppoe-ISP_A (ISP_A) - no fallback" connection-mark=force-ISP_A new-routing-mark=force_ISP_A passthrough=no
add action=mark-routing chain=route-mark comment="Force route mark outbound packets connection marked for ISP_B (ISP_B) - no fallback" connection-mark=force-ISP_B new-routing-mark=force_ISP_B passthrough=no

add action=jump chain=output comment="Add route-mark to ouput connection marked packets (for inbound traffic) " connection-mark=!no-mark jump-target=route-mark out-interface-list="True WAN"

/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN

/ip route
add comment="Link level route for ISP_B - check #1 Google" distance=1 dst-address=8.8.4.4/32 gateway=172.16.1.254 scope=10
add comment="Link level route for ISP_B - check #2 Cloudfare" distance=1 dst-address=1.0.0.1/32 gateway=172.16.1.254 scope=10
add check-gateway=ping comment="Virtual hop ISP_A (check #1 Google)" distance=1 dst-address=10.1.1.1/32 gateway=8.8.8.8 scope=10 target-scope=11
add check-gateway=ping comment="Virtual hop ISP_A (check #2 Cloudfare)" distance=1 dst-address=10.1.1.1/32 gateway=1.1.1.1 scope=10 target-scope=11
add check-gateway=ping comment="Virtual hop ISP_B (check #1 Google)" distance=1 dst-address=10.2.2.2/32 gateway=8.8.4.4 scope=10 target-scope=11
add check-gateway=ping comment="Virtual hop ISP_B (check #2 Cloudfare)" distance=1 dst-address=10.2.2.2/32 gateway=1.0.0.1 scope=10 target-scope=11
add comment="to_ISP_A using ISP_A as primary" distance=1 gateway=10.1.1.1 routing-table=to_ISP_A scope=30 target-scope=12
add comment="to_ISP_A using ISP_B as backup" distance=2 gateway=10.2.2.2 routing-table=to_ISP_A scope=30 target-scope=12
add comment="to_ISP_B using ISP_B as primary" distance=1 gateway=10.2.2.2 routing-table=to_ISP_B scope=30 target-scope=12
add comment="to_ISP_B using ISP_A as backup" distance=2 gateway=10.1.1.1 routing-table=to_ISP_B scope=30 target-scope=12
add comment="Route for unmarked traffic - ISP_A" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.1.1.1 pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no target-scope=12 
add comment="Route for unmarked traffic - ISP_B" distance=2 gateway=10.2.2.2 scope=30 target-scope=12
add comment="Force (no fallback) use of ISP_A" distance=1 dst-address=0.0.0.0/0 gateway=pppoe-ISP_A routing-table=force_ISP_A scope=30 target-scope=12
add blackhole comment="Blackhole to ensure that \"forced\" route will never fallback to default routing table - ISP_A" distance=20 routing-table=force_ISP_A
add comment="Force (no fallback) use of ISP_B" distance=1 dst-address=0.0.0.0/0 gateway=172.16.1.254 routing-table=force_ISP_B scope=30 target-scope=12
add blackhole comment="Blackhole to ensure that \"forced\" route will never fallback to default routing table - ISP_B" distance=20 routing-table=force_ISP_B
add comment="Link level route for ISP_A - check #1 Google" distance=1 dst-address=8.8.8.8/32 gateway=my.ispa-address scope=10
add comment="Link level route for ISP_A - check #2 Cloudfare" distance=1 dst-address=1.1.1.1/32 gateway=my.ispa-address scope=10
add disabled=no dst-address=10.10.192.0/20 gateway=wireguard1 routing-table=main suppress-hw-offload=no

sindy · Fri Feb 18, 2022 12:10 am

I don't know whether it is a consequence of anonymisation that went wrong, but if not, it should be the real reason:

/interface list member
add interface=ether1 list="True WAN"
add interface=ISP_B list="True WAN"
add interface=ISP_A list="True WAN"

/ip firewall mangle
...
add action=mark-connection chain=prerouting comment="Connection mark new inbound connections - ISP_A" \
connection-mark=no-mark in-interface=pppoe-ISP_A new-connection-mark=to-ISP_A passthrough=no
...
add action=jump chain=output comment="Add route-mark to ouput connection marked packets (for inbound traffic) "
connection-mark=!no-mark jump-target=route-mark out-interface-list="True WAN"

The assignment of routing marks to own outgoing traffic of the router is complicated - all the packets are first routed using routing table main, and only then they go through mangle in the output chain, and if they get a routing-mark there, they are routed once again (this is called "routing adjustment" on the packet flow diagram).

So if the response packet from the OpenVPN server gets routed via pppoe-ISP_A, the action=jump rule above doesn't match and send the packet to chain route-mark because pppoe-ISP_A is not a member of the "True WAN" interface list.

arnaldo · Mon Feb 21, 2022 10:25 pm

it is indeed a "consequence of anonymisation that went wrong" they are the same and they are correct in the real config.

The assignment of routing marks to own outgoing traffic of the router is complicated - all the packets are first routed using routing table main, and only then they go through mangle in the output chain, and if they get a routing-mark there, they are routed once again (this is called "routing adjustment" on the packet flow diagram).

Yes, it's tricky indeed, but I do mark the inbound connection on the magle preroute and mark-route on the magle output for the outbound packets. It works like a charm for TCP connections, including OpenVPN over TCP. But fails for OpenVPN over UDP, as if the conn-mark algorithm is not being able to fully handle the OpenVPN UDP traffic. It may be the case and it may required an ALG (or Helper) for it to work. As I said, if the return packets are to go through the active route it works. If it requires the route-mark to exit trough the inbound interface it will not work.

sindy · Tue Feb 22, 2022 1:05 pm

What I have problems to accept is that connection marking and routing marking should behave differently depending on IP protocol type (TCP, UDP), and that it should depend on which particular application sends it.

So I'd definitely run
/ip firewall connection print detail interval=1s where dst-address~"ip.of.wan.b:1194"
while trying to establish a connection from the client, to see whether the connection-mark is indeed assigned; if yes, I'd split the rule
action=jump chain=output connection-mark=!no-mark jump-target=route-mark out-interface-list="True WAN"
into two, one matching on protocol=udp and the other matching on protocol=tcp, and see whether the UDP one counts packets.
If the UDP one doesn't count, I'd replace out-interface-list="True WAN" by src-port=1194 and test again.

UDP connection tracking not working (OpenVPN)

UDP connection tracking not working (OpenVPN)

Re: UDP connection tracking not working (OpenVPN)

Re: UDP connection tracking not working (OpenVPN)

Re: UDP connection tracking not working (OpenVPN)

Re: UDP connection tracking not working (OpenVPN)

Re: UDP connection tracking not working (OpenVPN)

Re: UDP connection tracking not working (OpenVPN)

Re: UDP connection tracking not working (OpenVPN)