DUAL WAN PCC

zhex900 · Sat Jun 30, 2018 12:01 pm

Hi,

I need some help with dual wan PCC configuration. I have two routers connected by one cable on ether2. The two routers have their independent respective internet connection. What I want to achieve is for both routers to share their internet traffic.

Router A have subnet 192.168.80.0/24 and Router B have subnet 192.168.88.0/24. Router A ether2 have address 192.168.123.1 and Router B ether2 have address 192.168.123.2.

I have added a static router to connect Router A and Router B via ether2.

The problem is no internet traffic is going through ether2. Maybe the issue is when a packet arrives from ether2, the router does not know how to router to ether1.

Router A


[admin@MT-30] > export hide-sensitive  
# jun/30/2018 18:48:19 by RouterOS 6.42.3
# software id = 6I1X-92CP
#
# model = 2011UAS-2HnD
# serial number = 402602239286
/caps-man channel
add band=2ghz-b/g/n control-channel-width=20mhz name=2G
add band=5ghz-a/n/ac control-channel-width=20mhz name=5G
/interface bridge
add fast-forward=no name=COOKST
add admin-mac=00:0C:42:F8:A5:78 auto-mac=no comment=defconf name=bridge-local
/interface wireless
# managed by CAPsMAN
# channel: 2452/20-Ce/gn(36dBm), SSID: COOKST, CAPsMAN forwarding
set [ find default-name=wlan1 ] name=wlan2 ssid=MT-30A
/interface ethernet
set [ find default-name=ether1 ] mac-address=00:0C:42:F8:A5:77
set [ find default-name=ether2 ] mac-address=00:0C:42:F8:A5:78
set [ find default-name=ether3 ] mac-address=00:0C:42:F8:A5:79
set [ find default-name=ether4 ] mac-address=00:0C:42:F8:A5:7A
set [ find default-name=ether5 ] mac-address=00:0C:42:F8:A5:7B
set [ find default-name=ether6 ] mac-address=00:0C:42:F8:A5:7C
set [ find default-name=ether7 ] mac-address=00:0C:42:F8:A5:7D
set [ find default-name=ether8 ] mac-address=00:0C:42:F8:A5:7E
set [ find default-name=ether9 ] mac-address=00:0C:42:F8:A5:7F
set [ find default-name=ether10 ] mac-address=00:0C:42:F8:A5:80
set [ find default-name=sfp1 ] disabled=yes mac-address=00:0C:42:F8:A5:76
/caps-man datapath
add bridge=COOKST name=COOKST
/caps-man security
add authentication-types=wpa-psk,wpa2-psk comment=jesusislord encryption=aes-ccm,tkip name=COOKST
/caps-man configuration
add channel=2G country=australia datapath=COOKST distance=indoors mode=ap name=COOKST-2G security=COOKST ssid=COOKST
add channel=5G country=australia datapath=COOKST distance=indoors mode=ap name=COOKST-5G security=COOKST ssid=COOKST
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=COOKST ranges=10.0.0.20-10.0.0.200
add name=dhcp_pool3 ranges=192.168.80.20-192.168.80.254
/ip dhcp-server
add address-pool=COOKST disabled=no interface=COOKST name=COOKST
add address-pool=dhcp_pool3 disabled=no interface=bridge-local name=dhcp1
/caps-man manager
set enabled=yes
/caps-man provisioning
add action=create-dynamic-enabled hw-supported-modes=gn master-configuration=COOKST-2G name-format=identity name-prefix=COOKST
add action=create-dynamic-enabled hw-supported-modes=ac master-configuration=COOKST-5G name-format=identity
/interface bridge port
add bridge=bridge-local comment=defconf interface=ether3
add bridge=bridge-local comment=defconf interface=ether4
add bridge=bridge-local comment=defconf interface=ether5
add bridge=bridge-local comment=defconf interface=ether6
add bridge=bridge-local comment=defconf interface=ether7
add bridge=bridge-local comment=defconf interface=ether8
add bridge=bridge-local comment=defconf interface=ether9
add bridge=bridge-local comment=defconf interface=*1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge-local list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/interface wireless cap
# 
set caps-man-addresses=127.0.0.1 enabled=yes interfaces=wlan2
/ip address
add address=192.168.80.1/24 comment=defconf interface=bridge-local network=192.168.80.0
add address=192.168.123.1/30 interface=ether2 network=192.168.123.0
add address=10.0.0.1/24 interface=COOKST network=10.0.0.0
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=10.0.0.0/24 gateway=10.0.0.1
add address=192.168.80.0/24 gateway=192.168.80.1
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input dst-address-type=local src-address-type=local
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1 new-connection-mark=local_wan_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2 new-connection-mark=local_wan_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=local_wan_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=remote_wan_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:1/0
add action=mark-routing chain=prerouting connection-mark=local_wan_conn in-interface=bridge-local new-routing-mark=to_local_wan passthrough=yes
add action=mark-routing chain=prerouting connection-mark=remote_wan_conn in-interface=bridge-local new-routing-mark=to_remote_wan passthrough=yes
add action=mark-routing chain=output connection-mark=local_wan_conn new-routing-mark=to_local_wan passthrough=yes
add action=mark-routing chain=output connection-mark=remote_wan_conn new-routing-mark=to_remote_wan passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" out-interface=ether1
add action=masquerade chain=srcnat src-address=10.0.0.0/24
/ip route
add check-gateway=ping distance=1 gateway=192.168.123.2 routing-mark=to_remote_wan
add check-gateway=ping distance=1 dst-address=192.168.80.0/32 gateway=bridge-local
add check-gateway=ping distance=1 dst-address=192.168.88.0/24 gateway=192.168.123.2
/system clock
set time-zone-name=Australia/Sydney
/system identity
set name=MT-30
/system routerboard settings
set auto-upgrade=yes silent-boot=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

Router B

[admin@MT-30A] > export hide-sensitive  
# jun/30/2018 18:49:17 by RouterOS 6.42.1
# software id = YSFQ-H9GG
#
# model = 2011UAS-2HnD
# serial number = 419E0125FDC1
/caps-man channel
add band=2ghz-b/g/n control-channel-width=20mhz name=2G
add band=5ghz-a/n/ac control-channel-width=20mhz name=5G
/interface bridge
add fast-forward=no name=COOKST
add admin-mac=00:0C:42:F8:A5:78 auto-mac=no comment=defconf name=bridge-local
/interface wireless
# managed by CAPsMAN
# channel: 2447/20-Ce/gn(36dBm), SSID: COOKST1, CAPsMAN forwarding
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-Ce distance=indoors frequency=auto mode=ap-bridge ssid=MikroTik-F8A581 \
    wireless-protocol=802.11
/interface ethernet
set [ find default-name=sfp1 ] disabled=yes
/caps-man datapath
add bridge=COOKST name=COOKST
/caps-man security
add authentication-types=wpa-psk,wpa2-psk comment=jesusislord encryption=aes-ccm,tkip name=COOKST
/caps-man configuration
add channel=2G country=australia datapath=COOKST distance=indoors mode=ap name=COOKST-2G security=COOKST ssid=COOKST1
add channel=5G country=australia datapath=COOKST distance=indoors mode=ap name=COOKST-5G security=COOKST ssid=COOKST1
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=COOKST ranges=10.0.0.20-10.0.0.200
/ip dhcp-server
add address-pool=default-dhcp disabled=no interface=bridge-local name=defconf
add address-pool=COOKST disabled=no interface=COOKST name=COOKST
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/caps-man manager
set enabled=yes
/caps-man provisioning
add action=create-dynamic-enabled hw-supported-modes=gn master-configuration=COOKST-2G name-format=identity name-prefix=COOKST
add action=create-dynamic-enabled hw-supported-modes=ac master-configuration=COOKST-5G name-format=identity
/interface bridge port
add bridge=bridge-local comment=defconf interface=ether3
add bridge=bridge-local comment=defconf interface=ether4
add bridge=bridge-local comment=defconf interface=ether5
add bridge=bridge-local comment=defconf interface=ether6
add bridge=bridge-local comment=defconf interface=ether7
add bridge=bridge-local comment=defconf interface=ether8
add bridge=bridge-local comment=defconf interface=ether9
add bridge=bridge-local comment=defconf interface=wlan1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge-local list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/interface wireless cap
# 
set caps-man-addresses=127.0.0.1 enabled=yes interfaces=wlan1
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge-local network=192.168.88.0
add address=192.168.123.2/30 interface=ether2 network=192.168.123.0
add address=10.0.0.1/24 interface=COOKST network=10.0.0.0
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=10.0.0.0/24 gateway=10.0.0.1
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8
/ip dns static
add address=192.168.88.1 name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input dst-address-type=local src-address-type=local
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1 new-connection-mark=local_wan_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2 new-connection-mark=local_wan_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=local_wan_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=remote_wan_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:1/0
add action=mark-routing chain=prerouting connection-mark=local_wan_conn in-interface=bridge-local new-routing-mark=to_local_wan passthrough=yes
add action=mark-routing chain=prerouting connection-mark=remote_wan_conn in-interface=bridge-local new-routing-mark=to_remote_wan passthrough=yes
add action=mark-routing chain=output connection-mark=local_wan_conn new-routing-mark=to_local_wan passthrough=yes
add action=mark-routing chain=output connection-mark=remote_wan_conn new-routing-mark=to_remote_wan passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" out-interface=ether1
add action=masquerade chain=srcnat src-address=10.0.0.0/24
/ip route
add check-gateway=ping distance=1 gateway=192.168.123.1 routing-mark=to_remote_wan
add check-gateway=ping distance=1 dst-address=192.168.80.0/24 gateway=192.168.123.1
/system clock
set time-zone-name=Australia/Sydney
/system identity
set name=MT-30A
/system routerboard settings
set auto-upgrade=yes silent-boot=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool user-manager database
set db-path=user-manager

sindy · Sun Jul 01, 2018 12:08 pm

OK, you're almost there.

Just a few mistakes in /ip firewall filter, but each of them ruins it all:

if you want to use fasttracking, you must exclude connections marked with remote_wan_conn from it. Otherwise, mid-connection packets belonging to these connections wouldn't be handled by mangle and thus wouldn't get a routing mark. So the initial packet of these connections would take the marked route and almost all subsequent ones would take the non-marked route, so the recipient would receive them from a different public IP and would be unable to identify them as belonging to the same connection.
So add connection-mark=!remote_wan_conn to the rule with action=fasttrack-connection.
the first packet of a connection initiated by other router's client which comes to the local router via ether2 is not related,established yet, but you have made ether2 a member of interface list name=WAN, so the firewall rule referring to in-interface-list=WAN drops that packet and the connection never establishes. So either consider the connection between the two machines safe and remove ether2 from interface list name=WAN, or think about adding some narrowing conditions for this rule, such as dst-address=!192.168.0.0/16 src-address=192.168.0.0/16 (i.e. make the rule ignore and thus let through packets from private addresses to public ones - just an example, may need some fine-tuning).

Other than that, there is no need to mark packets with routing-mark=to-local-wan if there is no route with that routing mark, so you can remove that rule.

So try to fix the mistakes above and come back with the result.

BTW, router A's name is MT30, router B's name is MT30A - you must originally come from Russia or work in secret services

zhex900 · Tue Jul 03, 2018 1:23 pm

Hi,

It is almost working. There is still an ignoring problem. I get 20-50% packet loss! The result is the same if turn off either ether1 or ether2. At least I know both ports (ether1 and ether2) can serve internet traffic.

Packet loss behaviour is not consistent. Packet loss reduces as I ping more. But this is random too.

Maybe it is a persistent user sessions issue. Should I use NTH load balancing with masquerade? https://wiki.mikrotik.com/wiki/NTH_load ... masquerade

--- iinet.net.au ping statistics ---
41 packets transmitted, 20 packets received, 51.2% packet loss
round-trip min/avg/max/stddev = 64.100/147.139/594.936/137.748 ms

Here are the export for one router.

[admin@MT-30A] > export hide-sensitive    
# jul/03/2018 20:14:53 by RouterOS 6.42.1
# software id = YSFQ-H9GG
#
# model = 2011UAS-2HnD
# serial number = 419E0125FDC1
/caps-man channel
add band=2ghz-b/g/n control-channel-width=20mhz name=2G
add band=5ghz-a/n/ac control-channel-width=20mhz name=5G
/interface bridge
add admin-mac=00:0C:42:F8:A5:78 auto-mac=no comment=defconf name=bridge-local
/interface wireless
# managed by CAPsMAN
# channel: 2412/20-Ce/gn(36dBm), SSID: COOKST1, CAPsMAN forwarding
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-Ce distance=indoors frequency=auto mode=ap-bridge ssid=MikroTik-F8A581 \
    wireless-protocol=802.11
/interface ethernet
set [ find default-name=sfp1 ] disabled=yes
/caps-man datapath
add bridge=bridge-local name=COOKST
/caps-man security
add authentication-types=wpa-psk,wpa2-psk comment=jesusislord encryption=aes-ccm,tkip name=COOKST
/caps-man configuration
add channel=2G country=australia datapath=COOKST distance=indoors mode=ap name=COOKST-2G security=COOKST ssid=COOKST1
add channel=5G country=australia datapath=COOKST distance=indoors mode=ap name=COOKST-5G security=COOKST ssid=COOKST1
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server
add disabled=no interface=MT-30A-1 name=COOKST
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp disabled=no interface=bridge-local name=defconf
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/caps-man manager
set enabled=yes
/caps-man provisioning
add action=create-dynamic-enabled hw-supported-modes=gn master-configuration=COOKST-2G name-format=identity name-prefix=COOKST
add action=create-dynamic-enabled hw-supported-modes=ac master-configuration=COOKST-5G name-format=identity
/interface bridge port
add bridge=bridge-local comment=defconf interface=ether3
add bridge=bridge-local comment=defconf interface=ether4
add bridge=bridge-local comment=defconf interface=ether5
add bridge=bridge-local comment=defconf interface=ether6
add bridge=bridge-local comment=defconf interface=ether7
add bridge=bridge-local comment=defconf interface=ether8
add bridge=bridge-local comment=defconf interface=ether9
add bridge=bridge-local comment=defconf disabled=yes interface=wlan1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge-local list=LAN
add comment=defconf interface=ether1 list=WAN
/interface wireless cap
# 
set caps-man-addresses=127.0.0.1 enabled=yes interfaces=wlan1
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge-local network=192.168.88.0
add address=192.168.123.2/30 interface=ether2 network=192.168.123.0
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=10.0.0.0/24 gateway=10.0.0.1
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,198.142.152.164,198.142.152.165
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input dst-address-type=local src-address-type=local
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-mark=!remote_wan_conn connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1 new-connection-mark=local_wan_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2 new-connection-mark=local_wan_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=local_wan_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=remote_wan_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:1/0
add action=mark-routing chain=prerouting connection-mark=local_wan_conn in-interface=bridge-local new-routing-mark=to_local_wan passthrough=yes
add action=mark-routing chain=prerouting connection-mark=remote_wan_conn in-interface=bridge-local new-routing-mark=to_remote_wan passthrough=yes
add action=mark-routing chain=output connection-mark=local_wan_conn new-routing-mark=to_local_wan passthrough=yes
add action=mark-routing chain=output connection-mark=remote_wan_conn new-routing-mark=to_remote_wan passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" out-interface-list=WAN
add action=masquerade chain=srcnat src-address=10.0.0.0/24
/ip route
add check-gateway=ping distance=1 gateway=192.168.123.1 routing-mark=to_remote_wan
add check-gateway=ping distance=1 dst-address=192.168.80.0/24 gateway=192.168.123.1
/system clock
set time-zone-name=Australia/Sydney
/system identity
set name=MT-30A
/system routerboard settings
set auto-upgrade=yes silent-boot=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool user-manager database
set db-path=user-manager

sindy · Tue Jul 03, 2018 10:11 pm

there is one mistake in how you use the per-connection-classifier, its parameter in the second rule should read 2/1 rather than 1/0 - the per-connection-classifier does not work the same way like nth which indeed counts the passes through the rule so you have to decrease the parameter with each rule with nth; here, the hash of addresses and ports is the same in both rules, and the per-connection-classifier takes the hash, divides it by the first number and compares the reminder to the second number. So if the first number is 2, the valid values of the second one are 0 and 1.
the rules translating connection-mark=local_wan_conn to new-routing-mark=to_local_wan are actually redundant as leaving the packets without any routing-mark makes them use the default route in the default table, and as there is no route with routing-mark=to_local_wan, packets with that routing-mark are handled by the default table ("main") anyway.

However, none of these mistakes should cause packet loss. If it would be only on ether2 I would suspect the cable between the houses to be too long, but if it happens via both ether1 and ether2, I would rather suspect the cable between the test machine and the router. I cannot see anything in the firewall settings that should cause this.

zhex900 · Wed Jul 04, 2018 12:30 pm

It's better now. But I still get 25% packet loss. The ping packet loss is only after pinging for a few minutes. The initial ping is fine. Packet loss increases as ping continues. ether2 cable is brand new and it is very short. The routers are right next to each other.

If I switch off either ether1 or ether2, there are no packet loss.

[admin@MT-30A] > export hide-sensitive  
# jul/04/2018 19:29:18 by RouterOS 6.42.1
# software id = YSFQ-H9GG
#
# model = 2011UAS-2HnD
# serial number = 419E0125FDC1
/caps-man channel
add band=2ghz-b/g/n control-channel-width=20mhz name=2G
add band=5ghz-a/n/ac control-channel-width=20mhz name=5G
/interface bridge
add admin-mac=00:0C:42:F8:A5:78 auto-mac=no comment=defconf name=bridge-local
/interface wireless
# managed by CAPsMAN
# channel: 2452/20-Ce/gn(36dBm), SSID: COOKST1, CAPsMAN forwarding
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-Ce distance=indoors frequency=auto mode=ap-bridge ssid=MikroTik-F8A581 \
    wireless-protocol=802.11
/interface ethernet
set [ find default-name=sfp1 ] disabled=yes
/caps-man datapath
add bridge=bridge-local name=COOKST
/caps-man security
add authentication-types=wpa-psk,wpa2-psk comment=jesusislord encryption=aes-ccm,tkip name=COOKST
/caps-man configuration
add channel=2G country=australia datapath=COOKST distance=indoors mode=ap name=COOKST-2G security=COOKST ssid=COOKST1
add channel=5G country=australia datapath=COOKST distance=indoors mode=ap name=COOKST-5G security=COOKST ssid=COOKST1
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server
add disabled=no interface=MT-30A-1 name=COOKST
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp disabled=no interface=bridge-local name=defconf
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/caps-man manager
set enabled=yes
/caps-man provisioning
add action=create-dynamic-enabled hw-supported-modes=gn master-configuration=COOKST-2G name-format=identity name-prefix=COOKST
add action=create-dynamic-enabled hw-supported-modes=ac master-configuration=COOKST-5G name-format=identity
/interface bridge port
add bridge=bridge-local comment=defconf interface=ether3
add bridge=bridge-local comment=defconf interface=ether4
add bridge=bridge-local comment=defconf interface=ether5
add bridge=bridge-local comment=defconf interface=ether6
add bridge=bridge-local comment=defconf interface=ether7
add bridge=bridge-local comment=defconf interface=ether8
add bridge=bridge-local comment=defconf interface=ether9
add bridge=bridge-local comment=defconf disabled=yes interface=wlan1
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge-local list=LAN
add comment=defconf interface=ether1 list=WAN
/interface wireless cap
# 
set caps-man-addresses=127.0.0.1 enabled=yes interfaces=wlan1
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge-local network=192.168.88.0
add address=192.168.123.2/30 interface=ether2 network=192.168.123.0
/ip dhcp-client
add comment=defconf dhcp-options=hostname,clientid disabled=no interface=ether1
/ip dhcp-server network
add address=10.0.0.0/24 gateway=10.0.0.1
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,198.142.152.164,198.142.152.165
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input dst-address-type=local src-address-type=local
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-mark=!remote_wan_conn connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1 new-connection-mark=local_wan_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2 new-connection-mark=local_wan_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=local_wan_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=remote_wan_conn \
    passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1
add action=mark-routing chain=prerouting connection-mark=remote_wan_conn in-interface=bridge-local new-routing-mark=to_remote_wan passthrough=yes
add action=mark-routing chain=output connection-mark=remote_wan_conn new-routing-mark=to_remote_wan passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" out-interface-list=WAN
add action=masquerade chain=srcnat src-address=10.0.0.0/24
/ip route
add check-gateway=ping distance=1 gateway=192.168.123.1 routing-mark=to_remote_wan
add check-gateway=ping distance=1 dst-address=192.168.80.0/24 gateway=192.168.123.1
/system clock
set time-zone-name=Australia/Sydney
/system identity
set name=MT-30A
/system routerboard settings
set auto-upgrade=yes silent-boot=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool user-manager database
set db-path=user-manager

sindy · Wed Jul 04, 2018 1:25 pm

Wow. In this case, would you mind doing the following while both interfaces are enabled:

/tool sniffer set filter-interface=ether1,ether2 file-limit=8000 file-name=ping-loss.pcap filter-ip-protocol=icmp filter-operator-between-entries=and
Use /tool sniffer print to check that all filter-* items not listed above are empty.

Then, issue /tool sniffer start and start the ping from the connected PC and let it run until it starts reporting loss; finally, issue /tool sniffer stop, download the file and open it using Wireshark. If the file has reached the limit of 8 MB, it is likely that the last packets are not there so a higher limit may be necessary for another try. But for 140 bytes (icmp echo request+response) once per second, 8 MB file size should suffice for 15 hours.

What you'd be looking for is whether some of the icmp requests are missing completely in the file (they have serial numbers) and if not, whether all of them leave with the same source MAC address or whether some of them leave with a different one, indicating that they left via the other ethernet port (source IP address may differ as well but it is not a reliable indicator as it depends on additional factors).

If you prefer to post the file somewhere for inspection and ether1 has a public address, you should consider use of packetwrangler to anonymize the capture before posting it.

zhex900 · Wed Jul 04, 2018 2:26 pm

Sorry I cannot make any sense of the sniffer file.

https://drive.google.com/file/d/1dXzadC ... sp=sharing

How can I adapt to persistent user sessions PCC for my situation? Will it help?

https://wiki.mikrotik.com/wiki/NTH_load ... masquerade

sindy · Wed Jul 04, 2018 2:39 pm

In the file you've posted, there are only pings which the two RBs send to each other over the link between their ether2s due to the check-gateway=ping settings in /ip route.

So please post the output of /tool sniffer print.

zhex900 · Wed Jul 04, 2018 2:43 pm

check-gateway=ping Should I uncheck this in ip route?

[admin@MT-30] > /tool sniffer print 
                     only-headers: no
                     memory-limit: 100KiB
                    memory-scroll: yes
                        file-name: pingloss.pcap
                       file-limit: 8000KiB
                streaming-enabled: no
                 streaming-server: 202.58.60.194
                    filter-stream: no
                 filter-interface: ether1,ether2
               filter-mac-address: 
              filter-mac-protocol: 
                filter-ip-address: 
              filter-ipv6-address: 
               filter-ip-protocol: icmp
                      filter-port: 
                       filter-cpu: 
                 filter-direction: any
  filter-operator-between-entries: and
                          running: no

sindy · Wed Jul 04, 2018 2:51 pm

There is no need to uncheck check-gateway=ping, the Wireshark display filtering can be used to hide packets you're not interested in.

But as there are no other icmp packets than those generated by the to 'Tiks themselves, the question is what actually happens.

So please set the filter-interface in /tool sniffer to empty string and repeat the procedure.

It seems that you were sniffing on other 'Tik than the one to which your pinging PC was connected, or that the testing PC had another path to the internet and the ping went through that path, or that the sniffer filter hasn't parsed correctly the value of filter-interface parameter.

zhex900 · Wed Jul 04, 2018 3:15 pm

Here it is

https://drive.google.com/file/d/1dXzadC ... sp=sharing

192,168.80.253 is my laptop's ip address

sindy · Wed Jul 04, 2018 3:31 pm

Well, in this capture, I can see exactly two ping request/response requests from your laptop to 202.58.60.194 (whatever it is), with sequence numbers 0 and 1.

The rest is the check gateway ping between the devices and notifications about failures to contact various ports (dns, UPnP etc.) not related to your ping testing.

How many pings have you sent and received on the PC before you saw the loss and stopped the sniffing?

sindy · Wed Jul 04, 2018 3:36 pm

Another point, the last occurrence of the echo request and the first occurrence of echo reply have the same src and dst MAC address which is the same like the dst-address of the requests coming from the PC, so it seems as if 202.58.60.194 is actualy one of the own addresses of the 'Tik...?

zhex900 · Wed Jul 04, 2018 3:38 pm

I will post it again after packet loss occurs.

This is my university's ip address. unsw.edu.au 202.58.60.194

sindy · Wed Jul 04, 2018 3:48 pm

So it is external. In that case, I'm seriously derailed by the MAC address mystery. Have you configured any of the MAC addresses of any of the two devices manually, including the bridge ones?

And can you post the output of /interface ethernet print and /interface bridge print to make it possible to link the MAC addresses in the capture with Ethernet ports/bridges?

In particular, I can see that you have set that MAC address as admin-mac of the bridge; does it belong to one of member ports of that bridge?

zhex900 · Wed Jul 04, 2018 3:55 pm

Here is another file.
https://drive.google.com/file/d/1dXzadC ... sp=sharing

I don't know how to see the packet loss in Packet Sniffer.

sindy · Wed Jul 04, 2018 5:03 pm

After you open the file above in Wireshark, apply a display filter (icmp.ident == 58733) && !(eth.addr == 8c:85:90:86:a6:1d). It will show you only icmp echo requests and responses belonging to the same sequence (with same icmp id field) which went via ether1 (or, exactly, did not come from your PC nor were sent to it at L2).

By removing the filter-interface, we've made each packet be captured three times:

first on the etherX to which your PC is connected
next on the bridge of which this etherX is a member
next on the ether1 or ether2, whichever is chosen by the per-connection-classifier as the WAN.

By using the display filter above, we limit the display only to packets on ether1 or ether2 belonging to that sequence.

By adding "&& (icmp.no_resp)" to the end of the display filter expression and applying the change, you'll see that six packets have been sent out via ether1 and never received a response (the icmp dissector of Wireshark is tracking the requests and responses by IP addresses, icmp ids and icmp sequence numbers).

If you follow the same procedure for (icmp.ident == 12398) && !(eth.addr == 00:e0:4c:36:0c:12), you get approximately the same picture - all requests made it to the ether1 (no sequence number is missing), but some were not responded.

So to me this does not seem to be an issue of the policy routing - all request packets did go out through ether1 and some of them didn't get a response. Have you tried to disconnect ether2 (which automatically disables the route through it so even packets with routing-mark=to-remote-wan are routed via ether1) and ping for, say, 20 minutes that way?

I'm BTW not sure how per-connection-classifier deals the fact that there is no port number in icmp, maybe it uses icmp id instead of source or destination port when calculating the hash, maybe it doesn't. If it doesn't, all pings from the same LAN address to the same remote address should always take the same LAN.

Another point related to the above but not to the ping response loss - while disconnection of ether2 makes traffic automatically fail over to ether1 as described above, failover in the reverse direction is not that simple. If local port is included into the PCC hash, another attempt to establish TCP or UDP connections which has failed due to route unavailability should succeed because the PCC is likely to send the new attempt via the other WAN, but for icmp this may not be the case.

zhex900 · Wed Jul 04, 2018 11:18 pm

HI Sindy,

It is finally fixed. The problem I copied the config by restoring the backup from one router to another router. This caused the two routers to have the same mac address. I have reset the mac address on all interfaces. Changed the mac address on bridge-local. Now I have no packet loss.

Thank you!

Jake He

zhex900 · Wed Jul 04, 2018 11:27 pm

HI,
I have another question. The CPU will clock max at 93% when I run a speed test. Is it possible to use the switch chip to free up some load on the CPU? My guess this is not possible because mangle rules are processed by CPU.

sindy · Thu Jul 05, 2018 11:19 am

The problem I copied the config by restoring the backup from one router to another router. This caused the two routers to have the same mac address.

I already had this suspicion (copying config to the bit from one machine to another) written in one of my posts but then deleted it as too fantastic because I haven't even dreamed that you could have restored a backup, and by using the export this could happen only for the brigde where admin-mac is shown in your export posted but not for physical Ethernet ports where the MAC address is only shown in export if it has been manually changed.

The CPU will clock max at 93% when I run a speed test. Is it possible to use the switch chip to free up some load on the CPU? My guess this is not possible because mangle rules are processed by CPU.

Not only mangle rules, also the routing and NAT (and bridging of LAN-to-LAN traffic between the two swicth chips but that shouldn't be related to your observation). For all packets using the local WAN (including those forwarded between ether1 and ether2), fasttracking should minimize the amount of firewall processing of mid-connection packets; only packets between LAN devices and ether2 are handled by all firewall rules because we need to routing-mark them and thus we intentionally prevent these connections from getting fasttracked.

But 93% CPU load suggestss that your uplinks are several hundreds of Mbit/s, is that the case?

zhex900 · Thu Jul 05, 2018 1:50 pm

Hi,

My uplink is 25 MB/sec. 93% cpu only when I do a speed test. It only stays at this level for a few seconds.

Thank you for all your help.

I have just installed it in both houses. Hopefully, nothing will break.

sindy · Thu Jul 05, 2018 2:00 pm

If it is just a spike as many connections get established simultaneously, it makes more sense but can still cause packet loss during exploitation if similar situation happens (several clients establishing connection simultaneously). But there's nothing you could do except replacing the HW by a more powerful one.

DUAL WAN PCC

DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Re: DUAL WAN PCC

Who is online