PCC Mangle and routes reducing Client speeds ?

killa88 · Mon Oct 30, 2023 2:38 pm

Hi guys,
General tech enthusiast turned to in-house network guy for my office, dying at the hands of config issues

I've got 3 buildings on my Rb5009 on 1Gb Ethernet as well as fiber with Wan1 100Mb / wan2 100Mb / Wan3 50Mb / LTE 25Mb

I used PCC load balancing and followed the instruction form Martins Strods YT lecture, then modified it after stumbling upon a Mkt forum post from site of Daryll Swer
https://www.daryllswer.com/multi-wan-se ... -routeros/

I am facing certain issues that have forced me to be always present at the client side 24/7 physically or remotely at night to manage their issues and need help to resolve them (Haven’t left the office since 2 years+) - First the mangle rules were improper, then the failovers weren’t working and banking/email sites broke

So Need to basically understand the following
1) Are my PCC rules incorrect ?
2) My gateway changes frequently, so cpanels/ similar mail sites get sluggish
(email+uploads are priority)
Is this because of PCC? I Shifted them from -both add and ports- to -both add- only and it has gotten somewhat better otherwise were quite sluggish
3) I am always worried that if a WAN breaks down the other one is not seamlessly providing data to client side and if i am ever out in the field i am called back to check and verify
((1 WAN is problematic and i have to remove it/disable it))
4) I have to always physically go because THAT WAN1 above is the one with public IP address and i want to access Mkt remotely
((When WAN1 breaks i can't even access it via the cloud option that Mikrotik provides)) i tried setting wire guard for it but couldn’t get win box to work on it
5) is winbox accessible via multiple public ip's ?
6) My client side via ether5 is receiving almost half the bandwidth and I don’t know why, their speed and bandwidth should be clear and no latency.
Should i add them to LAN bridge? Due to thunderstorms our fiber with them broke almost 5 times a year and it is expensive for us to get it fixed so shifted them to CAT6 and test the link which was great
7) Some person i requested for help setup some CAKE bandwidth and I disabled them, but I think they might still be in effect, also shows it in the config

Lastly I have no confidence in the Routes I have made and think they have issues. Or maybe the distance?

Went through a lot of topics and tried to resolve the issues but could not get these done, any help appreciated

# oct/30/2023 13:56:06 by RouterOS 7.9.2
# software id = G15A-2IM9
#
# model = RB5009UG+S+

/interface bridge
add admin-mac=DC:2D:6E:DD:82:13 auto-mac=no comment=defconf name=bridge-LAN
/interface ethernet
set [ find default-name=ether1 ] disabled=yes name=ether1-PIE1
set [ find default-name=ether2 ] mtu=1508 name=ether2-TW
set [ find default-name=ether3 ] name=ether3-PIE3
set [ find default-name=ether4 ] disabled=yes name=ether4-LTE4
set [ find default-name=ether5 ] name=ether5-client-side-as
set [ find default-name=sfp-sfpplus1 ] name="sfp-sfpplus1-CLINET SIDE"
/interface wireguard
add disabled=yes listen-port=13231 mtu=1420 name=WG1
/interface vlan
add disabled=yes interface=bridge-LAN name=vlan11 vlan-id=11
add disabled=yes interface=bridge-LAN name=vlan12 vlan-id=12
add disabled=yes interface=bridge-LAN name=vlan13 vlan-id=13
add disabled=yes interface=bridge-LAN name=vlan14 vlan-id=14
add disabled=yes interface=bridge-LAN name=vlan15 vlan-id=15
add disabled=yes interface=bridge-LAN name=vlan16 vlan-id=16
add disabled=yes interface=bridge-LAN name=vlan17 vlan-id=17
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=pool-bridge-LAN ranges=192.168.100.2-192.168.100.254
add name=dhcp_pool1v11 ranges=192.168.11.2-192.168.11.254
add name=dhcp_pool2v12 ranges=192.168.12.2-192.168.12.254
add name=dhcp_pool3v13 ranges=192.168.13.2-192.168.13.254
add name=dhcp_pool4v14 ranges=192.168.14.2-192.168.14.254
add name=dhcp_pool5v15 ranges=192.168.15.2-192.168.15.254
add name=dhcp_pool6v16 ranges=192.168.16.2-192.168.16.254
add name=dhcp_pool7v17 ranges=192.168.17.2-192.168.17.254
add name=pool-ether5-as ranges=192.168.50.3-192.168.50.254
/ip dhcp-server
add address-pool=pool-bridge-LAN interface=bridge-LAN lease-time=10m name=defconf
add address-pool=dhcp_pool1v11 disabled=yes interface=vlan11 lease-time=10m name=dhcp1
add address-pool=dhcp_pool2v12 disabled=yes interface=vlan12 lease-time=10m name=dhcp2
add address-pool=dhcp_pool3v13 disabled=yes interface=vlan13 lease-time=10m name=dhcp3
add address-pool=dhcp_pool4v14 disabled=yes interface=vlan14 lease-time=10m name=dhcp4
add address-pool=dhcp_pool5v15 disabled=yes interface=vlan15 lease-time=10m name=dhcp5
add address-pool=dhcp_pool6v16 disabled=yes interface=vlan16 lease-time=10m name=dhcp6
add address-pool=dhcp_pool7v17 disabled=yes interface=vlan17 lease-time=10m name=dhcp7
add address-pool=pool-ether5-as interface=ether5-client-side-as name="dhcp for ether5"
/queue type
add cake-atm=ptm cake-diffserv=besteffort cake-mpu=88 cake-overhead=40 kind=cake name=cake-defaults
add cake-ack-filter=filter cake-atm=ptm cake-bandwidth=20.0Mbps cake-diffserv=besteffort cake-mpu=88 cake-nat=yes cake-overhead=40 kind=cake name=cake-upload
add cake-atm=ptm cake-bandwidth=20.0Mbps cake-diffserv=besteffort cake-mpu=88 cake-nat=yes cake-overhead=40 cake-wash=yes kind=cake name=cake-download
add cake-atm=ptm cake-bandwidth=100.0Mbps cake-diffserv=besteffort cake-mpu=88 cake-nat=yes cake-overhead=40 cake-wash=yes kind=cake name=cake_d_100m
add cake-ack-filter=filter cake-atm=ptm cake-bandwidth=100.0Mbps cake-diffserv=besteffort cake-mpu=88 cake-nat=yes cake-overhead=40 kind=cake name=cake_u_100m
/routing table
add disabled=no fib name="CIR Wan1"
add disabled=no fib name="TW Wan2"
add disabled=no fib name="PieFO Wan3"
add disabled=no fib name="Lte Wan4"
/interface bridge port
add bridge=bridge-LAN comment=defconf interface=ether6
add bridge=bridge-LAN comment=defconf interface=ether7
add bridge=bridge-LAN comment=defconf interface=ether8
add bridge=bridge-LAN interface="sfp-sfpplus1-CLINET SIDE" trusted=yes
/ip neighbor discovery-settings
set discover-interface-list=all
/interface list member
add interface=ether2-TW list=WAN
add interface=ether1-PIE1 list=WAN
add interface=ether3-PIE3 list=WAN
add interface=ether4-LTE4 list=WAN
add interface=ether7 list=LAN
add interface=ether6 list=LAN
add interface=ether8 list=LAN
add interface=ether5-client-side-as list=LAN
/interface wireguard peers
add allowed-address=XX interface=WG1 public-key="xx"
add allowed-address=XX interface=WG1 public-key="xx"
/ip address
add address=192.168.100.1/24 comment=defconf interface=bridge-LAN network=192.168.100.0
add address=Public Ip/29 interface=ether1-PIE1 network=100.80.90.48
add address=Public Ip/30 interface=ether2-TW network=110.39.145.136
add address=192.168.11.1/24 interface=vlan11 network=192.168.11.0
add address=192.168.12.1/24 interface=vlan12 network=192.168.12.0
add address=192.168.13.1/24 interface=vlan13 network=192.168.13.0
add address=192.168.14.1/24 interface=vlan14 network=192.168.14.0
add address=192.168.15.1/24 interface=vlan15 network=192.168.15.0
add address=192.168.16.1/24 interface=vlan16 network=192.168.16.0
add address=192.168.17.1/24 interface=vlan17 network=192.168.17.0
add address=192.168.1.1/24 interface=ether7 network=192.168.1.0
add address=192.168.50.1/24 interface=ether5-client-side-as network=192.168.50.0
add address=192.168.90.2/24 interface=ether4-LTE4 network=192.168.90.0
add address=Public Ip/30 interface=ether3-PIE3 network=X
add address=192.168.10.3/24 interface=ether3-PIE3 network=192.168.10.0
/ip arp
add address=XX interface=ether2-TW mac-address=XX
/ip cloud
set ddns-enabled=yes
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1-PIE1
add disabled=yes interface=ether3-PIE3 use-peer-ntp=no
/ip dhcp-server lease
add address=192.168.100.3 client-id=1:e8:6a:64:81:9d:d3 mac-address= server=defconf
add address=192.168.11.8 client-id=1:d4:25:8b:4d:e8:65 mac-address= server=dhcp1
/ip dhcp-server network
add address=192.168.1.0/24 gateway=192.168.1.1 netmask=24
add address=192.168.11.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.11.1
add address=192.168.12.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.12.1
add address=192.168.13.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.13.1
add address=192.168.14.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.14.1
add address=192.168.15.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.15.1
add address=192.168.16.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.16.1
add address=192.168.17.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.17.1
add address=192.168.50.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.50.1
add address=192.168.100.0/24 comment=defconf dns-server=8.8.8.8,8.8.4.4 gateway=192.168.100.1
/ip dns
set allow-remote-requests=yes cache-size=10000KiB max-udp-packet-size=512 servers=8.8.8.8,1.1.1.1
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall address-list
add address=192.168.100.0/24 list=LAN-ips
add address=192.168.0.0/19 list=LAN-ips
add address=192.168.12.0/24 list=VLAN-ips
add address=192.168.13.0/24 list=VLAN-ips
add address=192.168.14.0/24 list=VLAN-ips
add address=192.168.15.0/24 list=VLAN-ips
add address=192.168.16.0/24 list=VLAN-ips
add address=192.168.17.0/24 list=VLAN-ips
add address=10.10.0.0/29 list=VLAN-ips
add address=XxX list=Cpanel
add address=XxX list=Cpanel
add address=192.168.50.0/24 list="AS Client side IP"
add address=XxX list=BP
add address=x.x.x.x.254 comment="RC" list="LAN Printers"
add address=x.x.x.x.245 comment="XC - New Interface" list="LAN Printers"
add address=192.168.100.2-192.168.100.254 list=allowed_to_router
add address=192.168.11.2-192.168.11.254 list=allowed_to_router
add address=192.168.14.2-192.168.14.254 list=allowed_to_router
add address=10.10.10.2-10.10.10.254 list=allowed_to_router
add address=0.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=172.16.0.0/12 comment=RFC6890 list=not_in_internet
add address=192.168.0.0/16 comment=RFC6890 list=not_in_internet
add address=10.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=169.254.0.0/16 comment=RFC6890 list=not_in_internet
add address=127.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=224.0.0.0/4 comment=Multicast list=not_in_internet
add address=198.18.0.0/15 comment=RFC6890 list=not_in_internet
add address=192.0.0.0/24 comment=RFC6890 list=not_in_internet
add address=192.0.2.0/24 comment=RFC6890 list=not_in_internet
add address=198.51.100.0/24 comment=RFC6890 list=not_in_internet
add address=203.0.113.0/24 comment=RFC6890 list=not_in_internet
add address=100.64.0.0/10 comment=RFC6890 list=not_in_internet
add address=240.0.0.0/4 comment=RFC6890 list=not_in_internet
add address=192.88.99.0/24 comment="6to4 relay Anycast [RFC 3068]" list=not_in_internet
add address=192.168.100.207 comment="TMC Richo C305" list="LAN Printers"
add address=192.168.100.253 comment="HP Flow M880" list="LAN Printers"
add address=192.168.100.0/24 list=VLAN-ips
add address=192.168.50.0/24 list=LAN-ips
add address=192.168.10.0/24 list=Connected
add address=Public Ip list=Connected
add address=Public Ip list=Connected
add address=x.x.x.x list=BP
/ip firewall filter
add action=jump chain=forward comment="jump to ICMP filters" jump-target=icmp protocol=icmp
add action=fasttrack-connection chain=forward comment=FastTrack connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="Established, Related" connection-state=established,related
add action=accept chain=input comment="default configuration" connection-state=established,related
add action=accept chain=input src-address-list=allowed_to_router
add action=accept chain=input protocol=icmp
add action=drop chain=input connection-state=invalid
add action=accept chain=icmp comment="echo reply" icmp-options=0:0 protocol=icmp
add action=accept chain=forward src-address-list=VLAN-ips
add action=accept chain=forward dst-address-list="LAN Printers" dst-port=80,8010,9100-9103 protocol=tcp src-address-list=VLAN-ips
add action=accept chain=forward comment="EXECTIVE PRINTER" dst-address=192.168.100.254
add action=accept chain=forward comment="EXECTIVE PRINTER" dst-address=192.168.100.245
add action=accept chain=forward comment=NVR dst-address=192.168.100.50 src-address-list=LAN_IPS
add action=drop chain=forward comment="Drop invalid" connection-state=invalid log-prefix=invalid
add action=drop chain=forward comment="Drop incoming from internet which is not public IP" in-interface=ether1-PIE1 log=yes log-prefix=!public src-address-list=not_in_internet
add action=drop chain=forward comment="Drop incoming from internet which is not public IP" in-interface=ether2-TW log=yes log-prefix=!public src-address-list=not_in_internet
/ip firewall mangle
add action=accept chain=prerouting disabled=yes dst-address-list=LAN_IPS
add action=accept chain=prerouting connection-state="" dst-address-list=Connected src-address-list=Connected
add action=accept chain=prerouting comment="BHst+Cpanel traffic accept" connection-state="" dst-address-list=BP src-address-list=BP
add action=mark-connection chain=prerouting comment="BHst + Cpanel Mail traffic" connection-mark=no-mark connection-state=new dst-address-list=BP new-connection-mark=Mail-TW \
    passthrough=no src-address-list=LAN-ips
add action=mark-routing chain=prerouting comment="BHst + Cpanel Mail traffic" connection-mark=Mail-TW connection-state="" dst-address-list=BP new-routing-mark="TW Wan2" \
    passthrough=no src-address-list=LAN-ips
add action=mark-connection chain=input comment=";;;;;;;;;;;;Incoming Marking" connection-mark=no-mark connection-state="" in-interface=ether1-PIE1 new-connection-mark=\
    PIE1_Conn passthrough=no
add action=mark-connection chain=input comment=";;;;;;;;;;;;Incoming Marking" connection-mark=no-mark connection-state="" in-interface=ether2-TW new-connection-mark=TW_Conn \
    passthrough=no
add action=mark-connection chain=input comment=";;;;;;;;;;;;Incoming Marking" connection-mark=no-mark connection-state="" in-interface=ether3-PIE3 new-connection-mark=\
    PIE3_Conn passthrough=no
add action=mark-connection chain=input comment=";;;;;;;;;;;;Incoming Marking" connection-mark=no-mark connection-state="" in-interface=ether4-LTE4 new-connection-mark=\
    LTE4_Conn passthrough=no
add action=mark-connection chain=prerouting comment="nth for HTTP/s" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=PIE1_Conn nth=4,1 passthrough=yes
add action=mark-connection chain=prerouting comment="nth for HTTP/s" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=TW_Conn nth=4,2 passthrough=yes
add action=mark-connection chain=prerouting comment="nth for HTTP/s" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=PIE3_Conn nth=4,3 passthrough=yes
add action=mark-connection chain=prerouting comment="nth for HTTP/s" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=LTE4_Conn nth=4,4 passthrough=yes
add action=mark-connection chain=prerouting comment="PCC L/Bal UDP QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    dst-port=80,443 in-interface-list=LAN new-connection-mark=PIE1_Conn passthrough=yes per-connection-classifier=both-addresses:4/0 protocol=udp
add action=mark-connection chain=prerouting comment="PCC L/Bal UDP QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    dst-port=80,443 in-interface-list=LAN new-connection-mark=TW_Conn passthrough=yes per-connection-classifier=both-addresses:4/1 protocol=udp
add action=mark-connection chain=prerouting comment="PCC Load Balancing UDP Tr. W1" connection-mark=no-mark connection-state="" disabled=yes dst-address-list=!not_in_internet \
    dst-address-type=!local dst-port=80,443 in-interface-list=LAN new-connection-mark=TW_Conn passthrough=yes per-connection-classifier=both-addresses-and-ports:5/2 protocol=\
    udp
add action=mark-connection chain=prerouting comment="PCC L/Bal UDP QUIC." connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    dst-port=80,443 in-interface-list=LAN new-connection-mark=PIE3_Conn passthrough=yes per-connection-classifier=both-addresses:4/2 protocol=udp
add action=mark-connection chain=prerouting comment="PCC L/Bal UDP QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    dst-port=80,443 in-interface-list=LAN new-connection-mark=LTE4_Conn passthrough=yes per-connection-classifier=both-addresses:4/3 protocol=udp
add action=mark-connection chain=prerouting comment="PCC L/Bal Tcp QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    dst-port=80,443 in-interface-list=LAN new-connection-mark=PIE1_Conn passthrough=yes per-connection-classifier=both-addresses:4/0 protocol=tcp
add action=mark-connection chain=prerouting comment="PCC L/Bal Tcp QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    dst-port=80,443 in-interface-list=LAN new-connection-mark=TW_Conn passthrough=yes per-connection-classifier=both-addresses:4/1 protocol=tcp
add action=mark-connection chain=prerouting comment="PCC Load Balancing Http Tr. W1" connection-mark=no-mark connection-state="" disabled=yes dst-address-list=!not_in_internet \
    dst-address-type=!local dst-port=80,443 in-interface-list=LAN new-connection-mark=TW_Conn passthrough=yes per-connection-classifier=both-addresses-and-ports:5/2 protocol=\
    tcp
add action=mark-connection chain=prerouting comment="PCC L/Bal Tcp QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    dst-port=80,443 in-interface-list=LAN new-connection-mark=PIE3_Conn passthrough=yes per-connection-classifier=both-addresses:4/2 protocol=tcp
add action=mark-connection chain=prerouting comment="PCC L/Bal Tcp QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local \
    dst-port=80,443 in-interface-list=LAN new-connection-mark=LTE4_Conn passthrough=yes per-connection-classifier=both-addresses:4/3 protocol=tcp
add action=mark-routing chain=prerouting connection-mark=PIE1_Conn connection-state="" in-interface=ether1-PIE1 new-routing-mark="CIR Wan1" passthrough=yes
add action=mark-routing chain=prerouting connection-mark=TW_Conn connection-state="" in-interface=ether2-TW new-routing-mark="TW Wan2" passthrough=yes
add action=mark-routing chain=prerouting connection-mark=PIE3_Conn connection-state="" in-interface=ether3-PIE3 new-routing-mark="PieFO Wan3" passthrough=yes
add action=mark-routing chain=prerouting connection-mark=LTE4_Conn connection-state="" in-interface=ether4-LTE4 new-routing-mark="Lte Wan4" passthrough=yes
add action=mark-routing chain=output comment=";;;;;;;;;;;Outgoing Marking" connection-mark=PIE1_Conn connection-state="" new-routing-mark="CIR Wan1" passthrough=no
add action=mark-routing chain=output comment=";;;;;;;;;;;Outgoing Marking" connection-mark=TW_Conn connection-state="" new-routing-mark="TW Wan2" passthrough=no
add action=mark-routing chain=output comment=";;;;;;;;;;;Outgoing Marking" connection-mark=PIE3_Conn connection-state="" new-routing-mark="PieFO Wan3" passthrough=no
add action=mark-routing chain=output comment=";;;;;;;;;;;Outgoing Marking" connection-mark=LTE4_Conn connection-state="" new-routing-mark="Lte Wan4" passthrough=no
add action=mark-connection chain=forward connection-mark=no-mark connection-state="" disabled=yes in-interface=ether1-PIE1 new-connection-mark="CIR Wan1-LAN" passthrough=no
add action=mark-connection chain=forward connection-mark=no-mark connection-state="" disabled=yes in-interface=ether2-TW new-connection-mark="TW Wan2-LAN" passthrough=no
add action=mark-connection chain=forward connection-mark=no-mark connection-state="" disabled=yes in-interface=ether4-LTE4 new-connection-mark="Lte Wan4-LAN" passthrough=no
add action=mark-connection chain=forward connection-mark=no-mark connection-state="" disabled=yes in-interface=ether3-PIE3 new-connection-mark="PieFO Wan3-LAN" passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" out-interface-list=WAN
add action=dst-nat chain=dstnat comment="NVR Port forward 9099 Http" dst-address=Public Ip dst-port=80,443 protocol=udp to-addresses=192.168.100.50 to-ports=9099
add action=dst-nat chain=dstnat comment="NVR Port forward 9099 Http" dst-address=Public Ip dst-port=80 port="" protocol=tcp to-addresses=192.168.100.50 to-ports=9099
/ip firewall raw
add action=add-dst-to-address-list address-list="Bluehost IP" address-list-timeout=none-dynamic chain=prerouting content=bluehost.com disabled=yes
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=Public IP pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=Public IP pref-src="" routing-table="CIR Wan1" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=Public Ip pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=Public Ip pref-src="" routing-table="TW Wan2" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=3 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src="" routing-table="PieFO Wan3" scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=4 dst-address=0.0.0.0/0 gateway=192.168.90.1 routing-table="Lte Wan4" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=192.168.90.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/system clock
set time-zone-name=Asia/Karachi
/system note
set show-at-login=no
/system routerboard settings
set auto-upgrade=yes

anav · Mon Oct 30, 2023 3:52 pm

(1) Yes my understanding is that PCC has tradeoffs in terms of what you choose for addresses. One is better for banks and such that do not like different incoming IPs......

From discher.........
[u]Common Problems[/u]
Strange http issues, some images load, other don’t, problems with
some secure sites
Solution: Try using “both addresses” or “source address “ for PCC
classifier. While “both addresses and ports” gives the greatest
chance for randomization and better possibility for even
distribution, it can create these types of issues.

(2) Looks like you want one LAN to share 3 WANS? What is LTE Wan connections purpose??

(3) Why not use the SFP+ port heading to LAN, assuming you are going to a switch first and the switch being of matching capacity (at least one sfp+ port for the trunk to router )

(4) Agree that if you can only use WAN1 for wireguard access (acting as server ), then you need to create a second wireguard interface as a client and have a CHR in the cloud or perhaps a home MT router, that you can connect the RB5009 to as a client and then access the RB5009 through this connection.

Will have a look at he config in slow time.

anav · Mon Oct 30, 2023 4:26 pm

(1) Upgrade firmware to 7.12 stable when released.

(2) Wireguard:
/interface wireguard
add disabled=yes listen-port=13231 mtu=1420 name=WG-Server
add listen-port=14231 mtu=1420 name=WG-Client { future use }

(3) REMOVE bridge address, when doing multiple vlans keep it apples to apples.
No need to change pool but for the rest.
/interface vlan
add disabled=yes interface=bridge-LAN name=vlan99 vlan-id=99 comment=Management ??
/ip dhcp-server
add address-pool=pool-bridge-LAN interface=vlan99 lease-time=10m name=defconf
/ip address
add address=192.168.100.1/24 comment=defconf interface=vlan99 network=192.168.100.0

(4) would get rid of cake for now as I dont know it, and thus to remove all things getting in the way, or unknowns, for troubleshooting purposes, would be best.

(5) Set to something more secure.
/ip neighbor discovery-settings
set discover-interface-list=MGMT

/interface list
add name=MGMT

/interface list memberts
add interface=vlan99 list=MGMT { assuming this is the purpose of this subnet }
add interface=vlanXX list=MGMT { if any other vlans need to access the router }
add interface=WG-Server
add interface=WG-CLient

(6) Purpose of etherports 6,7,8 and what about sfp+ port??

(7) For future consideration
/interface wireguard peers
add allowed-address=XX interface=WG-Server public-key="xx"
add allowed-address=XX interface=WG-Server public-key="xx"
add allowed-address=XXsubnet interface=WG-Client public-key="yy" endpointaddress:port persistent keep-alive=35sec

(8) Purpose of ether7??

(9) Why are there so many vlans that have nothing to do with ether5, or ether7 ..................

(10) this makes no sense to me,
WAN3 --> set [ find default-name=ether3 ] name=ether3-PIE3

So why this???
add address=Public Ip/30 interface=ether3-PIE3 network=X
add address=192.168.10.3/24 interface=ether3-PIE3 network=192.168.10.0

(11) Your bridge ports make no sense to me............. which follows the fact that I have no clue what you are attempting to do on any of the etherports........

(12) Looks like WAN1 and WAN3 are dhcp client assigned,
How is WAN2 assigned??
LTE is a private IP (cgnat or not) but not public.

Are you sure that WAN2 and WAN3 are not pubic??

(13) Why are you using address lists for subnets??
If you have whole subnets that need to be described or grouped for firewall rules, use interface list instead.

(14) Get rid of any firewall rules that you invented. like jump for icmp.
Stick to keeping is simple and efficient especially to get the whole config working and ruling out interfering items.
Also you firewall rules should be fricken organized, all input rules together and then all forward rules together, makes them far easier to read and troubleshoot.
Okay not enough firewall rules and work is needed.

(15) Your allowed list to router is problematic and NOT accurate.
You need all users that require DNS or perhaps NTP services access to the router for those specific services. This can be done should be done by interface list.
ONLY THE ADMIN needs full access to the router.
So a real firewall address list would consist of all the ways the admin is identified on the router or coming in remotely.
Wired, wifi, and wireguard!

(16) Something like this is too vague....... as are many of your rules.......
add action=accept chain=forward src-address-list=VLAN-ips

(17) I see you have port forwarding in your config, do you have any input on how that was supposed to work ????

(18) Are you using any ipv6, if not disable ipV6, if you are using ipv6 then I will stop now as it out of my scope.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++

No point in looking at mangles or routes until one has a better understanding of the LANs and etherports relationships and what the bridge subnet was supposed to be used for, etc.......
For example, which users/subnets/devices should not be involved in PCC.

killa88 · Wed Nov 01, 2023 1:02 pm

Ages ago on the forum while i was learning i came upon posts from you and sob and rextended etc.. i found you guys very in-depth with all the mikroitk workings, but after your reply on my post, Damn you really go all the way and look at the config as a whole
I had written a few paragraphs actually about the scenario but i felt a too long post might make you guys loose interest so i edited it 5 times and removed all the detail. Well here’s the full deal;

3 WAN's + 1 LTE WAN as failover backup
ether1 - WAN1 - 8 Pool Public IP @100Mb received via Unifi powerbeam
ether2 - WAN2 - 1 Public IP via fiber
ether3 - WAN3 - Static IP, trying to get the public from them and put on bridge mode so can reduce latency and NAT issues
ether4 - LTE - Last resort in an event more than 1 or even 2 WANs fail

ether5 - Client side running a dhcp server at 192.168.50.0/24 for around 100 user devices
ether6 LAN Bridge to Office users
ether7 LAN Bridge to Office users
ether8 LAN Bridge to Office users (This port was mostly spare before)
SFP+ - 10g WAS uplinked to a ruckus brocade switch (8 ports 10g SFP+) but was unable to configure VLANs on it, local guys here also refused and asked me to switch over to Cisco but i had them already so i removed it altogether for the time being and connected the client side on 10g

(Ether5 is a backup now for our client side, as we switched our client side to sfp+ that are 180Ft away from us. But due to thunderstorms and many issues that i learnt with fiber, our fiber line to them got cut or damaged 4/5 times a year and the most recent damage was 2 weeks ago)

We have one internally distributed LAN at .100 series (192.168.100.0/24) for 3 buildings around 30-50 users with 100 total user devices
{{ We also had 7 VLAN's, dept. wise segregated and running on Unifi u6 AP's but since our previous mikoritk device got fried due to a power surge, i was unable to reconfigure the vlans within the Unifi OS Console so temporarily disabled them as they are not the problem but the load balancing is}}
the ultimate goal was:

-- to have a CRS305 4S+ to expand 10g capacity of our 5009 and envelop other clients within the 10g ecosystem via BGP (Which i am learning the moment, but won’t be implementing for some time, until i am confident in deployment), {thus the dirt cheap/ISP grade brocade came into play which i am still trying to configure}

The issue is even now that the client side is receiving only 45% of our speed capacity and i am not sure my bandwidth aggregation is working properly, along with my PCC rules, and no forum thread has dealt with routes distance for more than 1 wan with PCC deeply

So 3 WAN's with LTE failover with net average for 400Mbps bandwidth is available and i am distributing it to both Office staff as well as client sides via PCC
The requirement is just that Http/banking and cpanel mail sites dont break and the speed division/distribution works flawlessly
And per point these are the answers below;

(1) Upgrade firmware to 7.12 stable when released.
Trying, but it now says connection timed out when i try updating from winbox GUI which never happened before,
Also counter question, I have a direct bridged non NAT 8 Pool Public IP from my main provider, so when I click on cloud in IP section, why does it say router may be behind NAT

(2) Wireguard:
/interface wireguard
add disabled=yes listen-port=13231 mtu=1420 name=WG-Server
add listen-port=14231 mtu=1420 name=WG-Client { future use }
Ok, i disabled them and removed publics keys before posting thread, will setup again
But once setup, how will i access winbox on wireguard via my remote pc

(3) REMOVE bridge address, when doing multiple vlans keep it apples to apples.
No need to change pool but for the rest.
/interface vlan
add disabled=yes interface=bridge-LAN name=vlan99 vlan-id=99 comment=Management ??
/ip dhcp-server
add address-pool=pool-bridge-LAN interface=vlan99 lease-time=10m name=defconf
/ip address
add address=192.168.100.1/24 comment=defconf interface=vlan99 network=192.168.100.0
VLANs are actually under LAN bridge as Unifi AP's require it (NETVN person on YouTube saw his tutorial for it as if we don’t do that then it is nearly impossible for its discovery tool to adopt the AP’s, and our LAN of 192.168.100.0/24 has nothing to do with VLAN's it a different subnet which was supposedly only for executive members but now due to VLAN unavailability it has now been given to all LAN users temporarily

(4) would get rid of cake for now as I dont know it, and thus to remove all things getting in the way, or unknowns, for troubleshooting purposes, would be best.
Well, i removed the CAKE crap that some config guru guy came and setup but i didn’t know how to completely remove it. The CAKE bandwidth protocols were actually disabled but I deleted them, but I was worried if I changed queue types from the queue settings it might get messed up, It has been bugging me though, so going to search forums and get rid of it completely

(5) Set to something more secure.
/ip neighbor discovery-settings
set discover-interface-list=MGMT

/interface list
add name=MGMT
Changed this as i once loss access to mikrotik and it still haunts me to this day, being unable to see mirkotik in winbox, so will study up on it so i don’t make a mistake again and get locked out, and change it to this

/interface list memberts
add interface=vlan99 list=MGMT { assuming this is the purpose of this subnet }
add interface=vlanXX list=MGMT { if any other vlans need to access the router }
add interface=WG-Server
add interface=WG-CLient
Yup, gonna change it

(11) Your bridge ports make no sense to me............. which follows the fact that I have no clue what you are attempting to do on any of the etherports........
(9) Why are there so many vlans that have nothing to do with ether5, or ether7 ..................
(8) Purpose of ether7??
(6) Purpose of etherports 6,7,8 and what about sfp+ port??
In the paragraph explanation above, in short;
Bridge ports are just for the LAN
VLANS originally for Unifi, wanted this on LAN ether ports but got confused with trunk ports and inter VLAN routing so never dared to go into it (plus would have needed to pair a suitable manageable switch with it right ? and would have needed to program it also then.., if there is a really good site let me know where I can learn it
ethre6-8 LAN Bridge
SFP+ for client, will re enable and fiber is patched again

(7) For future consideration
/interface wireguard peers
add allowed-address=XX interface=WG-Server public-key="xx"
add allowed-address=XX interface=WG-Server public-key="xx"
add allowed-address=XXsubnet interface=WG-Client public-key="yy" endpointaddress:port persistent keep-alive=35sec
Yes, noted

(10) this makes no sense to me,
WAN3 --> set [ find default-name=ether3 ] name=ether3-PIE3
So why this???
add address=Public Ip/30 interface=ether3-PIE3 network=X
add address=192.168.10.3/24 interface=ether3-PIE3 network=192.168.10.0
Well i have public IP available here but behind a NAT (router) so i was trying to access it and added public IP in address, which was reachable, but i think it’s of no use since i can’t utilize it (or don’t know how to)

(12) Looks like WAN1 and WAN3 are dhcp client assigned,
How is WAN2 assigned??
LTE is a private IP (cgnat or not) but not public.
WAN1 - Static assigned
WAN 2 - Static assigned
WAN3 + LTE is on dhcp client
LTE is a carrier portable usb device with external antenna and not public ip

Are you sure that WAN2 and WAN3 are not pubic??
WAN1 + WAN2 = Confirmed Public
WAN3 I have a public IP but i don’t know how to utilize it. Since I am not in bridge mode with the router, the router assigns me a DHCP address, I set it to static form the router’s settings

(13) Why are you using address lists for subnets??
If you have whole subnets that need to be described or grouped for firewall rules, use interface list instead.
LAN user list for office
LAN user list with VLANs for another division
Client side subnet is separate from us
I thought this was not wrong? i will check up on it, does this hurt performance and packets ?

(14) Get rid of any firewall rules that you invented. like jump for icmp.
Stick to keeping is simple and efficient especially to get the whole config working and ruling out interfering items.
Also you firewall rules should be fricken organized, all input rules together and then all forward rules together, makes them far easier to read and troubleshoot.
Okay not enough firewall rules and work is needed.
The jump to icmp rule i directly obtained from mikroitk wiki 'for setting up your 1st firewall' if its not useful enough i will delete it, sure.. and yes I was so fixated on diagnosing the PCC I did’nt review the rules..
I thought like in the presentations if I moved pre-routing or output rules etc, it would change the effect and they need to be in a specific order

(15) Your allowed list to router is problematic and NOT accurate.
You need all users that require DNS or perhaps NTP services access to the router for those specific services. This can be done should be done by interface list.
OK (Got this and then modified it, from mikrotik article), will change
ONLY THE ADMIN needs full access to the router.
So a real firewall address list would consist of all the ways the admin is identified on the router or coming in remotely.
Wired, wifi, and wireguard!
I dont have a fixed ip on my system and if i am at senior managers desk connected with their VLAN i can access mikrotik, and on LAN also. Should i remove access and limit to only 1 IP ?

(16) Something like this is too vague....... as are many of your rules.......
add action=accept chain=forward src-address-list=VLAN-ips
The config buy before me did this, i figured it was to :
Tell the router that Unifi VLAN ip's should be allowed to communicate with each other

(17) I see you have port forwarding in your config, do you have any input on how that was supposed to work ????
PORT FORWARDING is perhaps the biggest thing that was required here as managers remotely view the NVR (Network video recorder) over public IP for instantaneous and lag free streaming of the security channels. Saw some tutorial and setup from YouTube. If it's done wrong i can change it once i know what parameters are wrong. The ports however that were mapped were changed before in the device end to randomize them

(18) Are you using any ipv6, if not disable ipV6, if you are using ipv6 then I will stop now as it out of my scope.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
No, i just copied and implemented them as an extra measure of security from the mikrotik wiki. I don’t use ipv6 though so i will disable and remove

No point in looking at mangles or routes until one has a better understanding of the LANs and etherports relationships and what the bridge subnet was supposed to be used for, etc.......
For example, which users/subnets/devices should not be involved in PCC.
Well mangle and PCC is what i want to check as i think this combined with my routes are something i am messing up

LASTLY
i now have 4 wan's with 4 routes
So
WAN1 Conn Mark - / WAN2 Conn Mark - / WAN3 Conn Mark - / WAN4 Conn Mark -
All these should have distance 1 so they can be simultaneously be active, correct ? But i was told that,
'if these routes fail, (recursive/failover routing), for the packets to have route to go to, you need to add backup routes for the main routing table'
This is what confuses me.. I know lesser distance would mean higher priority but all my marked WAN conn's are actively being used
So for 4 WAN's there would be 8 Routes? And if so then what distances.
If i want gateway priority for a WAN then would selecting dist 1 for it make it the default gateway out for my setup ? And if i set dist. 1 for more than one WAN would i have multiple active gateways?

anav · Wed Nov 01, 2023 11:18 pm

Your config and plans are so much in flux, and what ifs, and things broken at the moment its hard to be coherent.
Whatever the end result it will be complex and thus loathe to dive in at the moment. All doable.

If you have any external originated traffic hitting the router, then the mangle rules must include the following:

add chain=prerouting action=mark-connection connection-mark=no-mark in-interface=WAN1 \
new-routing-mark=incoming-ISP1 passthrough=yes
add chain=prerouting action=mark-connection connection-mark=no-mark in-interface=WAN1 \
new-routing-mark=incoming-ISP2 passthrough=yes
add chain=prerouting action=mark-connection connection-mark=no-mark in-interface=WAN1 \
new-routing-mark=incoming-ISP3 passthrough=yes
add chain=prerouting action=mark-connection connection-mark=no-mark in-interface=WAN1 \
new-routing-mark=incoming-ISP4 passthrough=yes

add chain=output action=mark-routing connection-mark=incoming-ISP1 \
new-routing-mark=useWAN1 passthrough=no
add chain=output action=mark-routing connection-mark=incoming-ISP2 \
new-routing-mark=useWAN2 passthrough=nos
add chain=output action=mark-routing connection-mark=incoming-ISP3 \
new-routing-mark=useWAN3 passthrough=no
add chain=output action=mark-routing connection-mark=incoming-iSP4 \
new-routing-mark=useWAN4 passthrough=nos

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

For PCC MANGLING ENSURE YOU HAVE THE USERS PROPERLY CAPTURED>
( assuming all subnets are VLANned )
a. is it the entire LAN interface list?
b. is it one subnet, is it multiple subnets? ( hint use another interface list to describe them )
c. What exceptions do you have to being PCCd,
- servers? how many?
- users? how many?
-subnets? how many?
d. Are there any subnets that need access to other subnets???

( Concerning APs, the Unifi controller and all the APs should be in the same VLAN if at all possible, otherwise gets tricky and we need to do more tricks )

Until those requirements are well understood, pcc rules will have to wait and they may invoke more than the usual mangle rules and our additional routing rules.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
For IP routes we can talk concepts and formats and standard approaches.

You need four normal
For PCC you need four additional routes.
For PCC failover you need three additional routes to each of the Routes/tables above.

so the question you need to answer is when WAN1 goes down what is the order of transfer.
A-->B->C-->D
1-->2, -->3 -->4
2-->1-->3--->4
3-->1-->2-->4
4-->1-->2-->3

Meaning if wan A fail which WAN B takes up the slack............. and if B goes, which one C ,,,,,,,,,,, and finally only one left is D
For a non-recursive situation it looks llike

add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=main
add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=main
add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP3-gatewayIP routing-table=main
add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP4-gatewayIP routing-table=main
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add dst-address=0.0.0.0/0 gateway=ISP1-gateway routing-table=useWAN1 distance=2
add dst-address=0.0.0.0/0 gateway=ISP2-gateway routing-table=useWAN1 distance=3
add dst-address=0.0.0.0/0 gateway=ISP3-gateway routing-table=useWAN1 distance=4
add dst-address=0.0.0.0/0 gateway=ISP4-gateway routing-table=useWAN1 distance=5

add dst-address=0.0.0.0/0 gateway=ISP2-gateway routing-table=useWAN2 distance=2
add dst-address=0.0.0.0/0 gateway=ISP1-gateway routing-table=useWAN2 distance=3
add dst-address=0.0.0.0/0 gateway=ISP3-gateway routing-table=useWAN2distance=4
add dst-address=0.0.0.0/0 gateway=ISP4-gateway routing-table=useWAN2 distance=5

ETC.. for the last two sets of tables....

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Another monster approach is to distribute the load equitably amongst three tables.......... so that failure of one, gets spread amongst the remaining.

THREE WAN PCC Here the trick is we use the same approach but USE SIX ROUTING TABLES and divide up the PCC amongs them.

add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=main
add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=main
add check-gateway=ping dst-address=0.0.0.0/0 gateway=ISP3-gatewayIP routing-table=main

add dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=useAthenB distance=2
add dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=useAthenB distance=3

add dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=useAthenC distance=2
add dst-address=0.0.0.0/0 gateway=ISp3-gatewayIP routing-table=useAthenC distance=3

add dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=useBthenA distance=2
add dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=useBthenA distance=3

add dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=useBthenC distance=2
add dst-address=0.0.0.0/0 gateway=ISP3-gatewayIP routing-table=useBthenC distance=3

add dst-address=0.0.0.0/0 gateway=ISP3-gatewayIP routing-table=useCthenA distance=2
add dst-address=0.0.0.0/0 gateway=ISP1-gatewayIP routing-table=useCthenA distance=3

add dst-address=0.0.0.0/0 gateway=ISP3-gatewayIP routing-table=useCthenB distance=2
add dst-address=0.0.0.0/0 gateway=ISP2-gatewayIP routing-table=useCthenB distance=3

THE PPC will be like so........
6/0 --> AthenB
6/1 --> AthenC
6/2 --> BthenA
6/3 --> BthenC
6/4 --> CthenA
6/5 --> CthenB

What we are doing is taking the 1/3 approach for each WAN and dividing by 2. Each table/each PCC gets 1/6 of the traffic so when one fails we divy up the 1/3 (any singular WAN gets) by giving the remaining wans 1/6 each.

killa88 · Tue Nov 28, 2023 1:46 pm

Well, its not flux really, but ambitious. Anyhow i tried your approach#1 and did the config, i am now going to change it to your 'monster approach',

And i would want to answer some questions you posted also;
For PCC MANGLING ENSURE YOU HAVE THE USERS PROPERLY CAPTURED>
( assuming all subnets are VLANned )
My subnets are not VLANed
a. is it the entire LAN interface list?
b. is it one subnet, is it multiple subnets? ( hint use another interface list to describe them )
its 3 subnets
why use interface lists as they are already in address lists and 2 different ether ports
c. What exceptions do you have to being PCCd,
- servers? how many?
3 servers
- users? how many?
none
-subnets? how many?
we only dont want HTTP and banking sites to not break, and that i have labeled as BP in address list
d. Are there any subnets that need access to other subnets???
we only want 2 static printers to be able to receive print commands from then users

( Concerning APs, the Unifi controller and all the APs should be in the same VLAN if at all possible, otherwise gets tricky and we need to do more tricks )
Well i followed a person;'s tutorial and unifi plays hardball with mikroitk. That why i had to make a LAN bridge only because unifi devices don't discover if i don't give them a LAN bridge
Until those requirements are well understood, pcc rules will have to wait and they may invoke more than the usual mangle rules and our additional routing rules.

i would still want to understand that why config1 was fine for a while like for 4 days and then;

my primary WAN1 got disconnected for 2 days and Havoc broke loose,
The other 2 WAN's (i removed the LTE for now) do not fully synchronize and handle the user Load and the users experienced extreme degradtion like disconnectivity, i.e. they are not being fully used but my users are constrained and not receiving bandwidth, whats more is we at our LAN side are receiving ;

LAN bridge - 70Mb
While ether5 (our client side on LAN) - 25Mb
whereas our total bandwidth is;
WAN1-110Mb - WAN2-90Mb-WAN3-50Mb = 250Mb

Meaning if wan A fail which WAN B takes up the slack............. and if B goes, which one C ,,,,,,,,,,, and finally only one left is D
For a non-recursive situation it looks llike

But this didnt happen as when wan1 failed my system slowed down and i had to get its fiber fixed the other 2 didnt pick up the slack

The client side also face huge latency and varying speeds even with all 3 WAN's now working
So this does not make sense and i should receive the balance speed as well, And if my WAN1 is on 'check gateway ping' then once its unreachable, in my Conn track table why can i still see new connection being made with the con mark of wan1 even though its now unreachable and shouldn't be used.
(connecting WAN1 on my laptop and testing shows 110-120Mb speed while 3 wan Lad balancing is giving me now 60-70 with very very light usage, with ether 5 receiving ping latency and speed of 2.5mb to 25mb. Where is this going wrong.. because this shouldn't be happening

/interface bridge
add admin-mac=1C:1C:1E:DD:91:03 auto-mac=no comment=defconf name=bridge-LAN
/interface ethernet
set [ find default-name=ether1 ] mac-address=DC:1C:9E:AD:81:12 name=ether1-PIE1
set [ find default-name=ether2 ] mac-address=DC:1C:9E:AD:81:13 mtu=1508 name=ether2-TW
set [ find default-name=ether3 ] mac-address=DC:1C:9E:AD:81:14 name=ether3-PIE3
set [ find default-name=ether4 ] mac-address=DC:1C:9E:AD:81:15 name=ether4-LTE4
set [ find default-name=ether5 ] mac-address=DC:1C:9E:AD:81:16 name=ether5-client-side-as
set [ find default-name=ether6 ] mac-address=DC:1C:9E:AD:81:17
set [ find default-name=ether7 ] mac-address=DC:1C:9E:AD:81:18
set [ find default-name=ether8 ] mac-address=DC:1C:9E:AD:81:19
# disabled due to high temperature
set [ find default-name=sfp-sfpplus1 ] disabled=yes mac-address=SC:SC:SE:SD:SS:1A name="sfp-sfpplus1-CLINET SIDE"
/interface vlan
add disabled=yes interface=bridge-LAN name=vlan11 vlan-id=11
add disabled=yes interface=bridge-LAN name=vlan12 vlan-id=12
add disabled=yes interface=bridge-LAN name=vlan13 vlan-id=13
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=pool-bridge-LAN ranges=192.168.100.2-192.168.100.254
add name=dhcp_pool1v11 ranges=192.168.11.2-192.168.11.254
add name=pool-ether5-as ranges=192.168.50.3-192.168.50.254
/ip dhcp-server
add address-pool=pool-bridge-LAN interface=bridge-LAN lease-time=10m name=defconf
add address-pool=dhcp_pool1v11 disabled=yes interface=vlan11 lease-time=10m name=dhcp1
add address-pool=pool-ether5-as interface=ether5-client-side-as name="dhcp for ether5"
/queue interface
set ether1-PIE1 queue=default
set ether2-TW queue=ethernet-default
set ether3-PIE3 queue=ethernet-default
set ether4-LTE4 queue=ethernet-default
set ether5-client-side-as queue=ethernet-default
/routing table
add disabled=no fib name="CIR Wan1"
add disabled=no fib name="TW Wan2"
add disabled=no fib name="PieFO Wan3"
add disabled=no fib name="Lte Wan4"
/interface bridge port
add bridge=bridge-LAN comment=defconf interface=ether6
add bridge=bridge-LAN comment=defconf interface=ether7
add bridge=bridge-LAN comment=defconf interface=ether8
add bridge=bridge-LAN interface="sfp-sfpplus1-CLINET SIDE" trusted=yes
/ip neighbor discovery-settings
set discover-interface-list=all
/interface list member
add interface=ether2-TW list=WAN
add interface=ether1-PIE1 list=WAN
add interface=ether3-PIE3 list=WAN
add interface=ether4-LTE4 list=WAN
add interface=ether7 list=LAN
add interface=ether6 list=LAN
add interface=ether8 list=LAN
add interface=ether5-client-side-as list=LAN

/ip address
add address=192.168.100.1/24 comment=defconf interface=bridge-LAN network=192.168.100.0
add address=100.90.90.5/29 interface=ether1-PIE1 network=100.90.90.0
add address=100.90.100.5/30 interface=ether2-TW network=100.90.100.0
add address=192.168.11.1/24 disabled=yes interface=vlan11 network=192.168.11.0
add address=192.168.50.1/24 interface=ether5-client-side-as network=192.168.50.0
add address=192.168.10.3/24 interface=ether3-PIE3 network=192.168.10.0
/ip arp
add address=100.90.90.5 interface=ether2-TW mac-address=30:E9:5E:59:55:55
/ip cloud
set ddns-enabled=yes
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1-PIE1
add disabled=yes interface=ether3-PIE3 use-peer-ntp=no
/ip dhcp-server
add address-pool=dhcp_pool4v14 disabled=yes interface=*E lease-time=10m name=dhcp4
add address-pool=dhcp_pool5v15 disabled=yes interface=*F lease-time=10m name=dhcp5
add address-pool=dhcp_pool6v16 disabled=yes interface=*10 lease-time=10m name=dhcp6
add address-pool=dhcp_pool7v17 disabled=yes interface=*11 lease-time=10m name=dhcp7
/ip dhcp-server lease
add address=192.168.100.3 client-id=1:e8:6a:64:81:9d:d3 mac-address=E8:6A:64:81:9D:D3 server=defconf
/ip dhcp-server network
add address=192.168.1.0/24 gateway=192.168.1.1 netmask=24
add address=192.168.11.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.11.1
add address=192.168.50.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.50.1
add address=192.168.100.0/24 comment=defconf dns-server=8.8.8.8,8.8.4.4 gateway=192.168.100.1
/ip dns
set allow-remote-requests=yes cache-size=10000KiB max-udp-packet-size=512 servers=8.8.8.8,1.1.1.1
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall address-list
add address=192.168.100.0/24 list=LAN-ips
add address=192.168.0.0/19 list=LAN-ips
add address=192.168.12.0/24 list=VLAN-ips
add address=192.168.13.0/24 list=VLAN-ips
add address=192.168.14.0/24 list=VLAN-ips
add address=10.10.0.0/29 list=VLAN-ips
add address=192.168.50.0/24 list="AS Client side IP"
add address=67.222.39.62 list="BP Cpanel"
add address=52.29.153.112 list="BP Cpanel"
add address=18.216.86.236 list="BP Cpanel"
add address=52.52.57.238 list="BP Cpanel"
add address=34.233.140.183 list="BP Cpanel"
add address=67.222.39.92 list="BP Cpanel"
add address=192.168.100.254 comment="Richo C307 - Sc822 error wala" list="LAN Printers"
add address=192.168.100.245 comment="Richo C307 - New Interface" list="LAN Printers"
add address=192.168.100.2-192.168.100.254 list=allowed_to_router
add address=192.168.11.2-192.168.11.254 list=allowed_to_router
add address=192.168.14.2-192.168.14.254 list=allowed_to_router
add address=10.10.10.2-10.10.10.254 list=allowed_to_router
add address=0.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=172.16.0.0/12 comment=RFC6890 list=not_in_internet
add address=192.168.0.0/16 comment=RFC6890 list=not_in_internet
add address=10.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=169.254.0.0/16 comment=RFC6890 list=not_in_internet
add address=127.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=224.0.0.0/4 comment=Multicast list=not_in_internet
add address=198.18.0.0/15 comment=RFC6890 list=not_in_internet
add address=192.0.0.0/24 comment=RFC6890 list=not_in_internet
add address=192.0.2.0/24 comment=RFC6890 list=not_in_internet
add address=198.51.100.0/24 comment=RFC6890 list=not_in_internet
add address=203.0.113.0/24 comment=RFC6890 list=not_in_internet
add address=100.64.0.0/10 comment=RFC6890 list=not_in_internet
add address=240.0.0.0/4 comment=RFC6890 list=not_in_internet
add address=192.88.99.0/24 comment="6to4 relay Anycast [RFC 3068]" list=not_in_internet
add address=192.168.100.207 comment="TMC Richo C305" list="LAN Printers"
add address=192.168.100.253 comment="HP Flow M880" list="LAN Printers"
add address=192.168.100.0/24 list=VLAN-ips
add address=192.168.50.0/24 list=LAN-ips
add address=100.90.90.5 list=Connected
add address=100.90.90.5 list=Connected
add address=95.216.102.241 list="BP Cpanel"

/ip firewall filter
add action=jump chain=forward comment="jump to ICMP filters" jump-target=icmp protocol=icmp
add action=accept chain=icmp comment="echo reply" icmp-options=0:0 protocol=icmp
add action=fasttrack-connection chain=forward comment=FastTrack connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="Established, Related" connection-state=established,related
add action=accept chain=input comment="default configuration" connection-state=established,related
add action=accept chain=input src-address-list=allowed_to_router
add action=accept chain=input protocol=icmp
add action=drop chain=input connection-state=invalid
add action=accept chain=forward src-address-list=LAN-ips
add action=accept chain=forward dst-address-list="LAN Printers" dst-port=80,8010,9100-9103 protocol=tcp src-address-list=VLAN-ips
add action=accept chain=forward comment="EXECTIVE PRINTER" dst-address=192.168.100.254
add action=accept chain=forward comment="EXECTIVE PRINTER" dst-address=192.168.100.245
add action=accept chain=forward comment=NVR dst-address=192.168.100.50 src-address-list=LAN_IPS
add action=drop chain=forward comment="Drop invalid" connection-state=invalid in-interface-list=WAN log-prefix=invalid
add action=drop chain=forward comment="Drop incoming from internet which is not public IP" in-interface=ether1-PIE1 log=yes log-prefix=!public src-address-list=not_in_internet
add action=drop chain=forward comment="Drop incoming from internet which is not public IP" in-interface=ether2-TW log=yes log-prefix=!public src-address-list=not_in_internet
add action=drop chain=forward comment="Drop incoming from internet which is not public IP" in-interface=ether3-PIE3 log=yes log-prefix=!public src-address-list=not_in_internet
/ip firewall mangle
add action=accept chain=prerouting connection-state="" dst-address-list=Connected src-address-list=Connected
add action=accept chain=prerouting comment="BHst+Cpanel traffic accept" connection-state="" dst-address-list=LAN-ips src-address-list="BP Cpanel"
add action=accept chain=prerouting comment="BHst+Cpanel traffic accept" connection-state="" dst-address-list="BP Cpanel" src-address-list=LAN-ips
add action=mark-connection chain=prerouting comment="BHst + Cpanel Mail traffic" connection-mark=no-mark connection-state=new dst-address-list="BP Cpanel" new-connection-mark=Mail-TW \
    passthrough=yes src-address-list=LAN-ips
add action=mark-routing chain=prerouting comment="BHst + Cpanel Mail traffic" connection-mark=Mail-TW connection-state="" dst-address-list="BP Cpanel" new-routing-mark="TW Wan2" passthrough=\
    no src-address-list=LAN-ips
add action=mark-connection chain=input comment=";;;;;;;;;;;;Incoming Marking" connection-mark=no-mark connection-state="" in-interface=ether1-PIE1 new-connection-mark=PIE1_Conn passthrough=no
add action=mark-connection chain=input comment=";;;;;;;;;;;;Incoming Marking" connection-mark=no-mark connection-state="" in-interface=ether2-TW new-connection-mark=TW_Conn passthrough=no
add action=mark-connection chain=input comment=";;;;;;;;;;;;Incoming Marking" connection-mark=no-mark connection-state="" in-interface=ether3-PIE3 new-connection-mark=PIE3_Conn passthrough=no
add action=mark-connection chain=prerouting comment="nth for HTTP/s" connection-mark=no-mark connection-state="" disabled=yes dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=PIE1_Conn nth=4,1 passthrough=yes
add action=mark-connection chain=prerouting comment="nth for HTTP/s" connection-mark=no-mark connection-state="" disabled=yes dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=TW_Conn nth=4,2 passthrough=yes
add action=mark-connection chain=prerouting comment="nth for HTTP/s" connection-mark=no-mark connection-state="" disabled=yes dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=PIE3_Conn nth=4,3 passthrough=yes
add action=mark-connection chain=prerouting comment="PCC L/Bal UDP QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 \
    in-interface-list=LAN new-connection-mark=PIE1_Conn passthrough=yes per-connection-classifier=both-addresses:3/0 protocol=udp
add action=mark-connection chain=prerouting comment="PCC L/Bal UDP QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 \
    in-interface-list=LAN new-connection-mark=TW_Conn passthrough=yes per-connection-classifier=both-addresses:3/1 protocol=udp
add action=mark-connection chain=prerouting comment="PCC L/Bal UDP QUIC." connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 \
    in-interface-list=LAN new-connection-mark=PIE3_Conn passthrough=yes per-connection-classifier=both-addresses:3/2 protocol=udp
add action=mark-connection chain=prerouting comment="PCC L/Bal Tcp QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 \
    in-interface-list=LAN new-connection-mark=PIE1_Conn passthrough=yes per-connection-classifier=both-addresses:3/0 protocol=tcp
add action=mark-connection chain=prerouting comment="PCC L/Bal Tcp QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 \
    in-interface-list=LAN new-connection-mark=TW_Conn passthrough=yes per-connection-classifier=both-addresses:3/1 protocol=tcp
add action=mark-connection chain=prerouting comment="PCC L/Bal Tcp QUIC" connection-mark=no-mark connection-state="" dst-address-list=!not_in_internet dst-address-type=!local dst-port=80,443 \
    in-interface-list=LAN new-connection-mark=PIE3_Conn passthrough=yes per-connection-classifier=both-addresses:3/2 protocol=tcp
add action=mark-routing chain=prerouting connection-mark=PIE1_Conn connection-state="" in-interface=ether1-PIE1 new-routing-mark="CIR Wan1" passthrough=yes
add action=mark-routing chain=prerouting connection-mark=TW_Conn connection-state="" in-interface=ether2-TW new-routing-mark="TW Wan2" passthrough=yes
add action=mark-routing chain=prerouting connection-mark=PIE3_Conn connection-state="" in-interface=ether3-PIE3 new-routing-mark="PieFO Wan3" passthrough=yes
add action=mark-routing chain=output comment=";;;;;;;;;;;Outgoing Marking" connection-mark=PIE1_Conn connection-state="" new-routing-mark="CIR Wan1" passthrough=no
add action=mark-routing chain=output comment=";;;;;;;;;;;Outgoing Marking" connection-mark=TW_Conn connection-state="" new-routing-mark="TW Wan2" passthrough=no
add action=mark-routing chain=output comment=";;;;;;;;;;;Outgoing Marking" connection-mark=PIE3_Conn connection-state="" new-routing-mark="PieFO Wan3" passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" out-interface-list=WAN
add action=dst-nat chain=dstnat comment="NVR Port forward 7911 Http" dst-address=100.90.90.5 dst-port=80,443 protocol=udp to-addresses=192.168.100.50 to-ports=7911
add action=dst-nat chain=dstnat comment="NVR Port forward 7911 Http" dst-address=100.90.90.5 dst-port=80 port="" protocol=tcp to-addresses=192.168.100.50 to-ports=7911

/ip route
add check-gateway=ping disabled=no dst-address=0.0.0.0/0 gateway=100.90.90.5 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=100.90.90.5 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=100.90.90.5 pref-src="" routing-table="CIR Wan1" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=4 dst-address=0.0.0.0/0 gateway=100.90.90.5 pref-src="" routing-table="CIR Wan1" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src="" routing-table="CIR Wan1" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=100.90.90.5 pref-src="" routing-table="TW Wan2" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=100.90.90.5 pref-src="" routing-table="TW Wan2" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=4 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src="" routing-table="TW Wan2" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=4 dst-address=0.0.0.0/0 gateway=100.90.90.5 pref-src="" routing-table="PieFO Wan3" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src="" routing-table="PieFO Wan3" scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=100.90.90.5 pref-src="" routing-table="PieFO Wan3" scope=30 suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system identity
[admin@AS C Side - BK] > : system/ reboot
[admin@AS C Side - BK] > : system/ reboo

anav · Tue Nov 28, 2023 2:26 pm

(1) Your doing PCC, drop any queueing of WANS for the moment.

(2) interface list members...... should be modified to the below
/interface list member
add interface=ether2-TW list=WAN
add interface=ether1-PIE1 list=WAN
add interface=ether3-PIE3 list=WAN
add interface=ether4-LTE4 list=WAN
add interface=bridge-LAN list=LAN
add interface=ether5-client-side-as list=LAN

(3) MISSING, the dhcp server settings for the bridge and ether 5???
/ip dhcp-server
???????? ether 5
???????? bridge-LAN
add address-pool=dhcp_pool4v14 disabled=yes interface=*E lease-time=10m name=dhcp4
add address-pool=dhcp_pool5v15 disabled=yes interface=*F lease-time=10m name=dhcp5
add address-pool=dhcp_pool6v16 disabled=yes interface=*10 lease-time=10m name=dhcp6
add address-pool=dhcp_pool7v17 disabled=yes interface=*11 lease-time=10m name=dhcp7

(4) As for the rest your firewall rules, mangles and routes are a complete mess. You are not even doing PCC and are using NTh load balancing.
When you are willing to remove and redo the firewall rules, mangles, and routes let me know.

killa88 · Wed Nov 29, 2023 9:33 am

(1) Your doing PCC, drop any queueing of WANS for the moment.
Ok, but i havent set any que for this, all i did was change Ethernet default, and changed pfifo

(2) interface list members...... should be modified to the below
/interface list member
add interface=ether2-TW list=WAN
add interface=ether1-PIE1 list=WAN
add interface=ether3-PIE3 list=WAN
add interface=ether4-LTE4 list=WAN
add interface=bridge-LAN list=LAN
add interface=ether5-client-side-as list=LAN
Ok have done this

(3) MISSING, the dhcp server settings for the bridge and ether 5???
/ip dhcp-server
???????? ether 5
???????? bridge-LAN
add address-pool=dhcp_pool4v14 disabled=yes interface=*E lease-time=10m name=dhcp4
add address-pool=dhcp_pool5v15 disabled=yes interface=*F lease-time=10m name=dhcp5
add address-pool=dhcp_pool6v16 disabled=yes interface=*10 lease-time=10m name=dhcp6
add address-pool=dhcp_pool7v17 disabled=yes interface=*11 lease-time=10m name=dhcp7
Forgot to remove these but to simplify, i deleted the old VLAN setup completely, just forgot to remove pool14-17 for the VLAn's

(4) As for the rest your firewall rules, mangles and routes are a complete mess. You are not even doing PCC and are using NTh load balancing.
When you are willing to remove and redo the firewall rules, mangles, and routes let me know.
ok sure, i am willing to do a complete overhaul however it seems fit and resolves the issue
originally i did'nt experience any problems after i setup your config for a while and it went good, but it then suddenly got ruined for whatever reason
i understand firewall rules could be a lot and daunting but i have already stated that these are not mine and i directly got them from mikrotik wiki
lastly,
In previous load balancing experiences we could not achieve bandwidth aggregation for multiple links, without breaking banking or https sites
So i found a blog and utilized the settings;
Nth for bandwith aggregation
pcc for dividing streams for tcp and udp traffic (QUIC)
if i can achieve similar results with purely pcc i would most gladly accept that..

let me know and i will completely remove the config and redo all parts

here is the config below with no fiewall all other rules, deleted and re configured;

# model = RB5009UG+S+

/interface bridge
add name=LAN_Bridge
/interface list
add name=LAN
add name=WAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name="LAN_Bridge Pool" ranges=192.168.100.2-192.168.100.254
/ip dhcp-server
add address-pool="LAN_Bridge Pool" interface=LAN_Bridge name=\
    "LAN_Bridge Server"
/routing table
add disabled=no fib name=AthenB
add disabled=no fib name=AthenC
add disabled=no fib name=BthenA
add disabled=no fib name=BthenC
add disabled=no fib name=CthenA
add disabled=no fib name=CthenB
/interface bridge port
add bridge=LAN_Bridge interface=ether8
add bridge=LAN_Bridge interface=ether7
/interface list member
add interface=LAN_Bridge list=LAN
add interface=ether8 list=LAN
add interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=ether3 list=WAN
/ip address
add address=192.168.100.1 interface=LAN_Bridge network=192.168.100.0
add address=100.80.90.00/29 interface=ether1 network=100.80.90.00
add address=100.10.10.10/30 interface=ether2 network=100.10.10.10
add address=192.168.90.1/24 interface=ether3 network=192.168.90.0
add address=192.168.50.1/24 interface=ether5 network=192.168.50.0

/ip dhcp-server network
add address=192.168.100.1/32 dns-server=8.8.8.8 gateway=192.168.100.1 netmask=\
    24
/ip firewall address-list
add address=192.168.100.2-192.168.100.254 list=allowed_to_router
add address=100.80.90.10/29 list="Connected Addresses"
add address=100.10.10.10 list="Connected Addresses"
add address=192.168.10.1 list="Connected Addresses"

/ip firewall filter
add action=fasttrack-connection chain=forward comment=FastTrack \
    connection-state=established,related hw-offload=yes
add action=accept chain=input comment="default configuration" connection-state=\
    established,related
add action=accept chain=input src-address-list=allowed_to_router
add action=accept chain=input protocol=icmp
add action=drop chain=input
add action=accept chain=forward comment="Established, Related" \
    connection-state=established,related
add action=drop chain=forward comment="Drop invalid" connection-state=invalid \
    log=yes log-prefix=invalid
add action=drop chain=forward comment=\
    "Drop incoming packets that are not NAT`ted" connection-nat-state=!dstnat \
    connection-state=new in-interface=ether1 log=yes log-prefix=!NAT
add action=drop chain=forward comment=\
    "Drop incoming from internet which is not public IP" in-interface=ether1 \
    log=yes log-prefix=!public src-address-list=not_in_internet

/ip firewall mangle
add action=accept chain=prerouting src-address-list="Connected Addresses"
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=established,related,new in-interface=ether1 \
    new-connection-mark=ISP1-incoming passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=established,related,new in-interface=ether2 \
    new-connection-mark=ISP2-incoming passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
    connection-state=established,related,new in-interface=ether3 \
    new-connection-mark=ISP3-incoming passthrough=yes
add action=mark-routing chain=output connection-state="" new-routing-mark=\
    AthenB passthrough=yes


/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=103.83.91.49 \
    routing-table=main suppress-hw-offload=no
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=110.39.145.138 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=103.83.91.49 pref-src=\
    "" routing-table=AthenB scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=110.39.145.138 \
    pref-src="" routing-table=AthenB scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=103.83.91.49 pref-src=\
    "" routing-table=AthenC scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src=\
    "" routing-table=AthenC scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=103.83.91.49 pref-src=\
    "" routing-table=BthenA scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=110.39.145.138 \
    pref-src="" routing-table=BthenA scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=110.39.145.138 \
    pref-src="" routing-table=BthenC scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src=\
    "" routing-table=BthenC scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src=\
    "" routing-table=CthenA scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=103.83.91.49 pref-src=\
    "" routing-table=CthenA scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src=\
    "" routing-table=CthenB scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=110.39.145.138 \
    pref-src="" routing-table=CthenB scope=30 suppress-hw-offload=no \
    target-scope=10


/system note
set show-at-login=no
/system routerboard settings
set auto-upgrade=yes
[admin@MikroTik] >

killa88 · Wed Nov 29, 2023 2:36 pm

As per the old approach it was easier, but with your 'monster approach' i dont know how to match the marked connection like isp1-incoming etc etc to all 6 routing tables in mangle (i.e, output chain) and PCC..
Is it possible that i can share you my desk remotely as i have 2 same routers, 1 running the old config, and in the other i have made the new one

anav · Wed Nov 29, 2023 11:04 pm

First step Basic firewall rules.
/ip firewall address-list { use static dhcp leases }
add address=192.168.100.X list=Authorized comment="local admin desktop"
add address=192.168.100.AB list=Authorized comment="local admin laptop"
add address=192.168.100.CD list=Authorized comment="local admin smartphone/ipad"
/ip firewall filter
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related connection-mark=no-mark
add action=accept chain=input comment="defconf: accept established,related,untracked" \
connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="Admin access" src-address-list=Authorized
add action=accept chain=input comment="Allow LAN DNS queries-UDP" \
dst-port=53 in-interface-list=LAN protocol=udp
add action=accept chain=input comment="Allow LAN DNS queries - TCP" \
dst-port=53 in-interface-list=LAN protocol=tcp
add action=drop chain=input comment="DROP ALL ELSE" { put this rule in last }
++++++++++++++++++++++++++++++++++++++++++++++++++++++

add action=fasttrack-connection chain=forward comment=FastTrack \
connection-state=established,related hw-offload=yes connection-mark=no-mark
add action=accept chain=forward comment="Established, Related,Untracked" \
connection-state=established,related,untracked
add action=drop chain=forward comment="Drop invalid" connection-state=invalid \
log=yes log-prefix=invalid
add action=accept chain=forward comment="internet" in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="port forwarding" connection-nat-state=dstnat disabled=yes {enable if required}
acction=drop chain=forward comment="DROP ALL ELSE"

anav · Wed Nov 29, 2023 11:35 pm

Non-PCC MANGLE RULES, ensuring traffic entering a WAN exits the same WAN deals with any traffic to the router itself or to any servers on the LAN.
These will not interfere with any normal traffic either.

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark \
in-interface=ether1 new-connection-mark=ISP1-incoming passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
in-interface=ether2 new-connection-mark=ISP2-incoming passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark \
in-interface=ether3 new-connection-mark=ISP3-incoming passthrough=yes
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
add action=mark-routing chain=output connection-mark=ISP1-incoming \
new-routing-mark=useWAN1 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2-incoming \
new-routing-mark=useWAN2 passthrough=no
add action=mark-routing chain=output connection-mark=ISP3-incoming \
new-routing-mark=useWAN3 passthrough=no

we will need tables for these as well
add fib name=useWAN1
add fib name=useWAN2
add fib name=useWAN3

and routes
/ip route
add dst-address=0.0.0.0/0 gateway=100.100.100.90 table=useWAN1
add dst-address=0.0.0.0/0 gateway=110.110.110.100 table=useWAN2
add dst-address=0.0.0.0/0 gateway=192.168.10.1 table=useWAN3

anav · Wed Nov 29, 2023 11:51 pm

Third Step lets do the PCC MANGLES. ( 6 mark connections and 6 route markings aka tables ) (using src-address ONLY not both)

/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface=LAN-bridge new-connection-mark=WANA-B passthrough=yes per-connection-classifier=src-addresses:6/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface=LAN-bridge new-connection-mark=WANA-C passthrough=yes per-connection-classifier=src-addresses:6/1

add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface=LAN-bridge new-connection-mark=WANB-A passthrough=yes per-connection-classifier=src-addresses:6/2
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface=LAN-bridge new-connection-mark=WANB-C passthrough=yes per-connection-classifier=src-addresses:6/3

add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface=LAN-bridge new-connection-mark=WANC-A passthrough=yes per-connection-classifier=src-addresses:6/4
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface=LAN-bridge new-connection-mark=WANC-B passthrough=yes per-connection-classifier=src-addresses:6/5

======================================================

add action=mark-routing chain=prerouting connection-mark=WAN1-2
new-routing-mark=useAB passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1-3
new-routing-mark=useAC passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN2-1
new-routing-mark=useBA passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2-3
new-routing-mark=useBC passthrough=yes

add action=mark-routing chain=prerouting connection-mark=WAN3-1
new-routing-mark=useCA passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3-2
new-routing-mark=useCB passthrough=yes

Tables required
add fib name=useAB
add fib name=useAC
add fib name=useBA
add fib name=useBC
add fib name=useCA
add fib name=useCB

anav · Thu Nov 30, 2023 12:04 am

Now for the ROUTES. CAUTION: In your actual implementation use GATEWAY IPS, the use of ether1,2 etc.. is for expediency only.

We have the ones we created for the non-pcc mangles as show above......
/ip route
add dst-address=0.0.0.0/0 gateway=100.100.100.90 table=useWAN1
add dst-address=0.0.0.0/0 gateway=110.110.110.100 table=useWAN2
add dst-address=0.0.0.0/0 gateway=192.168.10.1 table=useWAN3

Next we need to identify the MAIN ROUTES for non=PCC traffic.
/ip route
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=100.100.100.90 routing-table=main
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=110.110.110.100 routing-table=main
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=192.168.10.1 routing-table=main

Next the Routes we want traffic to take for PCC.

/ip route
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=100.100.100.90 routing-table=useAB
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=110.110.110.100 routing-table=useAB

add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=100.100.100.90 routing-table=useAC
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=192.168.10.1 routing-table=useAC

add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=110.110.110.100 routing-table=useBA
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=100.100.100.90 routing-table=useBA

add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=110.110.110.100 routing-table=useBC
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=192.168.10.1 routing-table=useBC

add check-gateway=ping distance2 dst-address=0.0.0.0/0 gateway=192.168.10.1 routing-table=useCA
add check-gateway=ping distance4 dst-address=0.0.0.0/0 gateway=100.100.100.90 routing-table=useCA

add check-gateway=ping distance2 dst-address=0.0.0.0/0 gateway=192.168.10.1 routing-table=useCB
add check-gateway=ping distance4 dst-address=0.0.0.0/0 gateway=110.110.110.100 routing-table=useCB

killa88 · Thu Nov 30, 2023 3:07 pm

Thank you so much man !!!!!!!

Ive set it up as below (config posted) and its working so far..

but i'm wondering;
1) If some users speed test will they receive the combined speed test result. If not can we make it so that they are able to achieve that result
(this is just a requirement and i understand that LB is not for this)

2) i saw you have added port forwarding, and i also needed it, so i can now add my required ports in it ? and will i be able to access my server/device via that assigned public IP ? or will i have to define it in 'addess'

3) i have tested a few sites and they thankfully seem to work with very minimal breakage, can i guarantee to my managers that this config won't break https sites and banking apps etc.. or should i play around with PCC classifier like both addresses
3) Can i also include a rule to exempt a subnet or particular site from our PCC and keep it forced on singular wan. I never asked because if a Wan goes down, and my rule was there to keep connections forced on it, then that site would be unreachable right (till that wan is restored) ?

4) our current config has mangle rules which are only applied for 'LAN-Bridge' as i have clients not on this bridge should i then make interface list and add those ether ports to it alongside the bridge ?

5) This is a part of our infrastructure diagram:
I have an internal office LAN and our clients and other buildings LAN separate from us(which we are expanding, no of clients and offices), so to keep this device pure and solely as a bandwidth provider, should i use my smaller 750Gr3 for the office ? or is it fine to have this major device do some office routing work alongwith the LB as well ?
(i was thinking of keeping both devices and networks seperate, because we will obvioulsy configure some vlan's and unifi later)

6) If 1 WAN or even 2 goes down then the remaining one (whichever) will take up the slack ?,
7) will the conn track table automatically clear those unreachable wan marked packets ? will i have to do it manually ?

if i intend to introduce wan4 and wan5 , will i have to then add 2/4 more routing tables along-with it ? because i saw you had changed the distance this time when you posted the scripting compared to earlier. Its scary to think that i add another wan or 2 and mess up the distance and it breaks the system.
(i also didnt specify gateways in routes like in the scripts parts you posted, instead the ether ports and it seems to work fine.. is that a coincidence or was it supposed to be that way)

Lastly if i decide to provide ppppoe session to our clients etc, or have some dedicated users or other queuing requirements, i would need to use mangle. so i will mark connections for those ? if i mark them and make new routing marks(which i would have to, supposedly), they would be exempted from our current pcc config ? or could i encompass them in this as well
meaning would i have the option to decide it or does such flexibility not exist ?
Because only this above and VoiP are my only major targets to impelement

# model = RB5009UG+S+

/interface bridge
add name="LAN bridge"
/interface list
add name=LAN
add name=WAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=192.168.100.2-192.168.100.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface="LAN bridge" name=dhcp1

/routing table
add fib name=useWAN1
add fib name=useWAN2
add fib name=useWAN3
add fib name=useAB
add fib name=useAC
add fib name=useBA
add fib name=useBC
add fib name=useCA
add fib name=useCB

/interface bridge port
add bridge="LAN bridge" interface=ether8
add bridge="LAN bridge" interface=ether7
add bridge="LAN bridge" interface=ether6
add bridge="LAN bridge" interface=ether5

/interface list member
add interface="LAN bridge" list=LAN
add interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=ether3 list=WAN

/ip address
add address=192.168.100.1/24 interface="LAN bridge" network=192.168.100.0
add address=100.100.100.90/29 interface=ether1 network=100.100.100.90
add address=110.110.110.100/30 interface=ether2 network=110.110.110.100
add address=192.168.10.3/24 interface=ether3 network=192.168.10.0

/ip arp
add address=192.168.100.100 interface="LAN bridge" mac-address=99:99:99:99:99:AB

/ip dhcp-server lease
add address=192.168.100.100 client-id=1:70:5a:f:c6:95:ab mac-address=\
    99:99:99:99:99:AB server=dhcp1

/ip dhcp-server network
add address=192.168.100.0/24 gateway=192.168.100.1

/ip dns
set servers=8.8.8.8,8.8.4.4

/ip firewall address-list
add address=192.168.100.100 comment="local admin desktop" list=Authorized
add address=192.168.100.200 comment="local admin laptop" list=Authorized
add address=192.168.100.250 comment="local admin smartphone/ipad" list=Authorized

/ip firewall filter
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-mark=no-mark connection-state=established,related hw-offload=yes
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="Admin access" src-address-list=Authorized
add action=accept chain=input comment="Allow LAN DNS queries-UDP" dst-port=53 \
    in-interface-list=LAN protocol=udp
add action=accept chain=input comment="Allow LAN DNS queries-TCP" dst-port=53 \
    in-interface-list=LAN protocol=tcp
add action=fasttrack-connection chain=forward comment=FastTrack connection-mark=\
    no-mark connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="Established, Related,Untracked" \
    connection-state=established,related,untracked
add action=drop chain=forward comment="Drop invalid" connection-state=invalid log=\
    yes log-prefix=invalid
add action=accept chain=forward comment=internet in-interface-list=LAN \
    out-interface-list=WAN
add action=accept chain=forward comment="port forwarding" connection-nat-state=\
    dstnat
add action=drop chain=forward comment="DROP ALL ELSE"
add action=drop chain=input comment="DROP ALL ELSE"


/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=\
    ether1 new-connection-mark=ISP1-incoming passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=\
    ether2 new-connection-mark=ISP2-incoming passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=\
    ether3 new-connection-mark=ISP3-incoming passthrough=yes

add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN1-2 \
    passthrough=yes per-connection-classifier=src-address:6/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN1-3 \
    passthrough=yes per-connection-classifier=src-address:6/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN2-1 \
    passthrough=yes per-connection-classifier=src-address:6/2
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN2-3 \
    passthrough=yes per-connection-classifier=src-address:6/3
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN3-2 \
    passthrough=yes per-connection-classifier=src-address:6/5
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN3-1 \
    passthrough=yes per-connection-classifier=src-address:6/4
add action=mark-routing chain=prerouting connection-mark=WAN1-2 new-routing-mark=\
    useAB passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1-3 new-routing-mark=\
    useAC passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2-1 new-routing-mark=\
    useBA passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2-3 new-routing-mark=\
    useBC passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3-1 new-routing-mark=\
    useCA passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3-2 new-routing-mark=\
    useCB passthrough=yes

add action=mark-routing chain=output connection-mark=ISP3-incoming \
    new-routing-mark=useWAN3 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2-incoming \
    new-routing-mark=useWAN2 passthrough=no
add action=mark-routing chain=output connection-mark=ISP1-incoming \
    new-routing-mark=useWAN1 passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN

/ip route
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    100.100.100.90 pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=\
    110.110.110.100 pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping disabled=no distance=3 dst-address=0.0.0.0/0 gateway=\
    192.168.10.1 pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add dst-address=0.0.0.0/0 gateway=ether1 routing-table=useWAN1
add dst-address=0.0.0.0/0 gateway=ether2 routing-table=useWAN2
add dst-address=0.0.0.0/0 gateway=ether3 routing-table=useWAN3
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=ether1 \
    routing-table=main
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether2 \
    routing-table=main
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=ether3 \
    routing-table=main
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether1 \
    routing-table=useAB
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether2 \
    routing-table=useAB
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether1 \
    routing-table=useAC
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether3 \
    routing-table=useAC
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether2 \
    routing-table=useBA
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether1 \
    routing-table=useBA
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether2 \
    routing-table=useBC
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether3 \
    routing-table=useBC
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether3 \
    routing-table=useCA
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether1 \
    routing-table=useCA
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether3 \
    routing-table=useCB
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether2 \
    routing-table=useCB

/system clock
set time-zone-name=America/Chicago
/system note
set show-at-login=no
/system routerboard settings
set auto-upgrade=yes
[admin@MikroTik] >

anav · Thu Nov 30, 2023 5:55 pm

(1) By the way, using ether1, ether2, ether3 WORKS in your config as all your WANIPs are static.
My example should reflect the IPs only, so as to not lead others astray. No need to change your config in that regard but I will change my example provided above.

(2) Also I may confuse people by using a numbering scheme and lettering scheme for A then B ( or 1-->2) So, will change my example above to be less confusing but NO need to change your actual config!!

Will attempt to answer your questions next.

anav · Thu Nov 30, 2023 6:48 pm

1) If some users speed test will they receive the combined speed test result. If not can we make it so that they are able to achieve that result
(this is just a requirement and i understand that LB is not for this)

Do not understand the question? Conducting a speed test is not a valid user requirement. Are they getting traffic they need? Is the work pace provided by the throughput available adequate?
If the throughput is not adequate, then increase the throughput of ISPs or add more ISPs.....

2) I saw you have added port forwarding, and i also needed it, so i can now add my required ports in it ? and will i be able to access my server/device via that assigned public IP ? or will i have to define it in 'address'

a. Is this only for external users and via which WANs.........??
b. Will internal users be using the servers and if so, how, direct LANIP or domain name??

3) i have tested a few sites and they thankfully seem to work with very minimal breakage, can i guarantee to my managers that this config won't break https sites and banking apps etc.. or should i play around with PCC classifier like both addresses

Good, that was the intent of using ONLY src-address. The outcome.result is that the equal spreading of sessions amonst all WANs is not exact but the benefit is that session breakage is far less.

3) Can i also include a rule to exempt a subnet or particular site from our PCC and keep it forced on singular wan. I never asked because if a Wan goes down, and my rule was there to keep connections forced on it, then that site would be unreachable right (till that wan is restored) ?

The question is vague. Any user on PCC, has backup to the other WANs through the config, so that should not be a concern.
When you state subnet, do you mean subnet on the router that is currently using PCC? or a non-PCC subnet on the router?
When you say particular site, what do you mean, an external public WANIP?

4) our current config has mangle rules which are only applied for 'LAN-Bridge' as i have clients not on this bridge should i then make interface list and add those ether ports to it alongside the bridge ?

Not quite true, we have mangled for the WANs and for PCC. As for the other LAN components that are not part of the LAN-Bridge and thus not part of PCC, correct they will go out the MAIN table in the order prescribed of WAN availability, first one, then two then WAN3 depending upon availability. This was assumed by the information provided through requirements. In other words, there may be good reasons I am not aware of that they are not PCC. Having LAN servers is one good spot to put these so they are not involved in PCC, less messy.

5) This is a part of our infrastructure diagram:
I have an internal office LAN and our clients and other buildings LAN separate from us(which we are expanding, no of clients and offices), so to keep this device pure and solely as a bandwidth provider, should i use my smaller 750Gr3 for the office ? or is it fine to have this major device do some office routing work alongwith the LB as well ?
(i was thinking of keeping both devices and networks seperate, because we will obvioulsy configure some vlan's and unifi later)

The RB5009 is a robust router overkill for HOME, excellent for SOHO, but not quite enterprise, but if you find there may be limitations then a proper analysis is required.
I am no expert in this field but if MOST of your traffic is inter-office (between users and devices on the LANS) then a switch is a better solution and if most of the traffic is to the internet via the router then a better router solution is preferred.

Router->https://mikrotik.com/product/ccr2116_12 ... estresults especially if your contemplating more WANs...........

6) If 1 WAN or even 2 goes down then the remaining one (whichever) will take up the slack ?,

YES, this is true for both PCC subnets (LAN-bridge) due to our PCC Failover, and other subnets via the main routes.

7) will the conn track table automatically clear those unreachable wan marked packets ? will i have to do it manually ?

I do not understand the question?? Please be more detailed about which packets heading/coming from where, going to????

if i intend to introduce wan4 and wan5 , will i have to then add 2/4 more routing tables along-with it ? because i saw you had changed the distance this time when you posted the scripting compared to earlier. Its scary to think that i add another wan or 2 and mess up the distance and it breaks the system.

Yes, the config will have to be changed, there will be both more mangles and tables, and then subsequently routes.
Mathematically take any number of WANS and that is your fraction example 10 WANs. so each WAN gets 1/10 of the flow. Divide that by the remaining WANS and you know what each WAN in total should receive from the inactive WAN. In the case above, with 3 WANS, 1/3 divided by 2= 1/6. In the case of 10 WANs, 1/10 divided by 9 = 1/90th. When one WAN fails, each other WAN will get 1/90 of the traffic 9/90 = 1/10 !!
To calculate the number of PCCs required, simply take the number of WANs x the remaining WANs 3x2=6 PCCs for 4 WANs=12PCC, 5 WANs=20PCC, 10 WANs=90 PCCs
Each PCC will require 2 routes, so do the math LOL. ( 12 routes / 24 routes / 40 routes / 180 routes

(i also didnt specify gateways in routes like in the scripts parts you posted, instead the ether ports and it seems to work fine.. is that a coincidence or was it supposed to be that way)

Good catch. You can use etherport=X ONLY because its referring to a static WANIP, not pppoe or cable or fibre dynamic IPs.
So technically either is correct but no need to change your setup as its working.
As you can see I changed the examples above to emphasize the point that the gatewayIP must be static/known!!

Lastly if i decide to provide ppppoe session to our clients etc, or have some dedicated users or other queuing requirements, i would need to use mangle. so i will mark connections for those ? if i mark them and make new routing marks(which i would have to, supposedly), they would be exempted from our current pcc config ? or could i encompass them in this as well
meaning would i have the option to decide it or does such flexibility not exist ?
Because only this above and VoiP are my only major targets to impelement

I don't fully understand the service. Do you mean you are going to "PLAY ISP" yourself and run a pppoe server to clients. ( AKA not adding another WAN via pppoe )
They can be dovetailed into the PCC if they are made part of the LAN bridge.
However, if I was a business client I would not want my business traffic mixed up with other folks private traffic.
Suggest a dedicated WAN connection for this type of endeavour would be simpler on the config and better business practice.
It would also allow more control of the traffic without affecting the rest of your traffic... Just my opinion, others may have different ones.

killa88 · Sat Dec 02, 2023 1:54 pm

Regarding speed testing our wan’s combined output is shared among the 3 office building’s directors (so like 3 shareholders), and since I truly have a combined wan output of 250m +, I want to be able to show this.. is this possible without config mess ?

2) I saw you have added port forwarding, and i also needed it, so i can now add my required ports in it ? and will i be able to access my server/device via that assigned public IP ? or will i have to define it in 'address'
a. Is this only for external users and via which WANs.........??
b. Will internal users be using the servers and if so, how, direct LANIP or domain name??

I suppose we will be using wan1/2 for it, and both are strong and fine enough for this, but usually not a lot of users, just those in the field will be using it.
for Internal users (would this require Hairpin nat) via lanIP, and we are going to be deploying ms server 2012 domain name for thin clients also. Would that be an issue ?

3]Good, that was the intent of using ONLY src-address. The outcome.result is that the equal spreading of sessions amongst all WANs is not exact but the benefit is that session breakage is far less.

Well, honestly I’m relieved that i’m not glued to my seat constantly staring at winbox and diagnosing or changing stuff, but my other wan’s are now sitting idle, can I get them to be utilized more, if I do both adr and ports will it cause site breakage or should I test it.. Individual wan’s are there for redundancy but as an important add-on we reduce the speeds of all from provider (thus our costs) so we can distribute load. Currently only wan1 is extremely loaded but others are not being utilized at all

3) Can i also include a rule to exempt a subnet or particular site from our PCC and keep it forced on singular wan. I never asked because if a Wan goes down, and my rule was there to keep connections forced on it, then that site would be unreachable right (till that wan is restored) ?

The question is vague. Any user on PCC, has backup to the other WANs through the config, so that should not be a concern.
When you state subnet, do you mean subnet on the router that is currently using PCC? or a non-PCC subnet on the router?
When you say particular site, what do you mean, an external public WANIP?
Yes external public wanIP, that some users access correct.
by subnet I mean a group of users on a particular subnet like (192.168.5.0/24) ((these are on pcc and lan)). These are our warehouse users. We want to isolate them so we can manage data consumption and limit/time social media usage to 3 hours
The other group of users who need to be separated, access their email from cpanel via hosting companies. I capture them and force them on singular wan’s and keep them there (usually that’s our dedicated wan1 which is pure business internet directly from port of upstream provider. Extremely Low latency as well as secure. So them and banking users all usually on it. Only at the instance of downtime should they be able to use other wan’s

7) will the conn track table automatically clear those unreachable wan marked packets ? will i have to do it manually ?

I do not understand the question?? Please be more detailed about which packets heading/coming from where, going to????
Last time when I applied your approach 1, I saw (when wan3 was down) in the connection tracking table, that users were sending out request (to sites etc, general dialy usage) but they had wan3 connection marks, which was down. So if wan3 was down then no ‘wan3 conn’ connection marks should have been made by mikrotik right? or was that some previous error

Yes, the config will have to be changed, there will be both more mangles and tables, and then subsequently routes.
Mathematically take any number of WANS and that is your fraction example 10 WANs. so each WAN gets 1/10 of the flow. Divide that by the remaining WANS and you know what each WAN in total should receive from the inactive WAN. In the case above, with 3 WANS, 1/3 divided by 2= 1/6. In the case of 10 WANs, 1/10 divided by 9 = 1/90th. When one WAN fails, each other WAN will get 1/90 of the traffic 9/90 = 1/10 !!
To calculate the number of PCCs required, simply take the number of WANs x the remaining WANs 3x2=6 PCCs for 4 WANs=12PCC, 5 WANs=20PCC, 10 WANs=90 PCCs
Each PCC will require 2 routes, so do the math LOL. ( 12 routes / 24 routes / 40 routes / 180 routes

Yeah about that, I get the math, I was just confused about AthenB an BthenC and that terminology, heck I even copied that from the script and didn’t change anything. Infact I also wanted to ask why the distance in all these ‘AthenB’ type marks was set to 2 & 4 respectively and not 3
What I didn’t get purely was: (if wan1 goes down, wan2 picks up and subsequently wan3. So these con marks mean if A goes down then b is active and so on..? These connections getting marked and being processed in mangle rules and where and how they exit got me confused, sorry about that. I am still not clear and going to physically draw like a map and try to better understand them

I don't fully understand the service. Do you mean you are going to "PLAY ISP" yourself and run a pppoe server to clients. ( AKA not adding another WAN via pppoe )
They can be dovetailed into the PCC if they are made part of the LAN bridge.
However, if I was a business client I would not want my business traffic mixed up with other folks private traffic.
Suggest a dedicated WAN connection for this type of endeavor would be simpler on the config and better business practice.
It would also allow more control of the traffic without affecting the rest of your traffic... Just my opinion, others may have different ones.
Well I guess I already am sort of playing that part because 9 warehouses (6 of them small, but still separate units for my mikrotik), 5 retail storefronts, 4 office buildings and est 28 shopkeeper agents means a lot of units.
I have seen that end users are not a lot per unit (because many of these buildings users don’t use mobile devices mostly LAN via systems) but having them all in pppoe sessions would help us identify and DRASTICALLY reduce downtime for anyone diagnosing this in my absence, because fiber breakage is our real issue. That’s why I was trying to achieve something like bgp/igp like spreading them out on an OLT and let contracted fiber workers always keep them up and running and free me from my office room. And all non office connections are specifically targeted to our dedicated and costly wan1 anyway

anav · Sat Dec 02, 2023 2:45 pm

Again, I dont understand the purpose. Showing someone combined WAN output is a useless exercise.
Firstly unless you have a bonded setup with the SAME iSP you cannot ADD the throughput of ISP connection and do a speed test that shows the addition of all of them.
What you do have is a larger total bandwidth to share with all the users and redundancy (assuming they are different ISPs with different sources of internet/

The failover is designed to take marked packets that were destined for WAN2 for example and then if WAN2 is not available, a fraction of that traffic is sent out wan1 and a fraction of that traffic is sent to WAN3jk ( in a three wan scenario that fraction is 1/2) so yes you will see traffic leaving WAN1 or WAN3 with wan 2 connection marks and that is because WAN2 is not available but we allow a path for that traffic to get out to the internet.

anav · Sat Dec 02, 2023 2:50 pm

Generally anything is possible but its best to detail all the requirements PRIOR to setting up a config.
I would stick to source for PCC because of the banking requirements etc....... I would also contemplate using the # of WANS you need to distribute traffic and then perhaps a couple of dedicated WANs for all the other requirements. email, port forwarding etc................. but again without a full scope of the requirements a plan/design shouldnt be speculated on.

killa88 · Tue Dec 05, 2023 1:48 pm

Whoa.. Something is wrong here..!

i have added the screen shot, i edited it in word and then made a pdf since i couldn't past the screen shot here to show you my routes tab;
When i disable a wan to just check if the fail-over was working and the secondary wan kicked in i said fine.., that is until i checked my routes and they were all showing as unreachable
then i checked my internet and it was working fine on LAN and the rest but the routes were unreachable (showing)

So when i thought that the ether ports must be the problem, i typed the original public static ip's in them and they were now showing as reachable .. EXCEPT my internet connectivity was gone !!

i tried replacing them with ether 1,2, etc like they were before but i still couldn't get connectivity back, until i restored the config and it then worked..
have done this more than 3 times and its the same issue

# model = RB5009UG+S+
/interface bridge
add name="LAN bridge"
/interface list
add name=LAN
add name=WAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=192.168.100.2-192.168.100.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface="LAN bridge" name=dhcp1
/routing table
add fib name=useWAN1
add fib name=useWAN2
add fib name=useWAN3
add fib name=useAB
add fib name=useAC
add fib name=useBA
add fib name=useBC
add fib name=useCA
add fib name=useCB
/interface bridge port
add bridge="LAN bridge" interface=ether8
add bridge="LAN bridge" interface=ether7
add bridge="LAN bridge" interface=ether6
add bridge="LAN bridge" interface=ether5
/interface list member
add interface="LAN bridge" list=LAN
add interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=ether3 list=WAN
/ip address
add address=192.168.100.1/24 interface="LAN bridge" network=192.168.100.0
add address=100.90.80.70/29 interface=ether1 network=100.90.80.70
add address=110.100.90.80/30 interface=ether2 network=110.100.90.80
add address=192.168.10.3/24 interface=ether3 network=192.168.10.0
/ip arp
add address=192.168.100.100 interface="LAN bridge" mac-address=AA:AA:AF:A6:A5:AB
/ip dhcp-server lease
add address=192.168.100.100 client-id=1:70:5a:f:c6:95:ab mac-address=A0:AA:AF:A6:A5:AB server=dhcp1
/ip dhcp-server network
add address=192.168.100.0/24 gateway=192.168.100.1
/ip dns
set servers=8.8.8.8,8.8.4.4
/ip firewall address-list
add address=192.168.100.100 comment="local admin desktop" list=Authorized
add address=192.168.100.200 comment="local admin laptop" list=Authorized
add address=192.168.100.250 comment="local admin smartphone/ipad" list=Authorized
/ip firewall filter
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-mark=no-mark connection-state=established,related \
    hw-offload=yes
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="Admin access" src-address-list=Authorized
add action=accept chain=input comment="Allow LAN DNS queries-UDP" dst-port=53 in-interface-list=LAN protocol=udp
add action=accept chain=input comment="Allow LAN DNS queries-TCP" dst-port=53 in-interface-list=LAN protocol=tcp
add action=fasttrack-connection chain=forward comment=FastTrack connection-mark=no-mark connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="Established, Related,Untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="Drop invalid" connection-state=invalid log=yes log-prefix=invalid
add action=accept chain=forward comment=internet in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="port forwarding" connection-nat-state=dstnat
add action=drop chain=forward comment="DROP ALL ELSE"
add action=drop chain=input comment="DROP ALL ELSE"
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether1 new-connection-mark=ISP1-incoming passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2 new-connection-mark=ISP2-incoming passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether3 new-connection-mark=ISP3-incoming passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN1-2 \
    passthrough=yes per-connection-classifier=src-address:6/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN1-3 \
    passthrough=yes per-connection-classifier=src-address:6/1
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN2-1 \
    passthrough=yes per-connection-classifier=src-address:6/2
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN2-3 \
    passthrough=yes per-connection-classifier=src-address:6/3
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN3-2 \
    passthrough=yes per-connection-classifier=src-address:6/5
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface="LAN bridge" new-connection-mark=WAN3-1 \
    passthrough=yes per-connection-classifier=src-address:6/4
add action=mark-routing chain=prerouting connection-mark=WAN1-2 new-routing-mark=useAB passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN1-3 new-routing-mark=useAC passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2-1 new-routing-mark=useBA passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2-3 new-routing-mark=useBC passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3-1 new-routing-mark=useCA passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3-2 new-routing-mark=useCB passthrough=yes
add action=mark-routing chain=output connection-mark=ISP3-incoming new-routing-mark=useWAN3 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2-incoming new-routing-mark=useWAN2 passthrough=no
add action=mark-routing chain=output connection-mark=ISP1-incoming new-routing-mark=useWAN1 passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
/ip route
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=100.80.90.70 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=110.100.90.80 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=3 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add dst-address=0.0.0.0/0 gateway=ether1 routing-table=useWAN1
add dst-address=0.0.0.0/0 gateway=ether2 routing-table=useWAN2
add dst-address=0.0.0.0/0 gateway=ether3 routing-table=useWAN3
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=ether1 routing-table=main
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether2 routing-table=main
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=ether3 routing-table=main
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether1 routing-table=useAB
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether2 routing-table=useAB
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether1 routing-table=useAC
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether3 routing-table=useAC
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether2 routing-table=useBA
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether1 routing-table=useBA
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether2 routing-table=useBC
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether3 routing-table=useBC
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether3 routing-table=useCA
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether1 routing-table=useCA
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether3 routing-table=useCB
add check-gateway=ping distance=4 dst-address=0.0.0.0/0 gateway=ether2 routing-table=useCB
/system note
set show-at-login=no
/system routerboard settings
set auto-upgrade=yes
[admin@MikroTik] >

anav · Tue Dec 05, 2023 2:32 pm

Sure, took me a couple of secs to find the problem.

/ip address
add address=192.168.100.1/24 interface="LAN bridge" network=192.168.100.0
add address=100.90.80.70/29 interface=ether1 network=100.90.80.70
add address=110.100.90.80/30 interface=ether2 network=110.100.90.80
add address=192.168.10.3/24 interface=ether3 network=192.168.10.0

/ip route
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=100.80.90.70 pref-src="" routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=110.100.90.80 pref-src="" routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=3 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src="" routing-table=main scope=30 \
suppress-hw-offload=no target-scope=1

anav · Tue Dec 05, 2023 2:37 pm

Also it would appear you have some duplicates..........
/ip route
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=100.80.90.70 pref-src="" routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=110.100.90.80 pref-src="" routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=3 dst-address=0.0.0.0/0 gateway=192.168.10.1 pref-src="" routing-table=main scope=30 \
suppress-hw-offload=no target-scope=10
add dst-address=0.0.0.0/0 gateway=100.80.90.70 routing-table=useWAN1
add dst-address=0.0.0.0/0 gateway=110.100.90.80 routing-table=useWAN2
add dst-address=0.0.0.0/0 gateway=192.168.10.1 routing-table=useWAN3
add check-gateway=ping distance=1 dst-address=0.0.0.0/0 gateway=ether1 routing-table=main
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=ether2 routing-table=main
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=ether3 routing-table=main

What is the difference between the first set and the last set of highlighted rules get rid of the second set..........

killa88 · Tue Dec 05, 2023 2:39 pm

Wait no.. if you want i can pm you the exact public ip's but these are just the dummy ones i enter, these are not correct..
So is our config a 100% ok ?
if so why (as in the pdf) it shows all our 'AthenB' and so on as unreachable while they carry the name 'ether' port if i disable and reenable them ..

And IF i add the correct public IP's it should work, but i loose internet connectivity that way.. and have to do a complete restore

edited:
i see the duplicates and i will remove them..

edit2: But the routes with 'useWan1 and etc are not supposed to be deleted right ? SO that means 3 rules have been duplicated ? and they were the cause of all this ??

anav · Tue Dec 05, 2023 2:44 pm

PM me the exact config, sure..........
For all ip routes its best to use the correct gateway vice etherX........... ( exception that comes to mind is wireguard )
If nothing else to demonstrate that the routes are meant for Static IPs/gateways, whereas one would need s cripts for dynamic ones.

PCC Mangle and routes reducing Client speeds ? [SOLVED]

PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ? [SOLVED]

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?

Re: PCC Mangle and routes reducing Client speeds ?