load-balancing don't work

Sat Mar 02, 2019 6:27 am

Hello,
I have searched on the forum but nothing seems to work because they are outdated.
The wiki did not yield anything either. That's why my new topic

I want to configure an 8x WAN load balancing and that did not work.
It also did not work when i testing a 2xWAN load-balancing configuration.

It seemed that the distance failed because when the two internet gateways were connected, gateway 2 was always used.
After a while search, I have downgraded the RouterOs to
6.37.4 bug fix on an old Mikrotik router.
Then the load balancing worked properly.
But I want to make this configuration with the newest 6.44.
This does not seem to be possible after the RouterOS 6.37.4

For clarity this code is made with RouterOS 6.44 on an RB760iGS
What am I doing wrong with this configuration?

/interface bridge
add name=bridge-local
/interface ethernet
set [ find default-name=ether1 ] name=WAN1
set [ find default-name=ether2 ] name=WAN2
set [ find default-name=ether3 ] name=ether3_lan
set [ find default-name=ether4 ] name=ether4_lan
set [ find default-name=ether5 ] name=ether5_trunk
/ip pool
add name=dhcp_pool0 ranges=192.168.100.2-192.168.100.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=bridge-local name=dhcp1
/interface bridge port
add bridge=bridge-local interface=ether3_lan
add bridge=bridge-local interface=ether4_lan
/ip address
add address=192.168.100.1/24 interface=bridge-local network=192.168.100.0
/ip dhcp-client
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=WAN1 use-peer-dns=no
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=WAN2 use-peer-dns=no
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=192.168.100.1,1.1.1.1 gateway=192.168.100.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,1.1.2.2
/ip dns static
add address=192.168.100.1 name=ns1.uwbedrijf.net
/ip firewall mangle
add action=accept chain=prerouting dst-address=192.168.1.0/24 in-interface=bridge-local
add action=accept chain=prerouting dst-address=10.98.213.0/24 in-interface=bridge-local
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=WAN2 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=WAN1_conn in-interface=bridge-local new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn in-interface=bridge-local new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn new-routing-mark=to_WAN2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN1
add action=masquerade chain=srcnat out-interface=WAN2
/ip route
add check-gateway=ping distance=1 gateway=192.168.1.1 routing-mark=to_WAN1
add check-gateway=ping distance=2 gateway=10.98.213.20 routing-mark=to_WAN2

Who can help me with a 2x WAN load balancing so that I can finish the 8x WAN configuration

anav · Sat Mar 02, 2019 4:46 pm

Get rid of the static DNS setting.
You dont have eth5 on a bridge port????
Next to peruse mangle rules..............

I'm no expert and thus recommend this MUM presentation.......
https://mum.mikrotik.com/presentations/ ... 749570.pdf

I note it uses in-interface=LAN for its two pre-routing accept rules where you have use in-interface=bridge?? Not sure if that makes a difference?

The final slide says
RECAP
•New connections inbound on each WAN get marked
•Outbound connections with that mark get a routing mark
•LAN traffic heading outbound gets load balanced with the same routing marks
•Routing marks match default gateway routes and head out that specified interface
•Verification of the Load balancing setup

Sat Mar 02, 2019 6:11 pm

Thanks for your reply, ether5 must become a trunk for vlan.
I try this link out and let you know if it works.
Thanks

Thu Mar 07, 2019 4:55 am

This configuration don't work.
I have not a commulatieve bandwidth and ISP2 become the main ISP in place off ISP1.

I want to set ISP1 to the main ISP because this is the fasted ISP.
WHo can help me.

Thu Mar 07, 2019 7:16 am

This is the code from the 8x failover and load-balancing configuration that not working correct.

/interface bridge
add fast-forward=no name=LAN
/interface ethernet
set [ find default-name=ether9 ] name=LAN9
set [ find default-name=ether10 ] name=LAN10
set [ find default-name=ether11 ] name=LAN11
set [ find default-name=ether12 ] name=LAN12
set [ find default-name=ether13 ] name=LAN13
set [ find default-name=ether14 ] name=LAN14
set [ find default-name=ether15 ] name=LAN15
set [ find default-name=ether16 ] name=LAN16
set [ find default-name=ether1 ] name=WAN1
set [ find default-name=ether2 ] name=WAN2
set [ find default-name=ether3 ] name=WAN3
set [ find default-name=ether4 ] name=WAN4
set [ find default-name=ether5 ] name=WAN5
set [ find default-name=ether6 ] name=WAN6
set [ find default-name=ether7 ] name=WAN7
set [ find default-name=ether8 ] name=WAN8
/ip pool
add name=dhcp_pool0 ranges=192.168.100.20-192.168.100.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=LAN name=dhcp1
/interface bridge port
add bridge=LAN interface=LAN9
add bridge=LAN interface=LAN10
add bridge=LAN interface=LAN11
add bridge=LAN interface=LAN12
add bridge=LAN interface=LAN13
add bridge=LAN interface=LAN14
add bridge=LAN interface=LAN15
add bridge=LAN interface=LAN16
/ip address
add address=192.168.100.1/24 interface=LAN network=192.168.100.0
add address=192.168.1.2/24 interface=WAN1 network=192.168.1.0
add address=192.168.2.2/24 interface=WAN1 network=192.168.2.0
add address=192.168.3.2/24 interface=WAN1 network=192.168.3.0
add address=192.168.4.2/24 interface=WAN1 network=192.168.4.0
add address=192.168.5.2/24 interface=WAN1 network=192.168.5.0
add address=192.168.6.2/24 interface=WAN1 network=192.168.6.0
add address=192.168.7.2/24 interface=WAN1 network=192.168.7.0
add address=192.168.8.2/24 interface=WAN1 network=192.168.8.0
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=192.168.100.1,1.1.2.2 gateway=192.168.100.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,1.1.2.2
/ip dns static
add address=192.168.100.1 name=ns1
/ip firewall mangle
add action=mark-connection chain=input in-interface=WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN2 new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN3 new-connection-mark=WAN3_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN4 new-connection-mark=WAN4_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN5 new-connection-mark=WAN5_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN6 new-connection-mark=WAN6_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN7 new-connection-mark=WAN7_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN8 new-connection-mark=WAN8_conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN3_conn new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN4_conn new-routing-mark=to_WAN4 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN5_conn new-routing-mark=to_WAN5 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN6_conn new-routing-mark=to_WAN6 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN7_conn new-routing-mark=to_WAN7 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN8_conn new-routing-mark=to_WAN8 passthrough=yes
add action=accept chain=prerouting dst-address=192.168.1.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.2.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.3.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.4.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.5.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.6.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.7.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.8.0/24 in-interface=LAN
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:8/0
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses:8/1
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN3_conn passthrough=yes per-connection-classifier=both-addresses:8/2
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN4_conn passthrough=yes per-connection-classifier=both-addresses:8/3
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN5_conn passthrough=yes per-connection-classifier=both-addresses:8/4
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN6_conn passthrough=yes per-connection-classifier=both-addresses:8/5
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN7_conn passthrough=yes per-connection-classifier=both-addresses:8/6
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN8_conn passthrough=yes per-connection-classifier=both-addresses:8/7
add action=mark-routing chain=prerouting connection-mark=WAN1_conn in-interface=LAN new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn in-interface=LAN new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3_conn in-interface=LAN new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN4_conn in-interface=LAN new-routing-mark=to_WAN4 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN5_conn in-interface=LAN new-routing-mark=to_WAN5 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN6_conn in-interface=LAN new-routing-mark=to_WAN6 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN7_conn in-interface=LAN new-routing-mark=to_WAN7 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN8_conn in-interface=LAN new-routing-mark=to_WAN8 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN1
add action=masquerade chain=srcnat out-interface=WAN2
add action=masquerade chain=srcnat out-interface=WAN3
add action=masquerade chain=srcnat out-interface=WAN4
add action=masquerade chain=srcnat out-interface=WAN5
add action=masquerade chain=srcnat out-interface=WAN6
add action=masquerade chain=srcnat out-interface=WAN7
add action=masquerade chain=srcnat out-interface=WAN8
/ip route
add check-gateway=ping distance=1 gateway=192.168.1.1 routing-mark=to_WAN1
add check-gateway=ping distance=1 gateway=192.168.2.1 routing-mark=to_WAN2
add check-gateway=ping distance=1 gateway=192.168.3.1 routing-mark=to_WAN3
add check-gateway=ping distance=1 gateway=192.168.4.1 routing-mark=to_WAN4
add check-gateway=ping distance=1 gateway=192.168.5.1 routing-mark=to_WAN5
add check-gateway=ping distance=1 gateway=192.168.6.1 routing-mark=to_WAN6
add check-gateway=ping distance=1 gateway=192.168.7.1 routing-mark=to_WAN7
add check-gateway=ping distance=1 gateway=192.168.8.1 routing-mark=to_WAN8
add check-gateway=ping distance=1 gateway=192.168.1.1
add check-gateway=ping distance=2 gateway=192.168.2.1
add check-gateway=ping distance=3 gateway=192.168.3.1
add check-gateway=ping distance=4 gateway=192.168.4.1
add check-gateway=ping distance=5 gateway=192.168.5.1
add check-gateway=ping distance=6 gateway=192.168.6.1
add check-gateway=ping distance=7 gateway=192.168.7.1
add check-gateway=ping distance=8 gateway=192.168.8.1

See the step by step video a made. In dutch but follow the test configuration.
https://www.youtube.com/watch?v=KO_9UsXb2_w

What do I want to achieve:
Load-balancing starting to WAN1, WAN2 etc.
I want to combine the 8 WAN bandwidths to create one larger bandwidth on 1 LAN infrastructure
A correct failover.

anav · Thu Mar 07, 2019 2:07 pm

from
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=192.168.100.1,1.1.2.2 gateway=192.168.100.1
to
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=192.168.100.1 gateway=192.168.100.1

remove this
/ip dns static
add address=192.168.100.1 name=ns1

Not sure but thinking that for fixed WANIPs.......
these should all be source nat rules not masquerade rules.

/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN1
add action=masquerade chain=srcnat out-interface=WAN2
add action=masquerade chain=srcnat out-interface=WAN3
add action=masquerade chain=srcnat out-interface=WAN4
add action=masquerade chain=srcnat out-interface=WAN5
add action=masquerade chain=srcnat out-interface=WAN6
add action=masquerade chain=srcnat out-interface=WAN7
add action=masquerade chain=srcnat out-interface=WAN8

Will look at mangles and routing later

solar77 · Thu Mar 07, 2019 3:34 pm

I have not used PCC but compare your config with https://wiki.mikrotik.com/wiki/Manual:PCC
the example does not use passthrough=yes,
without understand all your config, i suspect some of the traffic got marked twice because of this. then the result would be only WAN2_conn is left and that's why all traffic is going to WAN2

I'd suggest remove all passthrough=yes and enable log on the mangle rules to do with WAN1 and watch what is going through and what is not.

Let us know.

anav · Thu Mar 07, 2019 5:01 pm

Solar77, that is an interesting observation.
This begs the question for me, as to the purpose and use of passthrough=yes and passthrough=no!
Everytime I see that used I get confused.
I always thought that passthrough=no means there is no more rules for this data flow in this 'bounded ruleset'
I always thought that passthrough=yes means that there are more rules for this data flow in this "bounded rule set"
In other words, if there is no more ACTION of the same type required for this data flow, then ignore the rest of the same ACTIONs that follow??

okay doing some research, the Wiki tells me didly squat, too vague and does not address when passthrough is set to NO??
WIKI: passthrough - if packet is matched by the rule, increase counter and go to next rule (useful for statistics).

Okay one gem I understand LOL
(1) Passthrough = Yes Packet continues down the list of rules after this rule
Passthough = No Packet leaves the Mangle after this rule and is not affected by any of the following mangle rules

Other tidbits seen.
(2) For all "mark-packet" mangle rules it doesn't make sense to set passthrough=yes.
If packet is already marked then simply do not send to next mangle rules, to save CPU resources. ****

(3) Dont market packets, mark connections ....... much more efficient.

Conclusion: Use passthrough=yes if you want any of the rest of the rules in that section to apply to that particular packet flow
So based on the above I would conclude that for the OPs config shown above:

A. first set rules - wan input mark connection rules should have passthrough=no. These packets will enter the router without any further mangle processing job done!
(purpose is to ensure packets coming in one WAN go out the same WAN for routing purposes)
B. second set rules - wan output mark routing rules should have passthrough=no These packets are leaving the router no further mangle processing job done!
(purpose working with A. is to ensure connections coming from wanx are routed back out wanx and so on)
C. third set rules - This is in three parts
i. basic assigning of mangle rules to the available destination addresses (wanIPs) for later pcc processing [ no option here for passthrough so not to worry about ]
ii. assign pcc classifier to traffic from lan heading to wan, to an unspecified WANIP but we will mark the connection with wanx aka the specific connection markings we used in A.
(Im assuming we are saying for destination traffic not headed locally, assign this pcc classifier and mark the connections with those we marked in A - tieing things together slowly) and to let the actual routing be handled by ip routes)
KEY HERE IS THAT THESE PACKETS THEN NEED TO HIT THE NEXT RULE SO passthrough=YES
iii. mark the connections from the previous pcc rule with a routing mark, so that ip routes knows where they need to go.......
PASSTHROUGH=NO (no subsequent rules apply)

Based on this logic, the only place passthrough is required is during the PCC rules is to allow Marked Packets (via connection marking) to be Marked Routed!!!!

SERIOUS

I also want to point out that the OPs configuration is completely 100% in line with the MUM presentation I linked to, from Lebanon 2019. https://mum.mikrotik.com/presentations/ ... 749570.pdf
If that presentation is in error, then I would like to take MT to the court of ridicule and embarrassment for endorsing and allowing a presentation that is wrong and in direct conflict with the WIKI. If my supposition is true, if nothing else, someone at the next MUM should get Normis so drunk that he doesn't realize he is being given a spanking, and it would make a great viral MT video!!
(pssst.. Vienna attendees do it regardless)

Thu Mar 07, 2019 5:34 pm

Hello thanks for the reply's, i don't know what's going wrong but i made a video with the log file. I hope someone can help me correct this error.
Also the people from Mikrotik.

https://youtu.be/6u7dQB_rhAE
I made the video for more clarity.
I've set up a log for every mangle rule.
In this situation I can only use 2 WAN

anav · Thu Mar 07, 2019 5:49 pm

@plisken your config is not wrong but may benefit from some tweaking.
(as you read on later, the first thing to try is change wan interface incoming rules to PRE-ROUTING chain and NOT input chains.
(try this change first before changing any passthrough settings).

Try changing all passthrough rules to NO, except for these ones as they should remain YES.........
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:8/0
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses:8/1
etc...

@solar77
This is the config from the wiki , it is not correct from my understanding. The use of passthrough = yes ( my additions in red ) applies to the sequence of marking connections (Green text) on a packet flow and then that connection needs to be marked with routing marks (Blue text) and thus passthrough=no does NOT apply. After that connection flow has been route marked so to speak it can leave the mangle rules until mangle rules need to be reviewed again by the router.

/ ip firewall mangle
add chain=prerouting dst-address=10.111.0.0/24 action=accept in-interface=LAN
add chain=prerouting dst-address=10.112.0.0/24 action=accept in-interface=LAN
add chain=prerouting in-interface=ISP1 connection-mark=no-mark action=mark-connection \
new-connection-mark=ISP1_conn
add chain=prerouting in-interface=ISP2 connection-mark=no-mark action=mark-connection \
new-connection-mark=ISP2_conn
add chain=prerouting in-interface=LAN connection-mark=no-mark dst-address-type=!local \
per-connection-classifier=both-addresses:2/0 action=mark-connection new-connection-mark=ISP1_conn passthrough=yes
add chain=prerouting in-interface=LAN connection-mark=no-mark dst-address-type=!local \
per-connection-classifier=both-addresses:2/1 action=mark-connection new-connection-mark=ISP2_conn passthrough=yes
add chain=prerouting connection-mark=ISP1_conn in-interface=LAN action=mark-routing \
new-routing-mark=to_ISP1
add chain=prerouting connection-mark=ISP2_conn in-interface=LAN action=mark-routing \
new-routing-mark=to_ISP2
add chain=output connection-mark=ISP1_conn action=mark-routing new-routing-mark=to_ISP1
add chain=output connection-mark=ISP2_conn action=mark-routing new-routing-mark=to_ISP2

solar77 · Thu Mar 07, 2019 6:54 pm

nice presentation, well you making me doubt myself.

however, it's not complete identical. check page 22, the route section,
the presentation has 4 rules, the 1st two has identical distance, both been set o 1, and using routing mark. This is part of the load balancing route.
then the next two rules, I think, is the fail-over rule to the 1s two.
if any of the first two rules fails (gateway ping fails ), then Rule 3 will kick in, if it happens to be the gateway 192.168.0.1 that was failed, then rule 4 will kick in (that's why Rule 4 has distance of 2) , ensure the traffic go to 192.168.0.1

regarding the passthrough=yes.
I think that this does, is
when it is yes,router will go through the rest of the rule to check if any of them applies, in this case, the connection will be checked by the rules below it
when it is NO, the mangle part is done and traffic will not be checked by the rules below it

edit:
@anav, you are probably right on the passthrough thing, once traffic is marked by connection, it still need the routing mark so the only two rules don't need passthrough is the final two
reading more into it, mark connection will have passthrough=yes by default. so the manual does have them even it's not in the command.
also the passthrough=no is only needed once the routing mark is done. as per the last example here:
https://wiki.mikrotik.com/wiki/Manual:I ... all/Mangle

and it also say " whether to let the packet to pass further (like action passthrough) into firewall or not (property only valid some actions)." so yes we have to pass it through.
then It's more the routing side to check. OP if you can chagne the routing as per the pdf presentation and watch the the traffic counter,

NetWorker · Thu Mar 07, 2019 7:03 pm

I admittedly haven't read through all the code because I don't have a lot of time right now. But load balancing is all about mangle marking.

First mark connections that are NOT to be load balanced. You can either do action=accept or use some other mark. For example DNS requests or specific addresses that are present on one WAN but not on the other.

Then add one connection marking rule and one routing mark rule for each WAN. Connection marking can employ different selection criterias but I use NTH. Remember to set the "no-mark" flag in connection mark and passthrough=yes, otherwise it won't reach the routing rule. You can use passthrough=no on the routing mark rules.

Finally add the static routes with the routing mark.

Note that this setup does NOT provide failover. If a WAN goes down, all traffic marked for that WAN will be passed on to the default gateway (if check gateway is enabled on your static routes) or timeout. It's always good practice to set the distance for default gateways in order of preference (i.e. 1 for your preferred WAN, 2 for the second and so on). In case you don't want traffic to go to the default gateway, you'll have to script it.

anav · Thu Mar 07, 2019 7:32 pm

More information: • The “accept” action causes the packet to leave the mangle chain, thereby
not marking it and allowing that traffic to use the main routing table
/ ip firewall mangle
add chain=prerouting dst-address=10.111.0.0/24 action=accept in-interface=LAN
add chain=prerouting dst-address=10.112.0.0/24 action=accept in-interface=LAN

DONT FORGET ON THE MANGLE PCC RULES TO ADD
connection-mark=no-mark before the new-connection-mark=WAN1_conn

I would do this for the initial input chain rule when first assigning new-connection-mark.

anav · Thu Mar 07, 2019 7:46 pm

@solar77
I found Steve Dischers PCC presentation and its very similar to the MUM one where passthrough=yes is used everywhere........ arggg different from the WIKI.

But what I found interesting in that in Steves presentation and the WIKI they both OMITTED the input chain rules that the MUM presentation has (and the OP).
/ip firewall mangle
add action=mark-connection chain=input in-interface=WAN1 new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN2 new-connection-mark=WAN2_conn passthrough=yes
etc...............

(by the way I think there should be a connection-mark=no-connection-mark added to the above rules if these rules are even valid/useful..........)

What is the purpose of these rules, if THEY ARE NOT USED by either the wiki or Steve Discher?????
It has been nagging on my why we care about traffic headed to the router and not across the router????
Can we remove these rules and are they getting in the way????

OKAY Steve has these rules and I see them now on the WIKI, my eyes are glazed over - which make more sense to me PREROUTING RULES.
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=no \
in-interface=ether1 new-connection-mark=WAN1 passthrough=yes

SO what is possibly dead wrong from the MUM2019 presentation is the use of the input chain, vice the use of the prerouting chain..
I still need it explained to me why we need to mark this traffic what is the use case????????

solar77 · Thu Mar 07, 2019 8:57 pm

I had the input chain question this morning, ha ha !
what I did was create a rule on my own router, on prerouting chian, in-interface is WAN, which I thought won't have any traffic. but it did,
as all traffic heading to my WAN ip, it get translated (NATed) into my LAN IP. which then get marked with the correct WAN connection mark so the reply to the internet gets out through the correct gateway.

I don't know why they don't just use the input chain. suspect if NAT is involved it has to go through routing decision and it involves prerouting chain. but will check the packet flow again tomorrow .

enough time been spent on reading up things which is great but I want to do a lap test , somehow, to see for my self.

back to the OP's original issue, highly suspect it's the routing part needs to be changed.

anav · Thu Mar 07, 2019 9:02 pm

I think its the wan incoming interface rules.......
Using steve dischers MUM presentation on same subject.

a. he matches up with the wiki in that the input chain is NOT used, its properly the PRE-ROUTING chain.
b. he uses passthrough=yes up the ying yang.

So for the OP suggest change all wan interface rule chains from input to pre-routing.
keep your current passthrough rules in effect
See what happens.

Thu Mar 07, 2019 10:09 pm

I have try the tutorial from Steve but it's don't work also.
Thank you to all to find a solution.
I will keep you informed

solar77 · Fri Mar 08, 2019 3:33 pm

Right, I've done it myself. Its working and here is what I did:

my set up is not a real multiple WAN but 3 virtual wlan interface ( AP Bridge modem, broadcasting 3 SSID) from Mikrotik hAP, wihch has Internet access.
Each SSID has it's own network: 10.10.1.0/24, 10.10.2.0/24, 10.10.3.0/24
Tested each SSID has internet access.

then use another Mikrotik, with 3 virtual wlan interface (Station), connecting to each of these 3 SSID
I did not use DHCP client but added IP to each interface:

/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
add address=10.10.1.250/24 interface=wlan2 network=10.10.1.0
add address=10.10.2.250/24 interface=wlan3 network=10.10.2.0
add address=10.10.3.250/24 interface=wlan4 network=10.10.3.0

then the mangle part I have:

/ip firewall mangle
add action=accept chain=prerouting disabled=yes dst-address=10.10.1.0/24 in-interface=bridge
add action=accept chain=prerouting disabled=yes dst-address=10.10.2.0/24 in-interface=bridge
add action=accept chain=prerouting disabled=yes dst-address=10.10.3.0/24 in-interface=bridge
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes in-interface=wlan2 new-connection-mark=wan1_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes in-interface=wlan3 new-connection-mark=wan2_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes in-interface=wlan4 log=yes new-connection-mark=wan3_conn passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=wan1_conn passthrough=yes per-connection-classifier=both-addresses:3/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=wan3_conn passthrough=yes per-connection-classifier=both-addresses:3/2
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge new-connection-mark=wan2_conn passthrough=yes per-connection-classifier=both-addresses:3/1
add action=mark-routing chain=prerouting connection-mark=wan1_conn in-interface=bridge new-routing-mark=wan1_mark passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan2_conn in-interface=bridge new-routing-mark=wan2_mark passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan3_conn in-interface=bridge new-routing-mark=wan3_mark passthrough=no
add action=mark-routing chain=output connection-mark=wan1_conn disabled=yes new-routing-mark=wan1_mark passthrough=yes
add action=mark-routing chain=output connection-mark=wan2_conn disabled=yes new-routing-mark=wan2_mark passthrough=yes
add action=mark-routing chain=output connection-mark=wan3_conn disabled=yes new-routing-mark=wan3_mark passthrough=yes

I had all the connection_mark rules on prerouting chain, in-inerface-wan1 etc and the output chain but for testing, I've disabled them for now and it still works so far. I think they are for traffic intitialised from the Internet and also for DNS out of the router so will play with them later on.

NAT part, nothing special here

/ip firewall nat
add action=masquerade chain=srcnat out-interface=wlan2
add action=masquerade chain=srcnat out-interface=wlan3
add action=masquerade chain=srcnat out-interface=wlan4

Route:

/ip route
add distance=1 gateway=10.10.1.1 routing-mark=wan1_mark
add distance=1 gateway=10.10.2.1 routing-mark=wan2_mark
add distance=1 gateway=10.10.3.1 routing-mark=wan3_mark
add check-gateway=ping comment="fail-over route" disabled=yes distance=1 gateway=10.10.1.1
add comment="EMCP test" disabled=yes distance=1 gateway=10.10.3.1,10.10.1.1,10.10.2.1

Nnotes and tests:

Note 1 the disabled rule for ECMP test, I setup ECMP first to make sure all three gateway works and you can indeed load balance between them.
Note 2: I have to enable the last fail-over rule otherwise the router itself won't have Internet access. so it's outgoing traffic is not marked. I will have to test more on that.
Note 3, I used passthrough=no on the mark routing rules. I think there is no need to pass it on anyway.
However, so far all three connection works, all mangle rules counters goes up, I can see connections with each mark in IP - firewall - connections
further tested watching youtube from two computers each grabed one connection.
It's been few hours and traffic status on each wan connection is increasing nicely. so I think it's working for me.
Note 4, if I change the connection marking rules to passthrough=no, I have no internet access. I think this is because the mark routing did not get applied to these connection-marked traffic .
Note 5: It was not working initially because I missed the in-interface=bridge option in the connection marking rules

add action=mark-routing chain=prerouting connection-mark=wan1_conn in-interface=bridge new-routing-mark=wan1_mark passthrough=no

I guess this re-marks the return traffic from wan interface and it just won't reach the originating IP. that's only a guess.

There are few people on the forum struggling with PCC at the moment so hope this helps .

solar77 · Fri Mar 08, 2019 3:42 pm

Note 6, when winbox into the uplink router (Router 1 broadcasting 3 SSID), it used wan2, when I open 2nd winbox session, it still use the same wan2, so the bandwidth doubles,
then I opened 3rd, 4th winbox, they all use the same wan2, with increasing bandwidth flowing.

this shows the both address option in the PCC works, as long as the source IP and dst-ip are the same, it uses the same wan gateway.

anav · Fri Mar 08, 2019 5:14 pm

SOLAR, both the WIKI and Steve Discher show that rule as a prerouting chain BUT IN_INTERFACE=WAN ?????
The MUM2019 presentation shows that as input chain BUT IN_INTERFACE=WAN.

Can i surmize that you are using bridge because you are simply simulating wan input???

I like your logic. ALL rules with marking connections are marked so that something happens to them later in mangling (ie routing) and thus require passthrough=yes, whereas once route marked there is no further mangling involved.

Fri Mar 08, 2019 5:29 pm

Can someone the right configuration please?
With 2 WAN's so i can build the 8x WAN load-balancing.

I hope that this can.

I try the tutorial from Steve again and i have no solution with this tutorial.

WeWiNet · Fri Mar 08, 2019 5:57 pm

Hi Solar77,

Good Job!

Unfortunate I see your post only now, I went through the same nightmare:
Load balancing (LTE and DSL) with PCC, then Queueing subnets and remote GW failover (not only next GW)

Indeed all Wiki posts are outdated/not good or do not use one or the other.
Or this strange example of using fake 'hops" for remote GW test.

I had do dig into the original thread from ~2010 from Chupaka to find the right solution
for load balancing with failover AND with remote connection check (once you read it becomes simple to understand).
I can't find that thread right now unfortunately.

As I did not have time to read your complete post below comments/advice might already been addressed:
But in this context they might help you or someone else.

- PCC with "both address" mode works well as long as you use "RP Filtering" loose (took me 2 weeks to find that)
Keep in mind that connections and data rates are very unbalanced, so the packet traffic amount is not really equally distributed.

- I would suggest you make the "failover" route distance = 2 and enable it, then it can indeed take over if one of the other fails.

- Even better directly add the remote ping function for the routes (right now you only ping next hop, if something fails after that device
you are stuck as the router believes the connection is up (next hop can be pinged). All this is from Chupaka's solution
and applied here to two WAN lines:

use 8.8.8.8 for testing that GW1 is really connected to internet
use 8.8.4.4. for testing GW2.

/ip route
add check-gateway=ping comment="ping-tested GW-1" distance=1 gateway=8.8.8.8 routing-mark=to_WAN1 
add check-gateway=ping comment="ping-tested GW-2" distance=1 gateway=8.8.4.4 routing-mark=to_WAN2 
add check-gateway=ping comment="Backup onto GW1 all GW2 connections" distance=2 gateway=8.8.8.8
add check-gateway=ping comment="Backup onto GW2 all GW1 connections" distance=3 gateway=8.8.4.4
add comment="route to reach ping-test-GW-2" distance=1 dst-address=8.8.4.4/32 gateway=YOUR_WAN_IP_GW1 scope=10 
add comment="route to reach ping-test-GW-1" distance=1 dst-address=8.8.8.8/32 gateway=YOUR_WAN_IP_GW2 scope=10

This can be extended to more WAN, and also use more "external test points" as just Google DNS servers.
PS: If you use DHCP client on WAN , then disable "add route"

Hopefully this helps a bit.

solar77 · Fri Mar 08, 2019 6:00 pm

@plisken

my config can work at 2 wan as well, just remove anything to do with wan3 and change PCC to 2/0 and 2/1
the only reason I did not do a whole router export is I've used it to test vlan staff before and there are many unrelated code left.

what I did was step by step:

get each of wan connection working, make sure it goes to the internet

then I did test with ECMP, just add all 3 gateway IP into the route, (this route is left in my config, been disabled) . it should use them all,

once these are working, I applied rest mangle rules and added routes with marks but it did not work.
so disabled them,
I used connection-mark and routing mark to create rule for only one LAN IP 192.168.88.100 to use wan2, as an example. doing this allowed me to find the mistake where I missed in-interface.

solar77 · Fri Mar 08, 2019 6:06 pm

SOLAR, both the WIKI and Steve Discher show that rule as a prerouting chain BUT IN_INTERFACE=WAN ?????
The MUM2019 presentation shows that as input chain BUT IN_INTERFACE=WAN.

Can i surmize that you are using bridge because you are simply simulating wan input???

I like your logic. ALL rules with marking connections are marked so that something happens to them later in mangling (ie routing) and thus require passthrough=yes, whereas once route marked there is no further mangling involved.

I think the prerouting chain and in-interface=wan is to do with traffic initialised from the Internet so if you have dst-nat rule of somesort this will ensure traffic goes back to the correct gateway, I don't use them so I've disabled these:

add action=mark-connection chain=prerouting connection-mark=no-mark disabled=yes in-interface=wlan2 new-connection-mark=wan1_conn passthrough=yes

solar77 · Fri Mar 08, 2019 6:15 pm

@WeWiNet

Thanks for tips. This is only a trial for me to see how it would work out. If I was using it, yes fail-over will have to be there.

. I am still not clear where the DNS traffic from the router itself goes , I think this is what this rule is for but not tested and I am using 8.8.8.8 on local DHCP setting.

add action=mark-routing chain=output connection-mark=wan1_conn disabled=yes new-routing-mark=wan1_mark passthrough=yes

also, I'd expect ping 8.8.8.8 from the router itself to use the output chain and get marked but without the

add check-gateway=ping comment="fail-over route" distance=1 gateway=10.10.1.1

I am not able to ping 8.8.8.8 from the router, it just say no route to host. so clearly it's not getting marked and I don't understand this.

I will try to use this tomorrow and see if it gets marked

add action=mark-routing chain=output new-routing-mark=wan1_mark passthrough=yes

anav · Fri Mar 08, 2019 11:12 pm

Yes why does output chain rule with mark route rule need passthrough=yes....
Arggg I am bamboozled.

Sat Mar 09, 2019 5:46 am

@plisken

my config can work at 2 wan as well, just remove anything to do with wan3 and change PCC to 2/0 and 2/1
the only reason I did not do a whole router export is I've used it to test vlan staff before and there are many unrelated code left.

what I did was step by step:

get each of wan connection working, make sure it goes to the internet

then I did test with ECMP, just add all 3 gateway IP into the route, (this route is left in my config, been disabled) . it should use them all,

once these are working, I applied rest mangle rules and added routes with marks but it did not work.
so disabled them,
I used connection-mark and routing mark to create rule for only one LAN IP 192.168.88.100 to use wan2, as an example. doing this allowed me to find the mistake where I missed in-interface.

Thank you verry much Solar77, i try this out and let you know of its working.

WeWiNet · Sat Mar 09, 2019 9:50 am

Anav,

Indeed in my setup output chain also has passthrough enabled:
On one WAN connection (DSL) I do further queuing in mangle to prioritise traffic based on IP subnets.
On the other WAN as it is a bigger pipe I do nothing, but it does not harm neither
as everything I do after output chain rule does not alter the routing mark afterwards, I only using connection marks.
I could try put output rules at the end, then no passthrough, but as everything works I rather don't touch...

add action=mark-routing chain=output comment="DSL routing to interface" connection-mark=WAN1_conn new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output comment="LTE routing to interface" connection-mark=WAN2_conn new-routing-mark=to_WAN2 passthrough=yes
a

Both output rules are hit in my setup...not often but they are as this is traffic from the router itself I believe to the GW (DHCP, broadcast?).

Sun Mar 10, 2019 6:37 am

Hello, i have try another tutorial and let you see with a video.
Load balancing don't work, bandwidth is not cummulative i want create more bandwidth with 2 WAN"s
Why respond Mikrotik not.
https://youtu.be/sQOx1dAvta4

With RouterOS 37.4 i had now problems

WeWiNet · Wed Mar 13, 2019 3:30 pm

plisken,

not sure I understand your post:

Why respond Mikrotik not.

I am using the PCC setup as shown in this wiki:
https://wiki.mikrotik.com/wiki/Manual:PCC

this works fine, provides load balancing and has basic failover (only for next GW, but is a good starting point).
Once this works, you can add remote GW ping but this is just changing the routing rules, and will not do
anything to the load balancing (see my previous post on routes to add with recursive look up).

Make sure you use "loose" mode for RP filter if PCC / both address is selected or use source address only.
And yes, do not use "ADD DEFAULT ROUTE" in your DHCP client.

Sob · Sat Mar 16, 2019 2:25 pm

Load balancing don't work, bandwidth is not cummulative i want create more bandwidth with 2 WAN"s

Since I don't see that question asked, how exactly you test it?

If you have per-connection-classifier=both-addresses:x/y, it takes source and destination addresses and uses some hash to make them into number, which is then divided by x, gives you remainder y, and that's how you split the traffic. The rest is a matter of statistics. Communication between single local device a single remote device will always use only one WAN. If the same local device starts to communicate with another remote device, it might use different WAN, but there's still a change that the same one will be chosen. The more devices, the better. Have a thousand of local devices communicating each with thousand remote devices, and all WANs will be used equally. The less variability, the worse the distribution will be.

anav · Sat Mar 16, 2019 3:37 pm

I had another look at your original config.
As you can see in RED, I only detected one potential error (to be removed).
As you can see in BLUE, I only added one change that is Steves presentation and is probably better programming.
Other than that I dont see anything wrong with the config? (its not complete without seeing firewall rules)

/interface bridge
add fast-forward=no name=LAN
/interface ethernet
set [ find default-name=ether9 ] name=LAN9
set [ find default-name=ether10 ] name=LAN10
set [ find default-name=ether11 ] name=LAN11
set [ find default-name=ether12 ] name=LAN12
set [ find default-name=ether13 ] name=LAN13
set [ find default-name=ether14 ] name=LAN14
set [ find default-name=ether15 ] name=LAN15
set [ find default-name=ether16 ] name=LAN16
set [ find default-name=ether1 ] name=WAN1
set [ find default-name=ether2 ] name=WAN2
set [ find default-name=ether3 ] name=WAN3
set [ find default-name=ether4 ] name=WAN4
set [ find default-name=ether5 ] name=WAN5
set [ find default-name=ether6 ] name=WAN6
set [ find default-name=ether7 ] name=WAN7
set [ find default-name=ether8 ] name=WAN8
/ip pool
add name=dhcp_pool0 ranges=192.168.100.20-192.168.100.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=LAN name=dhcp1
/interface bridge port
add bridge=LAN interface=LAN9
add bridge=LAN interface=LAN10
add bridge=LAN interface=LAN11
add bridge=LAN interface=LAN12
add bridge=LAN interface=LAN13
add bridge=LAN interface=LAN14
add bridge=LAN interface=LAN15
add bridge=LAN interface=LAN16
/ip address
add address=192.168.100.1/24 interface=LAN network=192.168.100.0
add address=192.168.1.2/24 interface=WAN1 network=192.168.1.0
add address=192.168.2.2/24 interface=WAN1 network=192.168.2.0
add address=192.168.3.2/24 interface=WAN1 network=192.168.3.0
add address=192.168.4.2/24 interface=WAN1 network=192.168.4.0
add address=192.168.5.2/24 interface=WAN1 network=192.168.5.0
add address=192.168.6.2/24 interface=WAN1 network=192.168.6.0
add address=192.168.7.2/24 interface=WAN1 network=192.168.7.0
add address=192.168.8.2/24 interface=WAN1 network=192.168.8.0
/ip dhcp-server network
add address=192.168.100.0/24 dns-server=192.168.100.1,1.1.2.2 gateway=192.168.100.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,1.1.2.2
/ip dns static
add address=192.168.100.1 name=ns1
/ip firewall mangle
add action=mark-connection chain=input in-interface=WAN1 connection-mark=no mark new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN2 connection-mark=no mark new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN3 connection-mark=no mark new-connection-mark=WAN3_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN4 connection-mark=no mark new-connection-mark=WAN4_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN5 connection-mark=no mark new-connection-mark=WAN5_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN6 connection-mark=no mark new-connection-mark=WAN6_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN7 connection-mark=no mark new-connection-mark=WAN7_conn passthrough=yes
add action=mark-connection chain=input in-interface=WAN8 connection-mark=no mark new-connection-mark=WAN8_conn passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1_conn new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN3_conn new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN4_conn new-routing-mark=to_WAN4 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN5_conn new-routing-mark=to_WAN5 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN6_conn new-routing-mark=to_WAN6 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN7_conn new-routing-mark=to_WAN7 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN8_conn new-routing-mark=to_WAN8 passthrough=yes
add action=accept chain=prerouting dst-address=192.168.1.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.2.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.3.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.4.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.5.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.6.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.7.0/24 in-interface=LAN
add action=accept chain=prerouting dst-address=192.168.8.0/24 in-interface=LAN
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN connection mark=no-mark new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=both-addresses:8/0
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN connection mark=no-marknew-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=both-addresses:8/1
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN connection mark=no-marknew-connection-mark=WAN3_conn passthrough=yes per-connection-classifier=both-addresses:8/2
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN connection mark=no-marknew-connection-mark=WAN4_conn passthrough=yes per-connection-classifier=both-addresses:8/3
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN connection mark=no-marknew-connection-mark=WAN5_conn passthrough=yes per-connection-classifier=both-addresses:8/4
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN connection mark=no-marknew-connection-mark=WAN6_conn passthrough=yes per-connection-classifier=both-addresses:8/5
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN connection mark=no-marknew-connection-mark=WAN7_conn passthrough=yes per-connection-classifier=both-addresses:8/6
add action=mark-connection chain=prerouting dst-address-type=!local in-interface=LAN connection mark=no-marknew-connection-mark=WAN8_conn passthrough=yes per-connection-classifier=both-addresses:8/7
add action=mark-routing chain=prerouting connection-mark=WAN1_conn in-interface=LAN new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn in-interface=LAN new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3_conn in-interface=LAN new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN4_conn in-interface=LAN new-routing-mark=to_WAN4 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN5_conn in-interface=LAN new-routing-mark=to_WAN5 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN6_conn in-interface=LAN new-routing-mark=to_WAN6 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN7_conn in-interface=LAN new-routing-mark=to_WAN7 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN8_conn in-interface=LAN new-routing-mark=to_WAN8 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=WAN1
add action=masquerade chain=srcnat out-interface=WAN2
add action=masquerade chain=srcnat out-interface=WAN3
add action=masquerade chain=srcnat out-interface=WAN4
add action=masquerade chain=srcnat out-interface=WAN5
add action=masquerade chain=srcnat out-interface=WAN6
add action=masquerade chain=srcnat out-interface=WAN7
add action=masquerade chain=srcnat out-interface=WAN8
/ip route
add check-gateway=ping distance=1 gateway=192.168.1.1 routing-mark=to_WAN1
add check-gateway=ping distance=1 gateway=192.168.2.1 routing-mark=to_WAN2
add check-gateway=ping distance=1 gateway=192.168.3.1 routing-mark=to_WAN3
add check-gateway=ping distance=1 gateway=192.168.4.1 routing-mark=to_WAN4
add check-gateway=ping distance=1 gateway=192.168.5.1 routing-mark=to_WAN5
add check-gateway=ping distance=1 gateway=192.168.6.1 routing-mark=to_WAN6
add check-gateway=ping distance=1 gateway=192.168.7.1 routing-mark=to_WAN7
add check-gateway=ping distance=1 gateway=192.168.8.1 routing-mark=to_WAN8
add check-gateway=ping distance=1 gateway=192.168.1.1
add check-gateway=ping distance=2 gateway=192.168.2.1
add check-gateway=ping distance=3 gateway=192.168.3.1
add check-gateway=ping distance=4 gateway=192.168.4.1
add check-gateway=ping distance=5 gateway=192.168.5.1
add check-gateway=ping distance=6 gateway=192.168.6.1
add check-gateway=ping distance=7 gateway=192.168.7.1
add check-gateway=ping distance=8 gateway=192.168.8.1

sebastia · Sat Mar 16, 2019 3:48 pm

Looking good:
* some pass-through's are not needed, but won't hurt
* with this config FastTrack MAY NOT be used
(and use code tags next time...)

Sat Mar 16, 2019 5:10 pm

Hello, I tried all types and all tutorials, but still found no solution.
I already send 3 supout.rif files and there is still no solution.
In the past, that failover worked well with the 6.37 bugfix.
Since I now have newer Mikrotik routers with I can no longer use this 6.37 and this RouterOS is outdated.
This must be a bug, I know how to configure a failover but the failover fails.
I am sure there is a bug in the Mikrotik router Load balancing and failover does not work.
Mikrotik's answer is also not clear and when I ask for an example. No reaction. If anything has changed in this configuration, I have also tried the 2019 tutorial. But its not working.

anav · Sat Mar 16, 2019 5:18 pm

Looking good:
* some pass-through's are not needed, but won't hurt
* with this config FastTrack MAY NOT be used
(and use code tags next time...)

code tags do not allow me to highlight with colour and besides I was hoping that you didnt have the attention span to read it that way. ;-P

sebastia · Sat Mar 16, 2019 7:19 pm

scrolling is tiresome activity, agreed

anav · Sat Mar 16, 2019 7:46 pm

scrolling is tiresome activity, agreed

I'm going to send you some rubber bands, to help you strengthen up your scrolling finger muscles.

Sun Mar 17, 2019 12:16 am

@anav Do you mean me that you sent me rubber tires?

anav · Sun Mar 17, 2019 2:33 am

No unless you need major body work LOL, the comment was meant for Sebastia.

Sun Mar 17, 2019 6:52 am

@anav I would say otherwise, this would cost money to send the rubber tires to my house

Now seriously, the load balancing and failover configuration has always worked. I have a person who tested it and reduced the speed instead of increasing it. And the problem is that the 8xWAN failover chooses just one WAN interface.
All other interfaces are not chosen for load balancing.
It seems like a routing error. I have already tried all the tutorials.
But RouterOS larger than the 6.37 no longer works.
I think there is a bug in the RouterOS that does not allow load balancing.

Which tutorials have I already tried:
https://mum.mikrotik.com/presentations/US12/tomas.pdf
https://mum.mikrotik.com/presentations/US12/steve.pdf
https://systemzone.net/mikrotik-dual-wa ... using-pcc/
https://www.manitonetworks.com/mikrotik ... -balancing
https://wiki.mikrotik.com/wiki/Manual:PCC
https://www.askitmen.com/network/mikrot ... -mikrotik/

Can it be determined how much throughput will go through WAN1 and then send the rest through WAN2?

Greatings

https://mum.mikrotik.com/presentations/US12/tomas.pdf

anav · Sun Mar 17, 2019 3:20 pm

Why your config only chooses one WAN is very telling.
Are you sure your distance rules for the mangled routes are all the same???

Sun Mar 17, 2019 3:37 pm

Yes just the per connection classifier is different:

8:0
8:1
8:2
8:3
8:4
8:5
8:6
8:7

anav · Sun Mar 17, 2019 4:15 pm

Yes just the per connection classifier is different:

8:0
8:1
8:2
8:3
8:4
8:5
8:6
8:7

No thats part of the mangle rules.
I mean the route rules
where you use the route-mark
all 8 rules should have the same distance=1 default setting.

Sob · Sun Mar 17, 2019 4:49 pm

Did you check what exacly doesn't work? PCC rules that mark connections, do all of them get roughly equal number of hits? If you add another rule that forces specific traffic to selected WAN, does that work? Things like that.

sindy · Sun Mar 17, 2019 5:11 pm

From what I can see in the configuration in the OP, it should work, so there must be some issue with at least one of (per-connection-classifier, connection-mark, routing-mark). I'm also not sure whether the check-gateway=ping setting of /ip route doesn't cause some trouble if it eventually started to behave differently in 6.44.1 than before.

Two popular misunderstandings can be found in that configuration (which are not the reason why it does not work, they just may surprise you in another context):

address-type=local does not match any other addresses than Mikrotik's own ones. There is currently no firewall rule match condition available which would match all addresses from all connected subnets automatically, without need to provide the particular prefixes of these subnets.
there is no point in setting a different distance to routes with a different routing-mark. The distance parameter is only taken into account when choosing a route from a set whose dst-address and routing-mark exactly match and their gateway interfaces are up.

As for how to find out why it does not work:

first, check what /ip route print shows. If one (or both) the routes marked with a routing-mark is not marked as Active, the check-gateway=ping is most likely the culprit.
next, if the above doesn't help, log the activity of the PCC rules by adding two rules action=log log-prefix="connection-mark WANx_conn has been assigned" connection-state=new connection-mark=WANx_conn after the rules with action=mark-connection (where x is 1 in one rule and 2 in the other one), and log the activity of the connection-mark to routing-mark translation rules by two other rules saying action=log log-prefix="routing-mark to_WANx has been assigned" connection-state=new routing-mark=to_WANx" after them in chain=prerouting.

As @Sebastia has said, a rule with action=fasttrack-connection must not exist in /ip firewall filter chain=forward as fasttracking is incompatible with

anav · Sun Mar 17, 2019 5:32 pm

Thanks Sindy for your input. I have been at this thread for awhile and fresh eyes and smarter brains are required. Kisses hugs, whatever you prefer LOL.

Sun Mar 17, 2019 7:17 pm

Thanks Sindy Fasttrack has been removed because they bypass the mangle rules, connection tracking etc. I know that.
But I look at your advice and let you know how it went.
Thanks in advance for your response

Thu Mar 28, 2019 2:03 pm

Hello, here a'm back with news.

Can you explain that load balancing like this works with RouterOS 6.21.1 and not with the 44.1? On RB2011UiAS 2HnD-in
I have been searching for a solution for almost a month.
But no result. This must be a bug in RouterOS.
Can you help me solve this with a working configuration. Or RouterOS needs to be updated.
I think Mikrotik will have to work on that.
I'm not the only one with these complaints.

This is the export with 44.1

/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=192.168.12.50-192.168.12.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=ether5 name=dhcp1
/ip address
add address=192.168.12.1/24 interface=ether5 network=192.168.12.0
/ip dhcp-client
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=ether2
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=ether3
/ip dhcp-server network
add address=192.168.12.0/24 dns-server=192.168.12.1,8.8.8.8 gateway=192.168.12.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.12.1 name=ns1.testnet
/ip firewall mangle
add action=accept chain=prerouting dst-address=192.168.101.0/24
add action=accept chain=prerouting dst-address=192.168.102.0/24
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=ether5 new-connection-mark=WAN1 passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=ether5 new-connection-mark=WAN2 passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=WAN1 in-interface=ether5 new-routing-mark=ether2-mark passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2 in-interface=ether5 new-routing-mark=ether3-mark passthrough=yes
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=ether2-mark passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=ether3-mark passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2 new-connection-mark=WAN1 passthrough=yes
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether3 new-connection-mark=WAN2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether2
add action=masquerade chain=srcnat out-interface=ether3
/ip route
add check-gateway=ping distance=1 gateway=192.168.101.1 routing-mark=ether2-mark
add check-gateway=ping distance=1 gateway=192.168.102.1 routing-mark=ether3-mark
add distance=1 gateway=192.168.101.1
add distance=2 gateway=192.168.102.1
/lcd interface pages
set 0 interfaces=wlan1

This is the export with 6.21.1

/interface ethernet
set [ find default-name=ether1 ] speed=1Gbps
set [ find default-name=ether2 ] speed=1Gbps
set [ find default-name=ether3 ] speed=1Gbps
set [ find default-name=ether4 ] speed=1Gbps
set [ find default-name=ether5 ] speed=1Gbps
set [ find default-name=ether6 ] advertise=10M-half,10M-full,100M-half,100M-full
set [ find default-name=ether7 ] advertise=10M-half,10M-full,100M-half,100M-full
set [ find default-name=ether8 ] advertise=10M-half,10M-full,100M-half,100M-full
set [ find default-name=ether9 ] advertise=10M-half,10M-full,100M-half,100M-full
set [ find default-name=ether10 ] advertise=10M-half,10M-full,100M-half,100M-full
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g l2mtu=2290 proprietary-extensions=pre-2.9.25
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc,aes-192-cbc,aes-256-cbc
/ip pool
add name=dhcp_pool0 ranges=192.168.12.50-192.168.12.254
/ip dhcp-server
add address-pool=dhcp_pool0 authoritative=yes disabled=no interface=ether5 name=dhcp1
/port
set 0 name=serial0
/snmp community
set [ find default=yes ] addresses=::/0
/system logging action
set 2 remember=yes
set 3 remote=0.0.0.0 src-address=0.0.0.0
/interface bridge port
add
/ip address
add address=192.168.12.1/24 interface=ether5 network=192.168.12.0
/ip dhcp-client
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=ether2
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=ether3
/ip dhcp-server network
add address=192.168.12.0/24 dns-server=192.168.12.1,8.8.8.8 gateway=192.168.12.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=192.168.12.1 name=ns1.testnet
/ip firewall mangle
add chain=prerouting dst-address=192.168.101.0/24
add chain=prerouting dst-address=192.168.102.0/24
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=ether5 new-connection-mark=WAN1 \
    per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=ether5 new-connection-mark=WAN2 \
    per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=prerouting connection-mark=WAN1 in-interface=ether5 new-routing-mark=ether2-mark
add action=mark-routing chain=prerouting connection-mark=WAN2 in-interface=ether5 new-routing-mark=ether3-mark
add action=mark-routing chain=output connection-mark=WAN1 new-routing-mark=ether2-mark
add action=mark-routing chain=output connection-mark=WAN2 new-routing-mark=ether3-mark
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether2 new-connection-mark=WAN1
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=ether3 new-connection-mark=WAN2
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether2
add action=masquerade chain=srcnat out-interface=ether3
/ip hotspot user
add comment="counters and limits for trial users" name=default-trial
/ip route
add check-gateway=ping distance=1 gateway=192.168.101.1 routing-mark=ether2-mark
add check-gateway=ping distance=1 gateway=192.168.102.1 routing-mark=ether3-mark
add distance=1 gateway=192.168.101.1
add distance=2 gateway=192.168.102.1
/ip upnp
set allow-disable-external-interface=no
/lcd interface pages
set 0 interfaces=wlan1
/snmp
set trap-community=public trap-generators=*ABC0002
/system clock
set time-zone-name=Europe/Brussels
/system leds
add

This configuration is based on this tutorial.
https://lcx.wien/blog/load-balancing-us ... -routeros/

I created this movie during the configuration
Here you can follow the configuration, ignore my voice in Dutch

I cannot explain it more clearly.

https://www.youtube.com/watch?v=iwPW7Fp ... e=youtu.be

sindy · Thu Mar 28, 2019 3:03 pm

Can you help me solve this with a working configuration. Or RouterOS needs to be updated.

I have written a few posts above what to do to identify the part which does not work properly. Your newest post has neither shown the answers to my questions nor provided the necessary information in any other form, you just declare again that it does not work and provide the 6.44.1 configuration you've already provided before.

I'm willing to help you identify the single root cause to be circumvented by a configuration change or to be claimed with Mikrotik as a bug, but to do so, I need the answers to my questions. I won't watch a 30 minutes video showing the individual steps of configuration when it is clear in advance that the configuration is correct and someting is broken in 6.44.1 or at least works different than it did before. And I also won't copy your configuration into one of my devices and debug it on my own because I'd have to imitate also the surrounding environment with two uplinks.

TeleBeni · Mon Apr 01, 2019 10:19 am

This configuration don't work.
I have not a commulatieve bandwidth and ISP2 become the main ISP in place off ISP1.

I want to set ISP1 to the main ISP because this is the fasted ISP.
WHo can help me.

in this case i can tell you a little bug in it:
With ROS 6.44.1
when all working, it takes all data from WAN2.
when i disconnect WAN2
it will contineu the internet feeds
PROBLEM: the connection at the clients side must be restarted! so the client need to reboot the SXT receiver before internet is back!
When i put back the WAN2 no problems

With ROS 6.21
when all working, it takes all data from WAN2.
when i disconnect WAN2
The internet will go off
it will contineu the internet feeds after a 4 till6 seconds
And clients need only to refresh the webpage! no reboot internet SXT receiver!
SO thats OK
ONLY 1 PROBLEM:
when i put back the WAN2
the it will contineu the internet feeds after a 4 till6 seconds AGAIN!
So at any interruption of the ISP the clinet will disconnected 2 times instead of 1

So conclusion ROS6.44.1 is totaly not stable!
Also we having more problems with it

Who is online