Hello

I have two ISPs. Both have 50mbit download and 5mbit upload individually. I believe I have load balancing working correctly. When i do speed tests I can see around 90mbit download speeds. However during uploads, it never passes 5mbit. I even tried paralel but independent upload tests with iperf3. The result always seemed to be, one ISP using all upload bandwidth, and the other one doing nothing because there is nothing left for it. how can I setup mikrotik to know how much bandwith is available for each ISP and use that outgoing connection for higher speeds? Is this even possible? Thank you

You cannot learn what are the limitations of your upload bandwidth at ISP side or further in the network. You can use Mikrotik's queues to actively limit the upload bandwidth of different classes of uplink traffic generated in your LAN (and by the router itself) according to your preferences as it leaves your router. There's a nice guide on that by @pcunite.

But it is still weird that you'd be getting full 5 Mbit/s on one uplink and zero on the other one, it rather sounds like some mistake in the load balancing... do you use per-connection-classifier or some other strategy to choose an uplink for a connection? The per-connection-classifier may allocate the same uplink to all iperf3 instances due to inappropriate choice of the classifying hash.

If there is really no issue in your configuration, it may also be the ISP cheating.

I was hoping something like this.
I know my maximum upload bandwidth is 5mbit per ISP. If for ISP1, I'm using 70% of upload bandwidth, and for ISP2, I'm using 30% of upload bandwidth, then the new upload should use ISP2. Is this an unrealistic expectation?

As for my config. It is like this. I'm not sure what strategy it uses. I followed the instructions on https://www.youtube.com/watch?v=DDMD1GVg84M It just seemed to work.

When I do several times, I get returned different ips. It almost alternates between ISP1 and ISP2 Ips evenly.

My config is below

I'm not sure what strategy it uses. I followed the instructions on https://www.youtube.com/watch?v=DDMD1GVg84M It just seemed to work.
When I do

Code: Select all

 curl ifconfig.me 

several times, I get returned different ips. It almost alternates between ISP1 and ISP2 Ips evenly.
My config is below

This setup uses ECMP load balancing for connections initiated from LAN or the router itself, and makes sure that incoming requests to the router itself (like VPN connections initiated by remote client, remote management connections via WAN) are responded via the same WAN through which they arrived. ECMP load balancing is iterating the list of gateways on the chosen route using round robin, hence in your case, every odd connection uses one gateway and every even one uses the other one; I'm not sure whether it is still the case if both source IP and destination IP of the connection are the same, i.e. whether the routing cache is somehow linked with connection tracking and thus also takes source and destination ports into account.

I was hoping something like this.
I know my maximum upload bandwidth is 5mbit per ISP. If for ISP1, I'm using 70% of upload bandwidth, and for ISP2, I'm using 30% of upload bandwidth, then the new upload should use ISP2. Is this an unrealistic expectation?

I'm afraid that watching the current traffic volume at each uplink and choosing the least loaded one (in % of its maximum capacity) for a new connection would be too much work for too little effect as the networking environment is very dynamic. So even if you measure the currently occupied bandwidth every second (which is about the fastest pace you can obtain using scripts), during that second tens to hundreds of existing connections can finish and tens to hundreds new ones can emerge. Plus each connection usually requires a different bandwidth, as the speed accomodates to the bandwidth available at the bottleneck of the path between the endpoints (client and server). So the best you can do is to distribute the connections evenly (taking the absolute bandwidth into account, i.e. if you have a 100 Mbit/s link and a 20 Mbit/s link, "evenly" means 5:1) and hope that statistically, it will work as needed. Where the upload/download ratios differ between the uplinks (like 50/50 and 20/4), there's no "ideal" strategy.

I suspect that video does not shoe the "full truth"

ECMP is based on per connection, so if src and dst address is same, you will only use one of the uplinks, it is not a "per packet" solution

Hello

So is the recommended approach to abandon ECMP and instead use PCC? Will this allow me to distribute the connections evenly? Can i make the disrtibution ratio different for download and upload?
Thank yoh

So is the recommended approach to abandon ECMP and instead use PCC?

As for me, yes. But it will cost you a bit of CPU power, as mangling is incompatible with fasttracking (which is only used for forwarded packets, not for connections to/from the router itself). So depending on your Mikrotik model, the consequences may vary from "unnoticed" to "unusable".

Will this allow me to distribute the connections evenly?

Yes

Can i make the disrtibution ratio different for download and upload?

No, because connections almost always use both directions (even where the payload as such goes in only one direction, some control packets like TCP ACK must be sent in the opposite direction). There are few exceptions to this (an audio/video stream in UDP may theoretically not need the reverse channel). And the choice of WAN has to be done when the very first packet of the connection is handled, and at that moment, it is not known yet whether the connection will be used for upload or for download; later, while the connection is already running, the router could compare the download and upload rates, but it cannot change the WAN used by an ongoing connection if there is src-nat anywhere on the way between the endpoints, because the remote server asociates the public IP of that WAN with that connection. So it would ignore packets coming from another address, and it keeps sending to the address from which the initial request came.

Ok. Let's say I want to apply the example here. https://wiki.mikrotik.com/wiki/Manual:PCC How would it work to my current setup? I have many vlans and a bridge.

TTNET1 and TTNET2 are my ISPs. My modems only allows dhcp connection from it. Therefore, I have set DMZ on the modems, with a static ip for the mikrotik DHCP clients.

Since I have so many VLANs, what do I choose as the interface for lines such as in the example

One could not expect to receive more than 50Mbps download and thus this thread is suspect from the first post.
The MT router will not use both ISPs for a single session as far I understand, but then again my MTUNA certification doesnt cover magical MT thinking. ;-P

I think speedtest.net uses multiple connections for download. That is where i saw more than 50mbit. But yeah for the rest i get it. I just need help to change my config to pcc now. Can you help me with that please. I am not sure how to adapt my vlans and bridge to the pcc example

I think speedtest.net uses multiple connections for download. That is where i saw more than 50mbit. But yeah for the rest i get it. I just need help to change my config to pcc now. Can you help me with that please. I am not sure how to adapt my vlans and bridge to the pcc example

Yes, speedtest.net does use multiple connections, but it is unclear if it uses multiple servers.

If the src / dst ports are also used in the route caching algorithm, then it will make sense why you probably see full bandwidth from a single device during speedtest. The Wiki article only mentions source / destination addresses, etc being used, nothing about src/dst port

EDIT: To add, you should use ECMP if no NAT is involved and PCC if NAT is involved

I dont use NAT right now. I just have. Mikrotik vpn server setup. People vpn first, grab a local ip, and co nect to internal servers.

So if this is the case i should use ecmp? Then is what i currenctly have what i can have as the ideal setup?

I dont use NAT right now. I just have. Mikrotik vpn server setup. People vpn first, grab a local ip, and co nect to internal servers.

So if this is the case i should use ecmp? Then is what i currenctly have what i can have as the ideal setup?

Where does VPN come into the picture now?

Your config shows LAN IPs in 10.x.x.x range, which is private IPs, so there must be NAT somewhere in the path on its way out to the internet?

I just have this for NAT. The modems have DMZ set, and direct all traffic to mikrotik dhcp clients with static ip addresses. When I said, I don't use NAT, I meant no special port forwarding. This much NAT seems to work with ECMP.
I have something like this for VPN:

Is that VPN only for connection to company LAN or the VPN users can connect to something in the internet via the VPN? If only for LAN, there is no need to distribute the load using ECMP or PCC, as the WAN uplink is chosen already by the user which connects to one or the other public IP.

Hello

I think the conversation led off tangent. I dont have a problem with vpn or try to load balance my connections via vpn to the internet. I want to do it to the physical servers on the LAN.

Right now, i use ECMP, I underatanf PCC can distrubute the load better. I have many vlans and a bridge. I am not sure how to apply the PCC example on mikrotik wiki. Thabk you all for helping btw.

I have many vlans and a bridge. I am not sure how to apply the PCC example on mikrotik wiki.

The per-connection-classifier just provides always the same result (match or mismatch) for all the packets belonging to the same connection, because it calculates a hash from source address, source port, destination address, and destination port (a combination of which is the same for each connection). There are cases where it is better not to include some of these identifiers into the hash, but we'll get to these later.

What action the rule takes if the per-connection-classifier matches, and what other matchers you add to the rule, is up to you. Without connection marking, the typical use is that the rules matching on per-connection-classifier directly assign a routing-mark, which is then used to choose the necessary set of routes (routing table). But the caveat is that if a packet has a routing-mark assigned, routes without any routing mark are only used to route it if none of the routes with that routing-mark matches. As you only have default routes with a routing-mark, they match any packet with such a routing-mark, even if its destination is one of your local subnets.

So either the firewall mangle rules assigning the routing-mark must assign it only to packets which you know will need to be sent via WAN, or you have to override the routing-mark assignment later on using /ip route rule rows. So one possible solution would be:

/interface list
add name=WAN

/interface list member
add list=WAN interface=TTNET1
add list=WAN interface=TTNET2

/ip firewall address-list
add list=local-subnets address=local.sub.net.A/mask
...
add list=local-subnets address=local.sub.net.Z/mask

/ip firewall mangle
add chain=prerouting in-interface-list=!WAN dst-address-list=!local-subnetsper-connection-classifier=src-address:2/0 \
action=mark-routing new-routing-mark="TTNET1 ROUTING"
add chain=prerouting in-interface-list=!WAN dst-address-list=!local-subnetsper-connection-classifier=src-address:2/1 \
action=mark-routing new-routing-mark="TTNET2 ROUTING"

If nothing at private addresses needs to be reached via the WANs, you can use an address-list of all private IPs instead of al your local subnets, which consists of just three prefixes for this purpose - 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16.

Regarding calculating the hash only from the local address (the source one in these rules in particular) - some services have problems if different TCP sessions related to the same operation arrive from different addresses. So for a small number of clients who have a short enough path to complain, it may give a smoother traffic distribution to use both-addresses-and-ports; if the clients who eventually experience the issues with these picky servers cannot deliver the information to you efficiently enough, it is much safer to use the same WAN for all connections of a given local user, otherwise the information about these easy to solve problems reaches you in the form of "your service is a (beep), it doesn't work". But for just a few local users with different traffic volumes, this approach causes uneven traffic distribution.

Thank you. So just to be clear. All I have to do is to have this on my mangle, and remove existing records. Nothing else? The routing rules stays the same? I am afraid it did work. When I had these none of the servers were able to access the internet.

All I have to do is to have this on my mangle, and remove existing records.

Nope, if you should have removed the existing mangle rules, I would have said that. You cannot remove the ones added based on the video (mangle chains input and output) because this would prevent your VPN server from working.

Nothing else? The routing rules stays the same?

Routes stay the same; I don't know whether you use any routing rules as you haven't posted them. And yes, nothing else to be added, nothing to be removed.

I am afraid it did not work. When I had these none of the servers were able to access the internet.

That's strange. Post your complete configuration export following the hint in my automatic signature just below. Obfuscate also the usernames of the VPN users before posting.

My routes seem to be simple. It makes sense, I get it, but it doesn't work for some weird reason.

Show me /interface list member print.

That looks correct to me

To me too. Do the mangle rules count (/ip firewall mangle print stats where action~"mark-routing")? They should, as otherwise they couldn't break anything :)
So post the complete config, not just the pieces you deem relevant.

OK. Here is the whole thing. I hope it helps. When I append your mangle commands, the servers stop accessing the internet ☹️

How did you manage to get in-interface-name="!WAN" (with quotes)???

I am unable to do this, neither from commandline nor from winbox, so I don't know how that rule behaves in such case.

It is impossible to debug a configuration I cannot see - you have posted a working one, not the broken one.

Hah. Honestly, I don't know about the quotes. I just use winbox to select from the dropdown and check the negation checkbox. If quotes are not meant to be there, I can remove them.

The broken one is just your lines appended to it. Here it is:

Okay, so print shows that differently from export, and the reason for the quotes is that there is the space in the name. Why the ! is shown inside the quotes rather than outside is a different point but it works the expected way (on my test CHR, that is). So no idea again.

With "my" rules disabled, when you ping 8.8.4.4 from the command line, indicating routing-mark="TTNET1 ROUTING", do the pings get responded?

If you modify "my" rules to assign some nonexistent routing-mark values and enable them, do they prevent the servers from getting to the internet as well?

This really weird.

If both rules are enabled: ping doesn't work
if only one rule is enabled(sometimes first rule, sometimes second I think. Can not be sure): ping works
if rules have a XXXYYYZZZ-1 and XXXYYYZZZ-2 routing mark: ping works

So what can be wrong when both of them are enabled?

Ok. I figured it out. I don't know the reason, but at least I have it working.

For some reason, when is used. Things stop working as well.

Just adding didn't work either.

However when I tried 192.168.xx.yy is not part of my local network really. they are just the subnets for the modem and the dhcp client. I was assuming !local had the same effect. But maybe not.

However since it works, I want to thank Sindy for all the help, and others as well.

Now that this is working. Why do i dirextly mark routes? Rather then first mark connections and then routes just like the examples? Is there a difference?

For some reason, when dst-address-type=!local is used. Things stop working as well.

This is a popular misunderstanding, likely powered by a no less popular wishful thinking.
dst-address-type=local (or src-address-type=local) matches when the address is question is any of the router's own addresses. It does not match on the whole subnets to which the router's own addresses belong (these are called "connected subnets").

Just adding dst-address=!10.0.0.0/8 didn't work either.
However when I tried
/ip firewall address-list
add address=10.0.0.0/8 list=LOCAL-SUBNETS
add address=192.168.20.0/24 list=LOCAL-SUBNETS
add address=192.168.10.0/24 list=LOCAL-SUBNETS

192.168.xx.yy is not part of my local network really. they are just the subnets for the modem and the dhcp client. I was assuming !local had the same effect. But maybe not.

Could it be that the servers themselves are checking availability of the gateways (192.168.x0.1)? Since there are no ports in ICMP protocol itself, it could be that check pings got always delivered to the "wrong" gateway thanks to the routing-mark (if so, swapping the 2/0 and 2/1 values between the rules would make it work).

Now that this is working. Why do i dirextly mark routes? Rather then first mark connections and then routes just like the examples? Is there a difference?

Connection marking has two main purposes:

• to match all packets of the connection except the initial one to only a single property (the connection-mark value) when deciding what treatment to give them, and evaluate the complex conditions needed to choose the treatment only once, when handling the first (or at latest the second) packet
• storing information which can only be obtained/generated while handling packets in one direction of the connection (or when handling the first packet of the connection) for further use while handling packets in the opposite direction - this is what you do when you assign a connection-mark in chain=input depending on the in-interface, and then use it to choose the routing table (=assign the routing-mark) to packets sent by the router itself in chain=output

Matching on per-connection-classifier is a specific case where all packets of the same direction of the same connection always match on the same rule; by translating the connection-mark to routing-mark instead, you can use other type of rules (like random) to control the distribution of the connections among the WAN interfaces - the first packet of a connection is sent to a randomly chosen WAN, and the connection-mark is a way to remember that WAN so that subsequent packets of the same connection would use it too, whilst they might not match the same random rule again.

See some related thoughts here.

Thank you for all your explanations

I am just surprised with we got it to work with just a route marking, rather than a bunch of connection markings and then route markings on top of those as described in the PCC wiki.

So all is working.

How do I change from gateway address check to an ip address like 8.8.8.8 and 8.8.4.4, so that mikrotik checks the actual internet connection and route traffic to the healthy ISP?

Thank you

The principle of that solution is that the topmost recursive routes are only active if the destination address of the bottommost one is available via its "physical" gateway. If the rest of the configuration is correct, which cannot be seen from just a list of routes, disconnection of one WAN (or its failure further in the ISP's network) only affects connections which use that WAN, but the connections using the other WAN must not suffer any drop-outs (except if caused by overload of that remining WAN by the other traffic).

A typical case would be that you have several routing tables:

• two for traffic which must use a particular WAN (e.g. incoming connections from outside); these contain only a default route via the respective WAN each, so it doesn't matter whether that route's gateway is the "physical" one or the "virtual" (recursive) one. If the path is down, it's down, nothing to do about it, so no need to monitor it.
• two for traffic which should prefer one of the WANs; these are used for the traffic which can be freely distributed among the WANs. In one of these latter routing tables, the default route via WAN A's virtual gateway is preferred and the default route via WAN B's virtual gateway is a backup one (lower distance); in the other one, the roles of the WANs are swapped. So as long as both WANs work, the "distributable" traffic is distributed in the ratio imposed by the rules assigng routing-marks; if one of the WANs fails, the traffic which is marked to prefer that WAN is also routed via the remaining one as there's the backup route in that routing table.

/ip route print detail while one of the WANs is down will show you which routes are active, which should help you analyse what's going on. The topmost recursive route through the WAN which is down should become inactive in 10 seconds at the latest (check-gateway is sending a group of pings every 10 seconds and if none of them is responded, it considers the gateway unreachable and deactivates the route).

Thank you for all the insight. I have anothef question.

Lets say i get 2 more internet connections. So 4 in total.

Can i PPC only two at a time and assign it to a specific subnet?

So for example

Subnet 1 uses isp 1 and isp 2
Subnet 2 uses isp 3 and isp 4

And if for subnet 1, both isp 1 and isp 2 fails, use usp 3 and isp 4

Is this possible?

Is this possible?

Of course it is. For this purpose, you'll have 4 routing tables:
two for local subnet 1, with the following order of preference of the WANs:
a: 1,2,3,4
b: 2,1,4,3
two for local subnet 2, with the following order of preference of the WANs:
c: 3,4
d: 4,3

And two pairs of PCC rules, one matching on src-address(-list)=subnet1 and assigning routing marks a and b, and the other pair matching on src-address(-list)=subnet2 and assigning routing marks c and d.