Can someone write something like pros and cons for this three load balancing solutions?
Does multi wan configuration with NTH can be used on only one client PC? Lets say will the download accelerators or torrents works with all wans?
Is there solutions for ECMP flushing?
PCC is more like NTH or ECMP?
tnx
Re: ECMP vs NTH vs PCC
PCC is like extended ECMP without flushing =)
the main ideological difference between Nth and PCC is the client: in Nth each client uses one and only WAN interface; in PCC it uses many WAN interfaces
the main ideological difference between Nth and PCC is the client: in Nth each client uses one and only WAN interface; in PCC it uses many WAN interfaces
Re: ECMP vs NTH vs PCC
Tnx a lot - waiting for 3.24PCC is like extended ECMP without flushing =)
the main ideological difference between Nth and PCC is the client: in Nth each client uses one and only WAN interface; in PCC it uses many WAN interfaces
Re: ECMP vs NTH vs PCC
eeee..... if you write it other way around it can also be correct
the main ideological difference between Nth and PCC is the client: in Nth each client uses one and only WAN interface; in PCC it uses many WAN interfaces
- need to be more precisePCC is the same nth, but with additional feature - it can remember where does the previous packet from the "flow" went, and send the next packet the same way. You can identify flow by specifying set of options from src,dst addresses and ports
Re: ECMP vs NTH vs PCC
eeee..... if you write it other way around it can also be correct
the main ideological difference between Nth and PCC is the client: in Nth each client uses one and only WAN interface; in PCC it uses many WAN interfaces
PCC is the same nth, but with additional feature - it can remember where does the previous packet from the "flow" went, and send the next packet the same way. You can identify flow by specifying set of options from src,dst addresses and ports
So, that is great improvement for messingers, p2p, https .... tnx a lot. I got 3.24 beta from the support so I will test on my "guinea pigs" @ work
Re: ECMP vs NTH vs PCC
I've tried both pcc and nth. I'm having trouble understanding it. When i configure my mikrotik with PCC load balancing, bandwidth from both wan seems to marge. like i have 1MB in wan1 and 512 in wan2. in pcc i got around 1.4mb. but when i use nth. i seems to using only one wan1. the other wan BW remain useless. So my quarry is, Can i get BW using nth same as when i get bw using pcc?
Re: ECMP vs NTH vs PCC
so if i have 2 adsl line both is 2 Mb i have to use pcc to get 4Mb in internet surfing (download and browsing)?????
Re: ECMP vs NTH vs PCC
For anyone like me who has been doing endless googling and finding lots of examples that lack decent explanations, here is how I understand load balancing; nth vs pcc.
This assumes you read examples like https://aacable.wordpress.com/2011/08/2 ... te-script/ and are going cross-eyed.
Let's assume we have two equal wan links connecting us to the internet, one dsl and one cable modem.
Using firewall mangle rules 'nth' option we send every first packet out the dsl connection and every second packet out the cable connection:
Doing this will give you a completely balanced distribution across your two wan links, which is great if all your traffic is http unsecured stuff like in the good old days.
With this method, a user (host machine) can theoretically download two pictures at once giving the illusion of having doubled their speed. Why is this an illusion? Two trucks doing the speed limit each carrying a load of bricks will get the job done sooner than one truck doing the speed limit carrying a load of bricks. Nothing is moving any faster, you are just getting more done in the same time period.
But this method will break secure sessions such as ssl (https), ssh, tls, etc...
Why? Picture this overly simplified example of an ssh session:
See the difference? Our dsl link and our cable link have different public facing ip addresses.
Whenever you want to establish a secure session, it's connections have to come from the same
ip address every time. If the address suddenly changes, then the server can't verify that
it is coming the same user/host machine. It suspects a man in the middle attack, or that some third party machine is trying to hi-jack the data stream. Better safe than sorry, so it drops the connection and possibly breaks the session.
With nth, you can't run a secure session because your connections are always going out over different links.
"Ah" you ask, "but what if I mark-connections instead of marking packets?"
You should be doing this anyway as it is less processor intensive. Packets are already marked as being part of connections so marking them again is redundant, and a lot of extra work for the router.
Butt still no difference as far as sessions are concerned.
Connections are not the same as Sessions. A session can have multiple connections to a server. Facebook is a great example of this. One their main page, each picture, games, ads, status's are all transported over their own individual 'connection'. Once that object loads, that connection terminates and another can begin. (There is a limit on how many simultaneous connections you can run.)
All these connections are part of your session with Facebook.
(Yes, you can also have multiple sessions to a service, but lets stay on track.)
This is where PCC comes into play.
In this example, instead of just dealing out packets like a deck of cards ( one for you, one for you, one for you), we do a hash match on the details of a potential connection. You can choose from 9 combinations of details to hash to decide which modem link the connection should traverse, but each option has it's draw backs.
Long story short, "src-address" is the safest. Why? Back to Facebook: all those status updates, pictures, games, and ads all come from different servers all of which may have different ip addresses, but through the magic of money and shared session id's, they all remain part of the same session on their end. Good programmers and network engineers are worth the money.
Buttt this only works as long as all the requests from your computers session come from same ip address. If not, then the connection is broken for security reasons and that picture won't load, even though the rest of the page does. If you have been playing with load balancing, you have probably seen this happen.
Buttttt if you use only src-address in your pcc rules, then your computer will always use the same wan link every time because the source address is the ip address of your computer which probably doesn't change very often, and likely won't change in the middle of a session.
Unfortunately this means you have no load balancing at all and thus no illusion of increased speed.
Your sisters machine in the next room has a different ip address than yours and so she is probably using the cable modem, which at least means you have the dsl link to yourself. This is all well and good until your mother gets online to play farmville. Now one of you has to share a link.
So you use "both-addresses" in your pcc rule. This means that the hash will be different some times because even though your src-address is the same, if your destination-address is different, the link used may be different. This is the desired result of using PCC to load balance connections.
Butttttt now you run the risk of breaking sessions because the connections, though part of the same session, may arrive at the server farm from different ip addresses. Only one of those addresses is associated with the session. The other is ignored at best, or treated as a potential security problem at worst. A lot of this depends on how the service you are using distributes their content.
There are 9 options in the PCC drop down list. "src-address" is the only one guaranteed to maintain sessions such as ssh, tsl, ssl (https) etc... Many people use "both-addresses" and say they have no problems. Experiment at your own risk.
Your non-session http traffic will seem faster if you use one of the other 8 options, but almost everything is https these days (thanks to the paranoia about a certain TLA) and well it should be. Do you really want your buddies at work wiresharking your unencrypted voip conversation with your 'wife' and playing it back at the Christmas party? (yes this is possible, even if you don't have Christmas parties).
So what is the point of load balancing if everyday use won't be any faster?
In a single user environment, there's very little point, especially if you have a good ISP link and a high data cap. Or zero data-cap. You'd probably be better off bonding multiple connections (where possible but that's way outside our scope here).
If you have a low data cap and are worried about overages, you can use PCC rules to swap between
your links. Of course, you could also do this just by swapping cables twice a month, but what fun is that.
I have a script that runs once a day via /system scheduler that rotates my PCC rules for me to better utilize my multiple connections. Yes, this can still break sessions, but I'm not usually doing anything at 4 am when it happens (except posting rants on forums).
The real benefit of load balancing these days is in a multiple user environment. The more hosts on your network, the more even your wan utilization will be. You might not be any faster, but you won't slow down as much.
The other benefit is automatic failover. When one wan link goes down, a properly configured router
will automatically send the traffic out the next default route with the longest bit match. This will still
break sessions. Things like a vpn connection will drop and need to reconnect, but you will still be "online". Voip calls and other udp (connectionless) connections may even survive the switch if you have a script triggered by the fail, and clear any voip connections in /ip firewall connections. I have yet to test this.
One final note: The PCC rules in all these examples you find will not catch traffic originating from the router itself. DNS lookups, pinging from the console, /tool fetch; all of these will go out whatever link matches first, which will probably be the same link every time. This is uneven distribution.
Here's what I do to load balance dns queries from the router:
DNS lookups are not sessions, so there is no need to maintain connections for them.
The dst-address list 'dsn' contains several known dns servers open to the public:
Google: 8.8.8.8, 8.8.4.4
Level 3: 4.2.2.x
Open DNS: 208.67.222.222, 208.67.220.220
Norton Connect Safe: 199.85.126.20, 199.85.127.20 ....
You should add your ISP's dns servers here as well if you use them.
I know this isn't a technically accurate explanation, but it isn't meant to be.
I hope this saves someone like me a lot of fruitless Googling and frustration.
If there is anything I have glaringly wrong, please feel free to contribute to the conversation.
Good luck.
(edited to correct 4am sleep deprived typos)
This assumes you read examples like https://aacable.wordpress.com/2011/08/2 ... te-script/ and are going cross-eyed.
Let's assume we have two equal wan links connecting us to the internet, one dsl and one cable modem.
Using firewall mangle rules 'nth' option we send every first packet out the dsl connection and every second packet out the cable connection:
Code: Select all
/ip firewall mangle
add chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface=bridge-local nth=2,1 connection-state=new \
action=mark-connection new-connection-mark=cn_ether1
add chain=prerouting connection-mark=no-mark dst-address-type=!local \
in-interface=bridge-local nth=2/2 connection-state=new \
action=mark-connection new-connection-mark=cn_ether2
Doing this will give you a completely balanced distribution across your two wan links, which is great if all your traffic is http unsecured stuff like in the good old days.
With this method, a user (host machine) can theoretically download two pictures at once giving the illusion of having doubled their speed. Why is this an illusion? Two trucks doing the speed limit each carrying a load of bricks will get the job done sooner than one truck doing the speed limit carrying a load of bricks. Nothing is moving any faster, you are just getting more done in the same time period.
But this method will break secure sessions such as ssl (https), ssh, tls, etc...
Why? Picture this overly simplified example of an ssh session:
Code: Select all
you-dsl: hellow server
server-to-dsl: acknowledge, username please
you-cbl: my username is barney
server-to-cbl: wait what? who are you?
you-dsl: my password is hunter2
server-to-dsl: what? no, i didn't ask for your password, i asked for your username
Whenever you want to establish a secure session, it's connections have to come from the same
ip address every time. If the address suddenly changes, then the server can't verify that
it is coming the same user/host machine. It suspects a man in the middle attack, or that some third party machine is trying to hi-jack the data stream. Better safe than sorry, so it drops the connection and possibly breaks the session.
With nth, you can't run a secure session because your connections are always going out over different links.
"Ah" you ask, "but what if I mark-connections instead of marking packets?"
You should be doing this anyway as it is less processor intensive. Packets are already marked as being part of connections so marking them again is redundant, and a lot of extra work for the router.
Butt still no difference as far as sessions are concerned.
Connections are not the same as Sessions. A session can have multiple connections to a server. Facebook is a great example of this. One their main page, each picture, games, ads, status's are all transported over their own individual 'connection'. Once that object loads, that connection terminates and another can begin. (There is a limit on how many simultaneous connections you can run.)
All these connections are part of your session with Facebook.
(Yes, you can also have multiple sessions to a service, but lets stay on track.)
This is where PCC comes into play.
Code: Select all
/ip firewall mangle
add chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local \
per-connection-classifier=src-address:2/0 action=mark-connection new-connection-mark=cn_ether1
add chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local \
per-connection-classifier=src-address:2/1 action=mark-connection new-connection-mark=cn_ether2
Long story short, "src-address" is the safest. Why? Back to Facebook: all those status updates, pictures, games, and ads all come from different servers all of which may have different ip addresses, but through the magic of money and shared session id's, they all remain part of the same session on their end. Good programmers and network engineers are worth the money.
Buttt this only works as long as all the requests from your computers session come from same ip address. If not, then the connection is broken for security reasons and that picture won't load, even though the rest of the page does. If you have been playing with load balancing, you have probably seen this happen.
Buttttt if you use only src-address in your pcc rules, then your computer will always use the same wan link every time because the source address is the ip address of your computer which probably doesn't change very often, and likely won't change in the middle of a session.
Unfortunately this means you have no load balancing at all and thus no illusion of increased speed.
Your sisters machine in the next room has a different ip address than yours and so she is probably using the cable modem, which at least means you have the dsl link to yourself. This is all well and good until your mother gets online to play farmville. Now one of you has to share a link.
So you use "both-addresses" in your pcc rule. This means that the hash will be different some times because even though your src-address is the same, if your destination-address is different, the link used may be different. This is the desired result of using PCC to load balance connections.
Butttttt now you run the risk of breaking sessions because the connections, though part of the same session, may arrive at the server farm from different ip addresses. Only one of those addresses is associated with the session. The other is ignored at best, or treated as a potential security problem at worst. A lot of this depends on how the service you are using distributes their content.
There are 9 options in the PCC drop down list. "src-address" is the only one guaranteed to maintain sessions such as ssh, tsl, ssl (https) etc... Many people use "both-addresses" and say they have no problems. Experiment at your own risk.
Your non-session http traffic will seem faster if you use one of the other 8 options, but almost everything is https these days (thanks to the paranoia about a certain TLA) and well it should be. Do you really want your buddies at work wiresharking your unencrypted voip conversation with your 'wife' and playing it back at the Christmas party? (yes this is possible, even if you don't have Christmas parties).
So what is the point of load balancing if everyday use won't be any faster?
In a single user environment, there's very little point, especially if you have a good ISP link and a high data cap. Or zero data-cap. You'd probably be better off bonding multiple connections (where possible but that's way outside our scope here).
If you have a low data cap and are worried about overages, you can use PCC rules to swap between
your links. Of course, you could also do this just by swapping cables twice a month, but what fun is that.
I have a script that runs once a day via /system scheduler that rotates my PCC rules for me to better utilize my multiple connections. Yes, this can still break sessions, but I'm not usually doing anything at 4 am when it happens (except posting rants on forums).
The real benefit of load balancing these days is in a multiple user environment. The more hosts on your network, the more even your wan utilization will be. You might not be any faster, but you won't slow down as much.
The other benefit is automatic failover. When one wan link goes down, a properly configured router
will automatically send the traffic out the next default route with the longest bit match. This will still
break sessions. Things like a vpn connection will drop and need to reconnect, but you will still be "online". Voip calls and other udp (connectionless) connections may even survive the switch if you have a script triggered by the fail, and clear any voip connections in /ip firewall connections. I have yet to test this.
One final note: The PCC rules in all these examples you find will not catch traffic originating from the router itself. DNS lookups, pinging from the console, /tool fetch; all of these will go out whatever link matches first, which will probably be the same link every time. This is uneven distribution.
Here's what I do to load balance dns queries from the router:
Code: Select all
add chain=postrouting connection-mark=no-mark connection-state=new dst-address-list=dns nth=2,2 \
action=mark-connection new-connection-mark=cn_ether2
add chain=postrouting connection-mark=no-mark connection-state=new dst-address-list=dns nth=2,1 \
action=mark-connection new-connection-mark=cn_ether1
The dst-address list 'dsn' contains several known dns servers open to the public:
Google: 8.8.8.8, 8.8.4.4
Level 3: 4.2.2.x
Open DNS: 208.67.222.222, 208.67.220.220
Norton Connect Safe: 199.85.126.20, 199.85.127.20 ....
You should add your ISP's dns servers here as well if you use them.
I know this isn't a technically accurate explanation, but it isn't meant to be.
I hope this saves someone like me a lot of fruitless Googling and frustration.
If there is anything I have glaringly wrong, please feel free to contribute to the conversation.
Good luck.
(edited to correct 4am sleep deprived typos)
Last edited by LaZyLion on Sat Sep 12, 2015 8:24 am, edited 5 times in total.
Re: ECMP vs NTH vs PCC
Living with this setup for long time:
http://wiki.mikrotik.com/wiki/Manual:PC ... _Balancing
i haven't encounter what you stated here:
http://wiki.mikrotik.com/wiki/Manual:PC ... _Balancing
i haven't encounter what you stated here:
i'm pretty sure that all services that runs many parallel connections ether connect to same dst-address, or doesn't care about src-address of each connection.So you use "both-addresses" in your pcc rule. This means that the hash result will be different every time because
even though your src-address is always the same, your destination-address will be different.
Butttttt again, this will break your sessions because the connections, though part of the same session, will arrive at the
server farm from different ip addresses. Only one of those addresses is associated with the session.
The other is ignored at best, or treated as a potential security problem at worst.
Re: ECMP vs NTH vs PCC
LaZyLion, your post has in fact saved me much Googling and frustration, thank you.
I live in a building in a major city, but due to lack of density in my area, there is no drive for ISPs to upgrade infrastructure beyond the most basic DSL, compounded by a huge distance attenuating the DSL and no alternatives (cable, etc.). There are several high bandwidth businesses in a building all limited to about 200kbps upload and 3-4 Mbps download (at best). I have been salivating at the prospect of implementing some sort of link aggregation scheme (multiple users, multiple lines), but I have limited to no experience on the subject.
Could bonding/ECMP/PCC/<insert any other link aggregation scheme>/Magical Pink Unicorn to grant all my wishes/etc. be used to transparently choose when to aggregate links and not to aggregate links? If the user is doing a secured activity like internet banking, they could probably do it with a 14.4 modem so load balancing is not required. But if they are doing a bandwidth intensive application, like a UDP media stream from a website, I'd like it to aggregate its brains out. I think things get a bit trickier where you need a side order of security with the bandwith, such as an FTP upload. The link that macgaiver posted in his reply talks of routing interfaces but doesn't appear to pay attention to packet types.
This is also further complicated by bandwidth caps. Say one link can download 3 Mbps, and another could download at 4 Mbps, but both have the same badwidth cap. As-is, the ECMP protocol just looks at the fastest route, not how often the path is used. You could potentially overflow one cap while being under the cap in the other link.
The system doesn't have to be perfect. If a secure session breaks due to failover, that's an acceptable loss, as it isn't likely going to happen frequently (one hopes).
I live in a building in a major city, but due to lack of density in my area, there is no drive for ISPs to upgrade infrastructure beyond the most basic DSL, compounded by a huge distance attenuating the DSL and no alternatives (cable, etc.). There are several high bandwidth businesses in a building all limited to about 200kbps upload and 3-4 Mbps download (at best). I have been salivating at the prospect of implementing some sort of link aggregation scheme (multiple users, multiple lines), but I have limited to no experience on the subject.
Could bonding/ECMP/PCC/<insert any other link aggregation scheme>/Magical Pink Unicorn to grant all my wishes/etc. be used to transparently choose when to aggregate links and not to aggregate links? If the user is doing a secured activity like internet banking, they could probably do it with a 14.4 modem so load balancing is not required. But if they are doing a bandwidth intensive application, like a UDP media stream from a website, I'd like it to aggregate its brains out. I think things get a bit trickier where you need a side order of security with the bandwith, such as an FTP upload. The link that macgaiver posted in his reply talks of routing interfaces but doesn't appear to pay attention to packet types.
This is also further complicated by bandwidth caps. Say one link can download 3 Mbps, and another could download at 4 Mbps, but both have the same badwidth cap. As-is, the ECMP protocol just looks at the fastest route, not how often the path is used. You could potentially overflow one cap while being under the cap in the other link.
The system doesn't have to be perfect. If a secure session breaks due to failover, that's an acceptable loss, as it isn't likely going to happen frequently (one hopes).
Re: ECMP vs NTH vs PCC
Hi Guys,
Using PCC, slow down my LAN connections a lot whenever I have to go WAN through mikrotik. I use 2 of unequal WAN by aacble.wordpress.com - and the connections are just fine but the download speed decrease tremendously.
1st WAN = 3 Mbps
2nd WAN = 40 Mbps
Using 2nd WAN, I usually download 100 Mb file in approximately a minutes or less. Now my LAN is behind mikrotik and it took about +30 minutes download. I lost quite a lot by having PCC on for load balancing in Routerboard 750.
Does it make it any different if I use more powerful routerboard? like rb 450g ?
Or is it something wrong with my script?
hardware problem or software problem?
The link has no problem to download in hurry.
thanks
Using PCC, slow down my LAN connections a lot whenever I have to go WAN through mikrotik. I use 2 of unequal WAN by aacble.wordpress.com - and the connections are just fine but the download speed decrease tremendously.
1st WAN = 3 Mbps
2nd WAN = 40 Mbps
Using 2nd WAN, I usually download 100 Mb file in approximately a minutes or less. Now my LAN is behind mikrotik and it took about +30 minutes download. I lost quite a lot by having PCC on for load balancing in Routerboard 750.
Does it make it any different if I use more powerful routerboard? like rb 450g ?
Or is it something wrong with my script?
hardware problem or software problem?
The link has no problem to download in hurry.
thanks
Re: ECMP vs NTH vs PCC
LaZyLion, I also recognized that only a PCC based on the source address will preserve the session. However, my experience shows that most of the times using the destination address works fine even with secure connections like online banking. I don't know about Facebook because I don't use facebook.
IN the few times I have experienced problems that may be caused by PCC, I disable one of the WAN connections using the interface menu. Although this is quite clumsy, being able to bind together the bandwidth of my two 2Mbps connections for the extra bandwidth is worth the extra effort.
BTW, to simplify router processing, I only use destination addresses because most of the time I am the only one using my connections, so including source address doesn't really add any additional hash.
Great post! Cheers.
IN the few times I have experienced problems that may be caused by PCC, I disable one of the WAN connections using the interface menu. Although this is quite clumsy, being able to bind together the bandwidth of my two 2Mbps connections for the extra bandwidth is worth the extra effort.
BTW, to simplify router processing, I only use destination addresses because most of the time I am the only one using my connections, so including source address doesn't really add any additional hash.
Great post! Cheers.
Re: ECMP vs NTH vs PCC
One thing to consider when using load balancing/sharing in this scenario, any connection which gets mapped to the 3M ISP won't be able to go faster than 3M. Unfortunately, there's nothing that can be done about this - it's just how things are. Perhaps your test was being mapped to the 3M connection. Depending on your PCC mapping, you might be always getting mapped to the 3M connection based on your IP address. Open two or three different "what is my IP address" sites like myipaddress.com, ipchicken.com, and ipquail.com - see which provider's IP each comes from, and if they're all coming from one or the other. Of course, no matter what that shows, it's not necessarily going to be the same for your download test.
You could start a download and if it's going slow, look in torch and connection tracking to see which reply-address is being used - that will tell you which ISP the download is coming across - if it's the fast ISP, then there is a performance issue. Also check CPU load whenever a download is going slow.
You could start a download and if it's going slow, look in torch and connection tracking to see which reply-address is being used - that will tell you which ISP the download is coming across - if it's the fast ISP, then there is a performance issue. Also check CPU load whenever a download is going slow.
Who is online
Users browsing this forum: Ahrefs [Bot], Google [Bot], miankamran7100, rizan, sirbryan and 19 guests