Hi members,

I just thought of this and was quite stumped about what would actually happen. In case my explanation is unclear, I have attached a diagram of it.

Say, ether 3 and ether 4 ports are connected to the same switch via 2 cat. 5 LAN cables; and 4 devices are connected to the switch. What will happen?

• Will there be 100mbps x 2 bandwidth between Mikrotik and the switch?

• If yes, then will all traffic be divided equally between 2 LAN cables?

• Or each device will saturate one LAN cable and the excess traffic will spill over to another LAN cable?

• Or each device will use a different LAN cable, that is device #1 & #2 will use cable A, device #3 & #4 will have their traffic flow thru cable B?

• Or perhaps the second LAN cable is actually redundant?

Thanks.
Henry

Depending on config, you'll end up with either:

a) one useless cable
b) loop (not good at all)
c) bonding (switch needs to support it)

If its a smart switch, then you could set it up with port control vice vlan tags...........

On another note why not trying to plug in a paperclip into an electrical socket or even better drink lysol to defeat the coronovirus.

Clearly, trying crap without some teeny weeny bit of research is usually not advised.

Thanks for the reply! This novice don't know that he needs to configure the switch in order to increase the bandwidth available to the upstream.

The diagram appears wrong, I thought connections from MT ports to a 1gig switch would also be 1 gig connections.

The diagram is meant to illustrate my question of could you simply just add another LAN cable to such a connection and you could increase the available bandwidth between a mikrotik and the devices.

By the replies, it isn't as simple as that. I'd need switch that supports link aggregation feature.

Will there be 100mbps x 2 bandwidth between Mikrotik and the switch?

> Yes if you configured some form of link-aggregation / bonding. Obviously your LAN-switch needs to support this.

If yes, then will all traffic be divided equally between 2 LAN cables?

> Probably not so easy to really get 50/50 exact. There are several "distribution" scheme's based on MAC-addresses, IP, TCP/UDP ports all depending on the features of the products.
> Eg. If you have 1 single client it probably NEVER can utilize more then 1 link. Such bonding scheme's are made if you have a good pool/mix of users (eg. office environment)

Or each device will saturate one LAN cable and the excess traffic will spill over to another LAN cable?

> Nope, probably not an option, I don't know a distribution-scheme like that

Or each device will use a different LAN cable, that is device #1 & #2 will use cable A, device #3 & #4 will have their traffic flow thru cable B?

> Not likely, other product might have very complex policy sets to distribute the traffic at L2/Ethernet level, but I'm not aware Mikrotik would be able to do that.

Or perhaps the second LAN cable is actually redundant?

> It is anyway. When you are "bonding" and somebody pulls a cable, you loos 50% capacity but traffic keeps flowing. Whatever "in flight" data is lost but per direct the other link it utilised until saturation.
> If you do not have a bonding-setup, classic STP (Spanning-Tree protocol) will ensure a loop-free topology and 1 of your links will NOT carry any traffic. It will just sit there being "passive" (=STP terminology is 'blocked port'). If you pull other the cable doing traffic, after a short time this 'blocked' port is taking the traffic.

if you were to get another switch. get a Mikrotik CRS and setup bounding.
out of curiosity, what is the reason for such topology?

Or each device will saturate one LAN cable and the excess traffic will spill over to another LAN cable?

This is normally not done (or not advisable to do) because such dynamic balancing would mean that at the receiving end the packets arrive out-of-order.
When using a bad TCP stack this will lead to massive retransmission of data causing the performance to be likely worse than with a single link.

Normally a TCP connection would transfer packets in sequence like 1 2 3 4 5 6 7 8 (in reality the numbers count bytes, disregard that) and when you have several links in some 50:50 balancing mode they would arrive e.g. as 1 2 3 6 7 4 5 8
A good TCP stack collects all this and sends back "ACK 8" to indicate it has everything OK.
A very bad TCP stack will send "NO! I NEED 4" when it receives 6 and again when it receives 7.
Then, 4 and 5 arrive (the original transmission) and it receives 8 and sends "NO! I need 6", by then the sender has re-sent 4 (maybe twice) which already was received and is dropped, then 6, and it is still waiting for 7 and 8.
A slightly better version will still send the "NO! I NEED 4" when it receives 6 but it will store the 6 and 7 before receiving 4 etc, but that still results in an unnecessary re-transmission of 4.
Furthermore, frequent retransmissions should trigger TCP to reduce the transmit speed (these bad TCP stacks will not do that, but compliant ones will) and thus the throughput is further reduced below what is possible.

assuming bounding between two Mikrotiks should not have such problem?

This problem does not exist in the router, it exists in the end-systems!

802.3ad (LACP) bonding does not have any negative effect on TCP connections...
Every existing connection always chooses the same link, they never get split between links... So there is no misordering...

if you were to get another switch. get a Mikrotik CRS and setup bounding.
out of curiosity, what is the reason for such topology?

During this COVID-19 pandameic business is slow, so I'm planning to tidy up my cables in my office.

Then came a situation where should I have 1 LAN cable running from switch A (located in rackmount) to switch B (located at cctv corner), then have 2 NVRs connected to switch B. This arrangement would be more future proof in case I might need to expand the number of NVRs.

Or should I have 2 LAN cables linking each NVR to switch A directly. However, when I want to increase the number of NVR I need to install another LAN cable (of course I could install more cables, just in case).

So a thought came, what if 2 switches (desktop grade) are linked by 2 cables, would there be an increase in bandwidth between the 2 switches.

So a thought came, what if 2 switches (desktop grade) are linked by 2 cables, would there be an increase in bandwidth between the 2 switches.

As explained by posters in previous posts: this only works if both switches support bonding with the same bonding strategy (803.2ad is the most likely candidate to be supported). Meaning they have to be managed switches so one can actually configure the bond. I'm not sure if many "desktop grade" switches support this...

This is not a problem if using port vlan, maybe MT switches dont have that capability.

802.3ad (LACP) bonding does not have any negative effect on TCP connections...
Every existing connection always chooses the same link, they never get split between links... So there is no misordering...

It depends on how you configure it. When you select mode balanced rr, which appears to be the default, there will be reordering.
When you use one of the header-hashing modes, this problem will not occur but you won't be able to use the full bandwidth on a single session.

@pe1chl i do not see the point on what you said...
The balancing modes are:
802.3ad, balance-rr, active-backup, balance-xor etc...
So according to what you said: It depends on how you configure it the answer is simple, 802.3ad is a Bodning Mode and not a variation of the Balance-rr mode...
So you can not configure a 802.3ad bonding to a balance-rr, they are different things, different modes...

Ok when you selec 802.3ad indeed it should work OK but beware that balanced rr is the default selection, not 802.3ad.
So I wonder what would happen when a user of the type "can I connect 2 LAN cables to have more bandwidth" will go to the trouble of selecting 802.3ad, and if he would not go and try balanced rr when he discovers that his speedtest still yields 100 Mbps and no more...

I just made a comment on TCP connections and 802.3ad...
The OP can choose the Mode that betters fits to his needs and ofcorse the mode that is supported by his equpment...

I've actualy setup a lab test (once has some spare time during lockdown) :
RB2011 (eth9 and eth10) to hAP ac Lite (eth1 and eth2)
so each link is 100Mbps

BUT, I can only get 98Mps udp between the two when using 802.3ad when testing with udp, 140Mbps TCP
with balance-rr, 190Mbps UDP and 140Mbps TCP

here is my config I can see when using 802.3ad only one link is used for udp but both links are argreated for TCP. why?

There's a nice article on wikipedia about different bonding modes. Read it, it'll answer all your questions.

Section about Linux bonding driver is the most interesting (as this is essentially what ROS does).

BUT, I can only get 98Mps udp between the two when using 802.3ad when testing with udp, 140Mbps TCP
with balance-rr, 190Mbps UDP and 140Mbps TCP

@Zacharias: you see, that is how it usually goes. people configure it, and use speedtest to test it, get disappointing results.
So that is why balance rr is quickly selected instead of 802.3ad, and probably also why it is default...

(of course in real life the two links would be nicely used with 802.3ad, but people do not do real life testing, they do speedtest benchmarking)

I can see when using 802.3ad only one link is used for udp but both links are argreated for TCP. why?

Likely because of the way you are testing. The TCP test may use different TCP sessions parallel and the UDP test doesn't.

thanks @mkx and @pe1chl

quick check on the wiki answers most of my question but thanks for more details on the topic

for someone else reading this:

layer-2-and-3 - This policy uses a combination of layer2 and layer3 protocol information to generate the hash. Uses XOR of hardware MAC addresses and IP addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave. For non-IP traffic, the formula is the same as for the layer2 transmit hash policy. This policy is intended to provide a more balanced distribution of traffic than layer2 alone, especially in environments where a layer3 gateway device is required to reach most destinations. This algorithm is 802.3ad compliant.

further test: using Layer-2 only, TCP test is running at 98Mbps at well. so without using Layer-3, mutiple TCP sessions are bounded to the same slave as well.

but I think this should be OK for real internet traffick where you would have mixture of TCP session and UDP stream between different IP address.

Now think of it, it's the same as loadbalacing two gateway, if you want to keep the same IP session on the same gateway, you won't have total uplink bandwidth, .

Glad I have learned something new today!

BUT, I can only get 98Mps udp between the two when using 802.3ad when testing with udp, 140Mbps TCP

802.3ad does not double the bandwidth nor i ever said it does...
If you do not care about the misordering of the Frames as far as TCP connections are concerned and the negative effects of that do not use it...

Also i would suggest you at least take a look at the Wiki about the Bonding modes, testing random modes without knowing what mode does what is something that i can not understand...