question about transmit hash policy

blackmetal · Tue Sep 25, 2018 12:50 pm

Hello,
i have a mikrotik ccr 1016 and i have connected 5 ports from mikrotik to a brocade 6450 and this mikrotik configs :

Flags: X - disabled, R - running
0 R name="Cogent-Uplink" mtu=1500 mac-address=XXXXX arp=enabled
arp-timeout=auto slaves=Ether1->Uplink,Ether4>Uplink mode=802.3ad
primary=none link-monitoring=mii arp-interval=100ms arp-ip-targets=""
mii-interval=100ms down-delay=0ms up-delay=0ms lacp-rate=30secs
transmit-hash-policy=layer-2 min-links=0

1 R name="SWITCH-Uplink" mtu=1500 mac-address=XXXXXX arp=enabled
arp-timeout=auto slaves=Ether2->SWT-Uplink(BrocadeP47),Ether3->SWT-
Uplink(BrocadeP48),Ether5->SWT-Uplink(BrocadeP46),Ether6->SWT-
Uplink(BrocadeP45),Ether7->SWT-Uplink(BrocadeP44)
mode=802.3ad primary=none link-monitoring=mii arp-interval=100ms
arp-ip-targets="" mii-interval=100ms down-delay=0ms up-delay=0ms
lacp-rate=30secs transmit-hash-policy=layer-2 min-links=0

and this is brocade configs :

lag UPLINK-RJ45-SWT dynamic id 2
ports ethernet 1/1/44 to 1/1/48
primary-port 1/1/47
deploy

so my questions are :
1. right now can i have 5gb throughput from my brocade switch to mikrotik ccr 1016 ?
2. whats difference between transmit-hash-policy layer 2 AND layer2-layer3 and which one is better i choose it?
3. when i read https://wiki.mikrotik.com/wiki/Manual:L ... _balancing in "Problem" section i see in some cases we may be limit to 1gbps speed, but i do not understand it well, can anybody explain it better for me?

Thank you.

jprietove · Tue Sep 25, 2018 1:20 pm

so my questions are :
1. right now can i have 5gb throughput from my brocade switch to mikrotik ccr 1016 ?

You have configured your bonding mode=802.3ad. This means that "LACP balances outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. The hash includes the Ethernet source and destination address and if available, the VLAN tag, and the IPv4/IPv6 source and destination address. How this is calculated depends on transmit-hash-policy parameter. " https://wiki.mikrotik.com/wiki/Manual:I ... ng#802.3ad

As you have configured transmit-hash-policy=layer-2, every time a packet sends outgoing traffic across your bond interface, RouterOS will compute a HASH based on MAC-Address of source and destination. This number will be divided between 5(number of interfaces) and the result will be a number between 0 and 4, and an interface of the bonding will be selected.

If all the traffic goes from one SOURCE MAC to only one DESTINATION MAC, the hash will be always the same, so all the traffic will go through one interface. The result: 1 Gbps.

If there are multiples MAC addresses (source/destination), the hash will be equally distributed (more or less) and the traffic will use all the interfaces, achieving a MAX of 5Gbps.

If you want to be very near to 5Gbps, you can choose transmit-hash-policy=balance-rr. This way, first packet will go through the first interface, second will choose the second, and so on... the 6th packet will start again with the 1st interface. This is the way of achieving near 5Gbps. For this to work fine, all the interfaces should be the same speed and the transit time of the packets through every interface should be the same, so TCP packets are not received mis-ordered.

I think that, if your BROCADE is near your CCR, this is the best way to achieve your goal.

2. whats difference between transmit-hash-policy layer 2 AND layer2-layer3 and which one is better i choose it?

If in one side of your bonding you have one computer (this means, 1 MAC address only and 1 IP address only), and at the other side of the bonding you have one computer (1 MAC and 1 IP), the result of Layer 2 HASH will be the same ALWAYS, so bonding is for nothing, as the traffic will choose always the same interface.
If you choose Layer 2 and 3, the result will be the same: the traffic will choose always the same interface. No sum of speed.

If instead of having a computer at each side you have a Router, the traffic will have the same MAC (origin or destination) BUT different IPs (addresses of the traffic that the router is forwarding). In this case, HASH of Layer 2 will give always the same result (only one MAC as destination and one as source). But, HASH of Layer 2 and 3 will give different results, so the traffic will go through all the interfaces.

3. when i read https://wiki.mikrotik.com/wiki/Manual:L ... _balancing in "Problem" section i see in some cases we may be limit to 1gbps speed, but i do not understand it well, can anybody explain it better for me?

In the example, origin MAC is SERVER A mac address, and destination MAC is SERVER B. Hash policy Layer 2 will not result in 2Gbps, because the traffic will choose one interface (1 Gbps), and always the same.

blackmetal · Tue Sep 25, 2018 1:31 pm

so my questions are :
1. right now can i have 5gb throughput from my brocade switch to mikrotik ccr 1016 ?
You have configured your bonding mode=802.3ad. This means that "LACP balances outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. The hash includes the Ethernet source and destination address and if available, the VLAN tag, and the IPv4/IPv6 source and destination address. How this is calculated depends on transmit-hash-policy parameter. " https://wiki.mikrotik.com/wiki/Manual:I ... ng#802.3ad

As you have configured transmit-hash-policy=layer-2, every time a packet sends outgoing traffic across your bond interface, RouterOS will compute a HASH based on MAC-Address of source and destination. This number will be divided between 5(number of interfaces) and the result will be a number between 0 and 4, and an interface of the bonding will be selected.

If all the traffic goes from one SOURCE MAC to only one DESTINATION MAC, the hash will be always the same, so all the traffic will go through one interface. The result: 1 Gbps.

If there are multiples MAC addresses (source/destination), the hash will be equally distributed (more or less) and the traffic will use all the interfaces, achieving a MAX of 5Gbps.

If you want to be very near to 5Gbps, you can choose transmit-hash-policy=balance-rr. This way, first packet will go through the first interface, second will choose the second, and so on... the 6th packet will start again with the 1st interface. This is the way of achieving near 5Gbps. For this to work fine, all the interfaces should be the same speed and the transit time of the packets through every interface should be the same, so TCP packets are not received mis-ordered.

I think that, if your BROCADE is near your CCR, this is the best way to achieve your goal.

2. whats difference between transmit-hash-policy layer 2 AND layer2-layer3 and which one is better i choose it?
If in one side of your bonding you have one computer (this means, 1 MAC address only and 1 IP address only), and at the other side of the bonding you have one computer (1 MAC and 1 IP), the result of Layer 2 HASH will be the same ALWAYS, so bonding is for nothing, as the traffic will choose always the same interface.
If you choose Layer 2 and 3, the result will be the same: the traffic will choose always the same interface. No sum of speed.

If instead of having a computer at each side you have a Router, the traffic will have the same MAC (origin or destination) BUT different IPs (addresses of the traffic that the router is forwarding). In this case, HASH of Layer 2 will give always the same result (only one MAC as destination and one as source). But, HASH of Layer 2 and 3 will give different results, so the traffic will go through all the interfaces.

3. when i read https://wiki.mikrotik.com/wiki/Manual:L ... _balancing in "Problem" section i see in some cases we may be limit to 1gbps speed, but i do not understand it well, can anybody explain it better for me?
In the example, origin MAC is SERVER A mac address, and destination MAC is SERVER B. Hash policy Layer 2 will not result in 2Gbps, because the traffic will choose one interface (1 Gbps), and always the same.

Hi,
thank you for your clear explaination,
1. as i understand your explaination because i have a switch behind my routers and my computers are connected to switches i should choose layer2-layer3 transmit poliocy so in this case outgoing traffic will go towards all ports right? so i can have 5gbps throughput for outgoing traffic ? but if i choose layer 2 because it has same mac i will be limited to 1gbps outgoing traffic right?
2. if i choose layer 2 transmit hash policy when my computers behind brocade want send traffic to a network outside of my network can i have 5gbps throughput ?

thanks

blackmetal · Tue Sep 25, 2018 2:19 pm

in addition to my last posts please http://prntscr.com/kyh2a1
1. when i have set layter2-layer3 transmit hash i see outgoing traffic balanced over active LAG ports, but sometimes i see one of the ports outgoing traffic is around 180mb and its not balance, when i check flow i see its from one src ip to one dst ip.
2. when one of my ports bandwidth will be full other packet go towards others ports? my mean is can lacp detect if one of the ports bw wll be full send packets from other ports?
thanks

jprietove · Tue Sep 25, 2018 5:21 pm

1. as i understand your explaination because i have a switch behind my routers and my computers are connected to switches i should choose layer2-layer3 transmit poliocy so in this case outgoing traffic will go towards all ports right? so i can have 5gbps throughput for outgoing traffic ? but if i choose layer 2 because it has same mac i will be limited to 1gbps outgoing traffic right?
2. if i choose layer 2 transmit hash policy when my computers behind brocade want send traffic to a network outside of my network can i have 5gbps throughput ?

Suppose this is your configuration. Mikrotik bonding interface IP is, i.e. 10.0.0.1 and Gateway is 10.0.0.2:

PC: 192.168.1.10/24  +                                  + ether2 ---- 1/1/44 +
PC: 192.168.1.11/24  |                                  | ether3 ---- 1/1/45 |
PC: 192.168.1.24/24  |-- switch -- 192.168.1.1 Mikrotik + ether5 ---- 1/1/46 + Brocade ----- Gateway
...                                                     | ether6 ---- 1/1/47 |
PC: 192.168.1.150/24 +                                  + ether7 ---- 1/1/48 +

In this configuration, the traffic going from Mikrotik to the Gateway will have source MAC=Mtk Bonding MAC and destination MAC=Gateway MAC address.
If you choose Layer 2 Hash policy, Mikrotik will choose only one interface from the bonding interface. So the max expected speed = 1 Gbps.
If you choose Layer 3 and Layer 2 Hash policy, MikroTik will compute one hash for each packet using IP and MAC addresses. So:

[*] If Destination IP Address is the same, each PC will use an unique interface from the bonding. So Max Total Throughput will be 5 Gbps. But if the hash of traffic from two computers choose the same interface, each computers will share that individual 1 Gbps line.
[*] If Destination IP Address is not always the same, each PC will use different interfaces from the bonding to transmit. So Max Total Throughput will be 5 Gbps, and Individual computers will split their traffic across several interfaces, depending on the Destination IP address.

If you want to achieve 5 Gbps Aggregated, not depending on Source and Destination Addresses, you can use Balance-rr, because all the traffic will be equally splitted across the interfaces. You will see always the same bandwith use in each individual interface

jprietove · Tue Sep 25, 2018 5:23 pm

in addition to my last posts please http://prntscr.com/kyh2a1
1. when i have set layter2-layer3 transmit hash i see outgoing traffic balanced over active LAG ports, but sometimes i see one of the ports outgoing traffic is around 180mb and its not balance, when i check flow i see its from one src ip to one dst ip.
2. when one of my ports bandwidth will be full other packet go towards others ports? my mean is can lacp detect if one of the ports bw wll be full send packets from other ports?
thanks

1: traffic is not balanced because IP and MAC source/destination addresses traffic is not balanced. If you want balanced traffic, use Balance-rr

2: No. Use balance-rr if you want to balance the traffic so there will not be a full interface and the others unused

blackmetal · Tue Sep 25, 2018 5:48 pm

Hello,
in balance rr i think i will get retransmitting segments for tcp/ip so its better use layer2-layer3 transmit hash, so when layer2-layer3 transmit hash in enabled i do not have equal bandwidth like as balance rr but i can get throughput up to 5gbps , right?
thanks

jprietove · Tue Sep 25, 2018 6:25 pm

Hello,
in balance rr i think i will get retransmitting segments for tcp/ip

If you use same cables (good quality Cat6), same length, it should be OK: no retransmission errors, no jitter and no problems.

so its better use layer2-layer3 transmit hash, so when layer2-layer3 transmit hash in enabled i do not have equal bandwidth like as balance rr but i can get throughput up to 5gbps , right?
thanks

Right, UP to 5 Mbps and bandwidth not balanced between bonding interfaces

blackmetal · Tue Sep 25, 2018 6:32 pm

you helped me a lot,
i really thank you.,
and i think you have mistake i have up to 5gbps

you told i am able use up to 5Mbps

anyway thank you.
and alst question my friend is what will happen if one my slave port bandwidth will be full ? are the other packets drop ? or they will transmit from other slave ports?

jprietove · Tue Sep 25, 2018 7:42 pm

and i think you have mistake i have up to 5gbps you told i am able use up to 5Mbps anyway thank you.

You are right. 5 Gbps...

and alst question my friend is what will happen if one my slave port bandwidth will be full ? are the other packets drop ? or they will transmit from other slave ports?

I think that I already answered this... Using hash-policy=layer 2 or layer 2 and 3, when one port is full no other takes it place. If RouterOS makes the hash and decides that the packet should go through interface ether3 of then bonding interface, it will take that interface. If ether3 has no available bandwidth, the packet will be queued and it doesn't matters that ether4 has available bandwidth, for example.

BUT, using balance-rr, the packets will be equally balanced over all the interfaces of the bond. So, they will have the same bandwidth available at every moment

blackmetal · Tue Sep 25, 2018 8:02 pm

thanks man, you helped me a lot,
s you said if i use a good cat 6 cable for my bonding interface and use balance rr i have no problem with out of order tcp transmission and have no problem with udp or other protocols right?

again thank you you helped me alot

jprietove · Tue Sep 25, 2018 10:18 pm

That's it: it should work ok, as it was an unique 5Gbps interface

Enviado desde mi Redmi 3 mediante Tapatalk

wpsd2006 · Wed Apr 13, 2022 12:34 pm

Hi I know this is an old post, btw do you have the formula to calculate the hash so I can simulate the traffic both for layer2 and layer2-3 ?

question about transmit hash policy

question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Re: question about transmit hash policy

Who is online