I had to replace a TPlink TL-SG3424 so I got the CRS326 and CSS326.

I have two servers HP DL380 gen6 with 4 nics running PROXMOX 5.2-6 . CPU 2x 4C8T and 24GB RAM and a NAS with 4 nics too.

I tested throughput with this iPerf build https://github.com/Aquantia/iperf for FullDuplex support, also iperf3 from apt-get.

So I tested with linux bond. (not openvswitch) on the proxmox servers.

CRS326-24G-2S+ with RouterOS 6.43rc66

TRANSMIT-HASH POLICY L2
HP-2 Server -- HP-1 Client.
BONDs: Linux Bond Config: iPerf with 4 threads Monitor interfaces: From this setup: We are talking L2 Hash policy (MAC), So from 1 IP to another IP (L3). Both received on 1 Interface but send only from 2 interfaces.
(I thought this only would use 1 NIC (1 MAC source 1 MAC dest).

So the next test :
Iperf from a PC with 1 NIC 4 threads and bidirectional In this case all four HP-2 nics send traffic to 1 NIC and the server receives on 1 NIC.


TRANSMIT-HASH POLICY L2+L3

Same setup with L2+L3
CRS326 config:
PROXMOX servers: Iperf bidirectional with 4 threads Same behaviour as L2, only one receive and two send.

Test with 1 client bidirectional:
iperf bidirectional 4 threads And as with L2, all four interfaces send traffic instead of two (between linux).

TRANSMIT-HASH POLICY L3+L4
This method wasn't avaliable on my previous switch, and this is not supported on 802.3ad standard. Now with iperf bidirectional full duplex (both servers HP-1 and HP-2) send and received traffic with 4 threads. Now we have both servers received on 4 NICs! but only sending with 2 interfaces (Same behaviour on L2, L2+L3, L4) The receiving part is strange.

Testing with 1 client, iperf bidirectional 4 threads. This test results in very low performance on the receiving side (HP-2) so I did another test just sending traffic to the server. This yields full gigabit performance but receiving on 4 nics (HP-2).

Another test:
The HP-1 send traffic iperf -R to the HP-2 server with 4 threads and uses 2 NIC to send and HP-2 receives with 2 nic. 2 gigabit.


My concerns about how mikrotik handles the LACP, L2 shouldn't be ONLY 1 NIC per MAC? So two clients use both independent NIC?

Another thing I don't know is why the servers onlye use 2 NIC to send traffic, but with a client they use both 4 NIC.

I have the CSS326 to test. But could't get the LACP working on swOS

CRS3xx series switches are using L2+L3+L4 as transmit hash policy globally, regardless what you select on the CRS3xx side.
https://wiki.mikrotik.com/wiki/Manual:C ... es#Bonding

Note: The built-in switch chip will always use Layer2+Layer3+Layer4 for transmit hash policy, changing the transmit hash policy manually will have no effect.

It is called the "transmit" hash policy, it makes a difference when sending or receiving traffic since the hash will be generated on the sender's side. Also, it is very important how you test the bonding interfaces, you can't expect that LACP will load balance single destination traffic:
https://wiki.mikrotik.com/wiki/Manual:L ... _balancing

CRS3xx series switches are using L2+L3+L4 as transmit hash policy globally, regardless what you select on the CRS3xx side.
https://wiki.mikrotik.com/wiki/Manual:C ... es#Bonding

Note: The built-in switch chip will always use Layer2+Layer3+Layer4 for transmit hash policy, changing the transmit hash policy manually will have no effect.

So the hash policty settings on the CRS-326 don't affect. (I'll test that later)

It is called the "transmit" hash policy, it makes a difference when sending or receiving traffic since the hash will be generated on the sender's side. Also, it is very important how you test the bonding interfaces, you can't expect that LACP will load balance single destination traffic:
https://wiki.mikrotik.com/wiki/Manual:L ... _balancing

My expectations for LACP always were, 1 link per client. So the servers with LACP, will have 1Gigabit throughput to the number o interfaces bonded. But with the CRS326 I was getting traffic from the sender (LACP proxmox) from all 4 interfaces to a single client when using multiple threads , independent of the mode (test posted above).

Fox example this is a test from another LACP server, with OpenvSwitch with 4 NICs 802.3ad "TCP balance" (L3+L4) Using: iPerf3 -c N54L-bond -P4 -R
Sending from a 1nic computer, and the bond receives from 4 NICs. I have to test this again changing to L2+L3, but I remember with this setting the server only received through 1 NIC. (As it should).

My concerns are about how this realy behaves, because as you said one single destination shouldn't work properly with LACP (even multiple connections).

About using 2 or more clients, can easly reach 3,8Gigabits of traffic through the servers with no problem.

I'll have to try the CSS 326 but the LACP tried three options none of them worked (LACP not enabled on client). (Swapping cables from working on CRS and other switches).

------
Another thing I have to check is the Aggregator ID is the same on both servers (1), usually on other switches you set up the ID (group in SwOS), for each server, but when setting up a new bond ROS takes care of that.

CRS3xx series switches are using L2+L3+L4 as transmit hash policy globally, regardless what you select on the CRS3xx side.
https://wiki.mikrotik.com/wiki/Manual:C ... es#Bonding

Note: The built-in switch chip will always use Layer2+Layer3+Layer4 for transmit hash policy, changing the transmit hash policy manually will have no effect.

Just tested that and changing this on the CRS326 doesn't affect the LACP. Good to know that!