inner-VLAN Latency

cthompson · Wed Sep 02, 2015 4:02 pm

the purpose here is to improve the read/write to a QNAP NAS that was recently installed. It is on VLAN 24, users are coming from either wired[VLAN 20] or wireless[VLAN10] with the respective ip pools set as 192.168.<<VLAN-ID>>.0

The current problem:
# Traffic to our mail server is extremely slow, quite often timing out on simple things such as ssh or even telnet to snmp port[465] or port[25].
# Innter-Vlan traffic between VLAN 20/10 and VLAN 24 is extremely slow

rsync -av 100mb-dummy.test /mnt/test/
sending incremental file list
100mb-dummy.test

sent 104,883,294 bytes  received 35 bytes  3,555,367.08 bytes/sec
total size is 104,857,600  speedup is 1.00

this is from a wired gig connection:

ifconfig enp0s25
enp0s25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.20.196  netmask 255.255.255.0  broadcast 192.168.20.255
        inet6 fe80::56ee:75ff:fe51:7278  prefixlen 64  scopeid 0x20<link>
        ether 54:ee:75:51:72:78  txqueuelen 1000  (Ethernet)
        RX packets 2401386  bytes 398152298 (379.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2393253  bytes 911204394 (868.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xf4200000-f4220000

If it helps at all I did grab a packet capture of the transfer from my local machine:

cpdump -nes0 -i enp0s25 ether host 00:08:9b:ed:32:20
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s25, link-type EN10MB (Ethernet), capture size 262144 bytes
09:36:19.335393 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 458: 192.168.24.4.46117 > 239.255.255.250.ssdp: UDP, length 416
09:36:19.335435 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 462: 192.168.24.4.46117 > 239.255.255.250.ssdp: UDP, length 420
09:36:19.335442 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 494: 192.168.24.4.46117 > 239.255.255.250.ssdp: UDP, length 452
09:36:19.335446 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 524: 192.168.24.4.33860 > 239.255.255.250.ssdp: UDP, length 482
09:36:19.335449 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 522: 192.168.24.4.55217 > 239.255.255.250.ssdp: UDP, length 480
09:36:19.435522 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 458: 192.168.24.4.44762 > 239.255.255.250.ssdp: UDP, length 416
09:36:19.435550 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 462: 192.168.24.4.44762 > 239.255.255.250.ssdp: UDP, length 420
09:36:19.435558 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 494: 192.168.24.4.44762 > 239.255.255.250.ssdp: UDP, length 452
09:36:19.435564 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 524: 192.168.24.4.33476 > 239.255.255.250.ssdp: UDP, length 482
09:36:19.435570 00:08:9b:ed:32:20 > 01:00:5e:7f:ff:fa, ethertype IPv4 (0x0800), length 522: 192.168.24.4.48130 > 239.255.255.250.ssdp: UDP, length 480
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel

And lastly the configuration on the device, a CCR-1009 8 Port
I did try to paste the config here, but with all the layer 7 stuff I think I exceeded the size limit... so it's here
https://paste.fedoraproject.org/262486/11989221/

JJCinAZ · Wed Sep 02, 2015 6:14 pm

I didn't spend too much time looking through that config, but you're likely spending way too much time trying to classify traffic with layer-7 filters -- they are expensive in CPU time. Have you tried disabling all your filters and mangles so that you're merely routing between subnets in the most simple manner? Have you checked to make sure you're not NAT'ing inter-VLAN traffic where you shouldn't be?

cthompson · Wed Sep 02, 2015 7:11 pm

I didn't spend too much time looking through that config, but you're likely spending way too much time trying to classify traffic with layer-7 filters -- they are expensive in CPU time. Have you tried disabling all your filters and mangles so that you're merely routing between subnets in the most simple manner? Have you checked to make sure you're not NAT'ing inter-VLAN traffic where you shouldn't be?

I did test this by disabling all of those things. mangle tables, layer 7, firewall rules by means of the command line and saw no change. I don't believe any nat'ting is occurring based on the pcap included from above. I understand that layer 7 has a cost(cpu) to it, and mine certainly is extensive.. long enough that I had to pastebin it. but I don't forsee that causing the speeds to be 3MB/s

StubArea51 · Wed Sep 02, 2015 10:47 pm

Can you post your config? What code level are you on.

Typically you should see 1-2 ms in InterVLAN routing. I pushed more than 10 Gig of TCP based traffic through the CCR1009 we have in our lab.

Another important consideration for that specific platform is there have been reports of some physical issues with the first four ports that are in a switching group. I believe the issue was with links bouncing up and down but I don't remember all the details and if it was resolved in later versions of code. It might be useful to move your links to ports 5-9 and see if there are any improvements.

cthompson · Wed Sep 02, 2015 11:01 pm

I have noted a few things regarding the configuration of this:

1. the switch tagging(on the underlying device, a mikrotik CCS) was set up individually for tagging
ie. one line per tag
These can be grouped.. should they be?
ie.

10 X ports=ether3 service-vlan-format=any customer-vlan-format=any customer-vid=0 new-customer-vid=110 pcp-propagation=no sa-learning=yes 
and
13 X ports=ether21 service-vlan-format=any customer-vlan-format=any customer-vid=0 new-customer-vid=110 pcp-propagation=no sa-learning=yes

can be combined to be:
0 ports=ether3,ether4,ether6,ether21 service-vlan-format=any customer-vlan-format=any customer-vid=0 new-customer-vid=110 pcp-propagation=no sa-learning=yes

it has had no effect, as you can see above I have disabled the individual rules and combined the ports into one... with no effect.

I will most certainly try to test this further upstream and but to my knowledge the trunk on the device in question(ccr) is trunked on port 8. I will have to confirm the port of the NAS and get back.

Thank-You for your help.

cthompson · Thu Sep 03, 2015 7:06 pm

I have tested this by ensuring that the switch ports were bypassed on the router, the trunk port on this device is set to port 8. However, I moved the NAS up this device onto port 5 and tested again, the speeds were slightly better, but nothing significant

rsync -av speedtest/100mb-dummy.test /mnt/test/
sending incremental file list
100mb-dummy.test

sent 104,883,294 bytes  received 35 bytes  4,878,294.37 bytes/sec
total size is 104,857,600  speedup is 1.00

cthompson · Fri Sep 04, 2015 4:45 pm

I have tested a few things regarding this and had contact from Mikrotik Support, hopefully they will be able to assist in resolving this as it's difficult to see network improvements at this rate of transfer.

I did modify the switch configuration for the CRS226-24G-2S+

/system resource irq rps set [find] disabled=no

and am going to perform a controlled restart of both devices shortly.

The above code did not make a difference.

inner-VLAN Latency

inner-VLAN Latency

Re: inner-VLAN Latency

Re: inner-VLAN Latency

Re: inner-VLAN Latency

Re: inner-VLAN Latency

Re: inner-VLAN Latency

Re: inner-VLAN Latency