Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

ias · Sun Dec 29, 2024 9:06 pm

Hi,

I've recently introduced vlan filtering in my home network and wired connectivity seems to be working OK after implementing suggestions shared in the below post:

viewtopic.php?t=213134

One remaining issue I have is the WIFI performance where I consistently get speeds below 100 Mbps when connected to 5GHz ac network (all my tests were performed less than 2 meters from the AP).

My home network consists of:
hex poe - acting as router/gateway/firewall/CAPsMAN
2x hex - acting as bridge with vlan filtering
3x audience - acting as CAPs controlled by hex poe

Below is the simple diagram of how they are connected:

IAS-NET.png

The network uses following vlans:
vlan10 for 192.168.1.0 network meant for wired and wireless clients (this eventually becomes untagged traffic)
vlan90 meant for 192.168.90.0 network used for management access to Mikrotik devices.

I intend to introduce additional vlans got guest and IoT WIFI, but I need to resolve the current performance issues first.

Below is how I intended to configure the ports:
hex poe:
ether1 - WAN
ether2 - hybrid port (vlan10 untagged, vlan90 tagged)
ether3,4,5 - trunk ports (vlan10 and vlan90 tagged)

hex
ether1,5 - trunk ports (vlan10 and vlan90 tagged)
ether2 - hybrid port (vlan10 untagged, vlan90 tagged)
ether3,4 - access ports (vlan10 untagged)

audience - ether1 - trunk port (vlan10 and vlan90 tagged)
ether2 - hybrid port (vlan10 untagged, vlan90 tagged)

The idea behind hybrid ports is to have them working as access ports for most devices, but also to be able to access management vlan with PC/laptop with virtual NIC configured for vlan90.

The performance issue I'm trying to narrow down is mostly visible on WIFI clients, but it seems there is something off with the wired network as well, even though it's not noticeable in everyday use, without looking at network stats. Tests performed with Tamosoft Throughput Test gave me following results in below 3 scenarios:
a) Server and client on 192.168.90.0 network - speed above 900 Mbps, occasional single digit % packet loss
b) Server and client on 192.168.1.0 network - speed above 800 Mbps, frequent single and double digit % packet loss
c) Server on wired and client on wireless 192.168.1.0 network - speed below 100 Mbps, consistent packet drops above 90%

So, I clearly broke something big time when introducing vlan setup but I'm not sure where precisely I went wrong. I suspect some default (or lack of) routing settings, or misconfiguration of firewall.

Here are exports of my configurations (hex, and audience configs are basically copies of one another). These are full exports, baring passwords, DHCP reservations and port forwarding rules, which are somewhat sensitive.

hex poe:

# 2024-12-29 15:59:25 by RouterOS 7.16.2
# software id = E8G3-FMJZ
#
# model = RB960PGS

/caps-man channel
add band=2ghz-g/n extension-channel=Ce frequency=2412 name=2g-1
add band=5ghz-a/n/ac extension-channel=Ceee frequency=5180 name=5g-36
add band=2ghz-g/n extension-channel=Ce frequency=2437 name=2g-6
add band=2ghz-g/n extension-channel=eC frequency=2472 name=2g-13
add band=5ghz-a/n/ac extension-channel=Ceee frequency=5260 name=5g-52
add band=5ghz-a/n/ac extension-channel=Ceee frequency=5320 name=5g-64
/interface bridge
add igmp-snooping=yes name=bridge-lan
/interface ethernet
set [ find default-name=sfp1 ] disabled=yes
/interface vlan
add interface=bridge-lan name=MGMT vlan-id=90
add interface=ether1 name=vlan-internet vlan-id=20
add interface=bridge-lan name=vlanGUEST vlan-id=30
add interface=bridge-lan name=vlanIAS vlan-id=10
/caps-man datapath
add bridge=bridge-lan local-forwarding=no name=datapath-int vlan-id=10 \
    vlan-mode=use-tag
/interface pppoe-client
add add-default-route=yes disabled=no interface=vlan-internet name=Benet \
    use-peer-dns=yes
/caps-man security
add authentication-types=wpa2-psk encryption=aes-ccm name=DefSec
/caps-man configuration
add channel=2g-1 channel.band=2ghz-g/n country=poland datapath=datapath-int \
    installation=indoor mode=ap name=2g-1-cfg security=DefSec ssid=domek-wifi
add channel=5g-36 channel.band=5ghz-a/n/ac country=poland datapath=\
    datapath-int installation=indoor mode=ap name=5g-36-cfg security=DefSec \
    ssid=domek-wifi
add channel=2g-6 channel.band=2ghz-g/n country=poland datapath=datapath-int \
    installation=indoor mode=ap name=2g-6-cfg security=DefSec ssid=domek-wifi
add channel=2g-13 channel.band=2ghz-g/n country=poland datapath=datapath-int \
    installation=indoor mode=ap name=2g-13-cfg security=DefSec ssid=\
    domek-wifi
add channel=5g-52 channel.band=5ghz-a/n/ac country=poland datapath=\
    datapath-int installation=indoor mode=ap name=5g-52-cfg security=DefSec \
    ssid=domek-wifi
add channel=5g-64 channel.band=5ghz-a/n/ac country=poland datapath=\
    datapath-int installation=indoor mode=ap name=5g-64-cfg security=DefSec \
    ssid=domek-wifi
/interface ethernet switch port
set 1 default-vlan-id=10 vlan-mode=secure
set 2 vlan-mode=secure
set 3 vlan-mode=secure
set 4 vlan-mode=secure
set 5 vlan-mode=secure
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=TRUSTED
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=pool_lan ranges=192.168.1.2-192.168.1.254
add name=pool_guest ranges=192.168.3.2-192.168.3.254
add name=pool_mgmt ranges=192.168.90.2-192.168.90.254
/ip dhcp-server
add address-pool=pool_lan interface=vlanIAS name=dhcp_lan
add address-pool=pool_guest interface=vlanGUEST name=dhcp_guest
add address-pool=pool_mgmt interface=MGMT name=dhcp_mgmt
/caps-man manager
set enabled=yes
/caps-man provisioning
add action=create-dynamic-enabled hw-supported-modes=gn identity-regexp=\
    "WIFI-2\$" master-configuration=2g-1-cfg name-format=prefix-identity \
    name-prefix=2G
add action=create-dynamic-enabled hw-supported-modes=a,ac,an identity-regexp=\
    "WIFI-2\$" master-configuration=5g-36-cfg name-format=prefix-identity \
    name-prefix=5G
add action=create-dynamic-enabled hw-supported-modes=gn identity-regexp=\
    "WIFI-1\$" master-configuration=2g-6-cfg name-format=prefix-identity \
    name-prefix=2G
add action=create-dynamic-enabled hw-supported-modes=gn identity-regexp=\
    "WIFI-0\$" master-configuration=2g-13-cfg name-format=prefix-identity \
    name-prefix=2G
add action=create-dynamic-enabled hw-supported-modes=a,ac,an identity-regexp=\
    "WIFI-1\$" master-configuration=5g-52-cfg name-format=prefix-identity \
    name-prefix=5G
add action=create-dynamic-enabled hw-supported-modes=a,ac,an identity-regexp=\
    "WIFI-0\$" master-configuration=5g-36-cfg name-format=prefix-identity \
    name-prefix=5G
/interface bridge port
add bridge=bridge-lan interface=ether2
add bridge=bridge-lan interface=ether3
add bridge=bridge-lan interface=ether4
add bridge=bridge-lan interface=ether5
/ip firewall connection tracking
set udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=TRUSTED
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface ethernet switch vlan
add independent-learning=yes ports=ether2,ether3,ether4,ether5,switch1-cpu \
    switch=switch1 vlan-id=10
add independent-learning=yes ports=ether3,ether4,ether5,switch1-cpu switch=\
    switch1 vlan-id=30
add independent-learning=yes ports=ether2,ether3,ether4,ether5,switch1-cpu \
    switch=switch1 vlan-id=90
/interface list member
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=vlanIAS list=LAN
add interface=vlanGUEST list=LAN
add interface=MGMT list=LAN
add interface=MGMT list=TRUSTED
/ip address
add address=192.168.1.1/24 interface=vlanIAS network=192.168.1.0
add address=192.168.3.1/24 interface=vlanGUEST network=192.168.3.0
add address=192.168.90.1/24 interface=MGMT network=192.168.90.0
/ip dhcp-client
add interface=vlan-internet
/ip dhcp-server network
add address=192.168.1.0/24 gateway=192.168.1.1
add address=192.168.3.0/24 gateway=192.168.3.1
add address=192.168.90.0/24 gateway=192.168.90.1
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat out-interface=Benet
/ip firewall service-port
set sip disabled=yes
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set api disabled=yes
/system clock
set time-zone-name=Europe/Warsaw
/system identity
set name=IAS-NET-1
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp server
set enabled=yes
/system ntp client servers
add address=212.1.104.9
/system routerboard settings
set auto-upgrade=yes
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=TRUSTED
/tool mac-server ping
set enabled=no

hex:

# 2024-12-29 18:11:14 by RouterOS 7.16.2
# software id = AZ9Q-ZAMF
#
# model = RB750Gr3

/interface bridge
add name=bridge-lan protocol-mode=none pvid=90 vlan-filtering=yes
/interface vlan
add interface=bridge-lan name=MGMT vlan-id=90
add interface=bridge-lan name=vlanIAS vlan-id=10
/interface list
add name=TRUSTED
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip smb users
set [ find default=yes ] disabled=yes
/interface bridge port
add bridge=bridge-lan comment="trunk port from hex" frame-types=\
    admit-only-vlan-tagged interface=ether1
add bridge=bridge-lan comment="hybrid port for PC" interface=ether2 pvid=10
add bridge=bridge-lan comment="access port to client" frame-types=\
    admit-only-untagged-and-priority-tagged interface=ether3 pvid=10
add bridge=bridge-lan comment="access port to client" frame-types=\
    admit-only-untagged-and-priority-tagged interface=ether4 pvid=10
add bridge=bridge-lan comment="trunk port for AP" frame-types=\
    admit-only-vlan-tagged interface=ether5
/ip neighbor discovery-settings
set discover-interface-list=TRUSTED
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=bridge-lan tagged=ether1,ether5 untagged=ether2,ether3,ether4 \
    vlan-ids=10
add bridge=bridge-lan tagged=ether1,ether2,ether5,bridge-lan vlan-ids=90
/interface list member
add interface=MGMT list=TRUSTED
/ip address
add address=192.168.90.3/24 disabled=yes interface=MGMT network=192.168.90.0
/ip dhcp-client
add interface=MGMT
/ip service
set telnet disabled=yes
set ftp disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ip smb shares
set [ find default=yes ] directory=/flash/pub
/system clock
set time-zone-name=Europe/Warsaw
/system identity
set name=IAS-NET-2
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=192.168.90.1
/system routerboard settings
set auto-upgrade=yes
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=TRUSTED
/tool mac-server ping
set enabled=no

audience:

# 2024-12-29 18:14:25 by RouterOS 7.16.2
# software id = 0YDV-TKFC
#
# model = RBD25G-5HPacQD2HPnD

/interface bridge
add name=bridge-lan protocol-mode=none pvid=90 vlan-filtering=yes
/interface wireless
# managed by CAPsMAN
# channel: 2472/20-eC/gn(16dBm), SSID: domek-wifi, CAPsMAN forwarding
set [ find default-name=wlan1 ] ssid=MikroTik
# managed by CAPsMAN
# channel: 5180/20-Ceee/ac/P(18dBm), SSID: domek-wifi, CAPsMAN forwarding
set [ find default-name=wlan2 ] ssid=MikroTik
set [ find default-name=wlan3 ] ssid=MikroTik
/interface vlan
add interface=bridge-lan name=MGMT vlan-id=90
/interface list
add name=TRUSTED
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge-lan comment="trunk port from hex" frame-types=\
    admit-only-vlan-tagged interface=ether1
add bridge=bridge-lan comment="hybrid port for PC" interface=ether2 pvid=10
/ip neighbor discovery-settings
set discover-interface-list=TRUSTED
/interface bridge vlan
add bridge=bridge-lan tagged=ether1 untagged=ether2 vlan-ids=10
add bridge=bridge-lan tagged=ether1,ether2,bridge-lan vlan-ids=90
/interface list member
add interface=MGMT list=TRUSTED
/interface wireless cap
# 
set caps-man-addresses=192.168.90.1 enabled=yes interfaces=wlan2,wlan1
/ip address
add address=192.168.90.6/24 disabled=yes interface=MGMT network=192.168.90.0
/ip dhcp-client
add interface=MGMT
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/Warsaw
/system identity
set name=IAS-WIFI-2
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=192.168.90.1
/system routerboard settings
set auto-upgrade=yes
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=TRUSTED

In case you're wondering about disabled IP addresses, I initially configured them as static but then changed to the same addresses being served by hex poe via DHCP reservation.

I would appreciate any suggestions on how to improve this configuration to eliminate the WIFI bottleneck (and potentially also the reason behind other pocket drops).

Sorry for the lengthy post but I tried to include all information which may be relevant - although chances are that I still managed to miss something

sindy · Sun Dec 29, 2024 10:43 pm

You have configured the only datapath row for capsman forwarding, which means that the CAP receives a wireless frame, encapsulates it into an Ethernet transport one and sends the result to the CAPsMAN device; the CAPsMAN device extracts the Ethernet frame from the transport one on a virtual wireless interface. That virtual interface is then a member port of the local bridge on the CAPsMAN sevice (the hEX PoE). I'm not sure whether the encapsulated traffic doesn't attempt to use L2 at some point and hit a wrong VLAN tagging/untagging, but if that was the case and something wouldn't pass due to that, I would expect a 100% loss.

So the only thing to come to my mind is to change the local-forwarding on the datapath row to yes, causing the wireless frames to be converted straight to the final Ethernet ones already on each cAP, relieving the hEX PoE from that extra load and maybe avoid some issue I haven't exactly identified.

Other than that, the Audiences have almost twice as powerful CPEs than the hEX PoE, so depending on your uplink bandwidth, it may make more sense to make one of them the router and use the hEX PoE only as a switch. But I'd definitely do this as a separate step.

And finally, as you do not use any fancy featues of CAPsMAN like placement of individual clients into VLANs dynamically depending on RADIUS response or access list rules, you may also consider switching to qcom-ac (wifi) drivers on the Audiences, the benefits being beamforming in the 5 GHz band and assisted roaming in both bands.

ias · Mon Dec 30, 2024 12:16 am

Hi sindy,

Thanks for your prompt response,

I've just tried to enable the local forwarding on the CapsMan datapath, but the effect was quite opposite to what I hoped for i.e. the wireless client couldn't even obtain an IP address after the change

Also, before submitting the original post I tired experimenting with moving CAPsMAN to one of the audiences (thinking that it had far more resources than hex poe) but even thought the first test results were encouraging, as the WIFI speeds on a client connected to audience running both CAPsMAN and Cap were around 300/300 Mbps (down/up) compared to my 600/600 WAN, when switching connection to other Caps, the speeds dropped to single digits (like 6-8 Mbps). This makes me think it's not about resources but configuration.

Also, before I configured hardware offloading, hex poe was often hitting 100% CPU utilisation during speedtests, but with the current configuration it doesn't go above 60-70% during speedtests that hits or go above 600/600 Mbps.

Do you have other suggestions as for what I might have misconfigured?

sindy · Mon Dec 30, 2024 10:36 am

On the CAPs (the audiences), what does /interface wireless cap print show?

ias · Mon Dec 30, 2024 11:13 am

It prints the same on all three of them:

/interface/wireless/cap> print
                            enabled: yes
                         interfaces: wlan2,wlan1
                        certificate: none
                   lock-to-caps-man: no
               discovery-interfaces: 
                 caps-man-addresses: 192.168.90.1
                     caps-man-names: 
  caps-man-certificate-common-names: 
                             bridge: none
                     static-virtual: no

sindy · Mon Dec 30, 2024 11:19 am

OK, so change bridge from none to bridge-lan on all three and try with local-forwarding=yes on the datapath row again. What is above my head is why it worked at all with none - again, I would expect a 100 % loss with this misconfiguration.

ias · Mon Dec 30, 2024 11:33 am

I think I got where your question was going (or maybe I got it wrong?) and I tried enabling local forwarding on the datapath once again, but this time I also set the bridge on tha CAPs to the local bridge, meaning that the print now looks like this:

/interface/wireless/cap> print
                            enabled: yes
                         interfaces: wlan2,wlan1
                        certificate: none
                   lock-to-caps-man: no
               discovery-interfaces: 
                 caps-man-addresses: 192.168.90.1
                     caps-man-names: 
  caps-man-certificate-common-names: 
                             bridge: bridge-lan
                     static-virtual: no

This restored the connectivity (i.e. the wireless clients are again being assigned their IPs on 192.168.1.0 network), but the bandwidth test results are exactly as they were before.

The last Tamosoft test from a Pixel phone to a PC running Tamosoft server (connected to the same hex the Audience serving that phone is connected to) is showing 97 Mbps UDP Up (interestingly there is 0% packet drop there) and 60 Mbps UDP Down with packet drop ranging between 80% and 98%

sindy · Mon Dec 30, 2024 11:37 am

While the PC is connected to etherX of the hEX, what does /interface ethernet monitor etherX once show?

ias · Mon Dec 30, 2024 11:43 am

The PC is on ether2 on IAS-NET-2 (hex). Here is the output:

/interface ethernet monitor ether2
                      name: ether2
                    status: link-ok
          auto-negotiation: done
                      rate: 1Gbps
               full-duplex: yes
           tx-flow-control: no
           rx-flow-control: no
                 supported: 10M-baseT-half,10M-baseT-full,100M-baseT-half,
                            100M-baseT-full,1G-baseT-half,1G-baseT-full
               advertising: 10M-baseT-half,10M-baseT-full,100M-baseT-half,
                            100M-baseT-full,1G-baseT-half,1G-baseT-full
  link-partner-advertising: 10M-baseT-half,10M-baseT-full,100M-baseT-half,
                            100M-baseT-full,1G-baseT-full

sindy · Mon Dec 30, 2024 11:48 am

And if you connect the PC to ether2 of the same Audience to which the wireless client is connected (still with local forwarding of course), what are the results of the throughput test, and what does the /interface ethernet monitor ether2 once show?

ias · Mon Dec 30, 2024 12:14 pm

Here is the the output of Audience ether2 monitor with laptop now connected the that port:

/interface ethernet monitor ether2
                      name: ether2
                    status: link-ok
          auto-negotiation: done
                      rate: 1Gbps
               full-duplex: yes
           tx-flow-control: no
           rx-flow-control: no
                 supported: 10M-baseT-half,10M-baseT-full,100M-baseT-half,
                            100M-baseT-full,1G-baseT-half,1G-baseT-full
               advertising: 10M-baseT-half,10M-baseT-full,100M-baseT-half,
                            100M-baseT-full,1G-baseT-half,1G-baseT-full
  link-partner-advertising: 10M-baseT-half,10M-baseT-full,100M-baseT-half,
                            100M-baseT-full,1G-baseT-full

As the laptop has physical NIC handling untagged traffic and virtual NIC configured for tagged vlan90 traffic (similarly to the PC running Tamasoft server), I can perform tests on both 192.168.1.0 and 192.168.90.0 networks. Below are the results:

Untagged vlan10:

Audience2_Ether2_vlan10.png

Tagged vlan90

Audience2_Ether2_vlan90.png

The speeds are fairly stable and averages shown on the screenshots are reflective of the speed. I'm somewhat concerned about these UDP packet drops, though (however, it's nothing near what I see when connected via WIFI).

sindy · Mon Dec 30, 2024 12:34 pm

Did you disable the 192.168.1.11 address on the test PC when you've told the client to connect to 192.168.90.11, or change it to some address in another subnet than 192.168.1.0/24 and 192.168.90.0/24? I don't know the Tamosoft product so I want to be sure it does not take shortcuts - a "normal" application that uses the Windows networking stack would use the interface to which 192.168.1.11 is attached to send data to 192.168.1.x even if using 192.168.90.11 as the source address in the packets.

Because when both the wireless client and the PC are in VLAN 10, the traffic between them is forwarded at L2 so it never leaves the Audience, so the result of almost 1000 Mbit/s in both directions is possible. But when the PC is in VLAN 90, the traffic from the wireless client must get from the Audience via the hEX Gr3 to hEX PoE, get routed from one subnet to the other, and go back to Audience. Since the Tamosoft thing seems to test both directions simultaneously, there is no way how the two 1 Gbit/s streams (192.168.1.x -> 192.168.90.11 and 192.168.90.11 -> 192.168.1.x) could fit into the same 1 Gbit/s capacity of the Ethernet uplink of the Audience. So here, I suspect that only the 192.168.1.x -> 192.168.90.11 traffic takes the long path, but the 192.168.90.11 -> 192.168.1.x one takes the shortcut.

ias · Mon Dec 30, 2024 2:26 pm

Interesting. I didn't consider testing across vlans. In my above test both: the PC acting as the Tamosoft server (192.168.1.11 / 192.168.90.11) and the test laptop acting as the Tamosoft client (192.168.1.x / 192.168.90.x) were connected to hybrid ports and got IPs from both network on their physical and virtual NICs. I assumed that the tests in this scenario would go through respective vlans depending on which "server" address I'd choose (and the speeds suggested that they did).

Anyway, I've now disabled IPv4 in the properties of the physical NIC of the test laptop and checked that it is now only assigned the 192.168.90.x address. Then I repeated the tests to 192.168.90.11 (no real change there as far as I can tell) and subsequently to 192.168.1.1 (which in this case became a cross vlan scenario).

Results are below:
vlan90 to vlan90 ( IPv4 disabled on "untagged NIC")

Audience2_Ether2_vlan90_Disabled_vlan10.png

vlan90 to vlan10 ( IPv4 disabled on "untagged NIC")

Audience2_Ether2_vlan10_Disabled_vlan10.png

Also, while during the first test the CPU on hex PoE remained between 5-9% during the second test it was regularly jumping to 30% indicating that in 2nd scenario the router had more work to do.

ias · Mon Dec 30, 2024 2:33 pm

Sorry, I only now understood that you wanted me to connect the server PC to the ether2 on audience, while I misunderstood and connected the test laptop (client there). I'll repeat the tests in the scenario you suggested i.e. "server PC" connected to the audience ether2 and test laptop (client) connected to audience over WIFI. Give me a moment.

sindy · Mon Dec 30, 2024 2:45 pm

In my above test both: the PC acting as the Tamosoft server (192.168.1.11 / 192.168.90.11) and the test laptop acting as the Tamosoft client (192.168.1.x / 192.168.90.x) were connected to hybrid ports and got IPs from both network on their physical and virtual NICs. I assumed that the tests in this scenario would go through respective vlans depending on which "server" address I'd choose (and the speeds suggested that they did).

Sorry, I only now understood that you wanted me to connect the server PC to the ether2 on audience, while I misunderstood and connected the test laptop (client there).

Yes, you have actually tested something else than I've thought I've asked you to test. Initially, you had a wireless client and a server connected to ether2 of hEX Gr3. What I wanted you to do was to only move the server from ether2 of hEX Gr3 to ether2 of the Audience to which the wireless client was connected (and obviously test again using the wireless client, why would I mention the local forwarding otherwise); instead, you've poked another PC with a wired port from your magician's hat and tested between two Ethernet ports. As I haven't expected that, I also haven't expected that the client could have an interface in both subnets. As it did, one test was indeed 10 to 10 and the other one was 90 to 90, no routing involved and hence no path via hEX PoE involved in either direction.

But OK, the ethernet to ethernet testing has shown us that even though vlan filtering is enabled on both the hEX Gr3 and the Audience, the hEX-ether2-to-Audience-ether2 L2-only test can reach wire speeds even when tagging and untagging VLAN 10, so there is nothing wrong with the VLAN setup, at least on the path between those two.

Now with the second round of tests when the path via hEX PoE was involved, the speed has dropped to ~100 Mbit/s again for most measurements. So let's see what the wireless-to-Ethernet test shows on the Audience alone and we'll see where to run next.

ias · Mon Dec 30, 2024 3:21 pm

Indeed you got precisely right, what I got wrong

I've basically performed a number of Ethernet tests in different scenarios. Sorry for the confusion.

So coming back to wireless; below is the missing test where the PC running the Tamosoft server is connected to the ether2 on hex and test laptop is connected to Audience over WIFI (connection properties showed connection over 5GHz 866/866 Mbps).

SerHexEther2_vlan10_CliWIFI_vlan10.png

Below is the test performed with the "server PC" connected to the ether2 on Audience and test laptop is still connected to Audience over WIFI.

SerAudienceEther2_vlan10_CliWIFI_vlan10.png

There is only marginal difference in the average speeds in these two scenarios as far as I can see.

However, these speeds are still much higher than an external speedtest from the test laptop which is still below 100/100 Mbps (compared to ~600/600 Mbps over wired network).

SpeedtestWIFI.png

What should I target next?

sindy · Mon Dec 30, 2024 3:55 pm

OK, these tests seem much more realistic to me; even though the difference is not big, it is surprising that the throughput is better when the server is connected to hEX Gr3 than when it is connected to Audience, but that may be related to interference unless you live in the countryside without any neighbors. Does /caps-man interface scan interface-name spawned on hEX PoE show any 5 GHz networks around except your other APs?

Now I would check that the link between hEX PoE and hEX Gr3 has established on 1 Gbit/s full duplex (again using /interface ethernet monitor ether1 once on hEX Gr3). If yes, I'd check whether any of the Ethernet ports of both the hEX Gr3 and hEX PoE is not linked at 100 Mbit/s - it's a bit esoteric but that's for later. If it is, I'd disable it if possible and try the speedtest via internet again and also a test from a wireless client to a server connected to ether2 of hEX PoE, both within VLAN 10 and when only the address from the .90. subnet is up on the server (so that we could compare the "bridging" and "routing" performance).

ias · Mon Dec 30, 2024 5:16 pm

First of all you were right in searching for any 100 Mbps ports. I was pretty certain I won't find anything as I checked the negotiated port speeds before, but there is was: the connection between hex (ether1) and hex poe (ether 3) was showing 100 Mbps

I disconnected it and connected again and it's now showing 1 Gbps.

/interface ethernet monitor ether1
                      name: ether1
                    status: link-ok
          auto-negotiation: done
                      rate: 1Gbps
               full-duplex: yes
           tx-flow-control: no
           rx-flow-control: no
                 supported: 10M-baseT-half,10M-baseT-full,100M-baseT-half,
                            100M-baseT-full,1G-baseT-half,1G-baseT-full
               advertising: 10M-baseT-half,10M-baseT-full,100M-baseT-half,
                            100M-baseT-full,1G-baseT-half,1G-baseT-full
  link-partner-advertising: 10M-baseT-half,10M-baseT-full,100M-baseT-half,
                            100M-baseT-full,1G-baseT-half,1G-baseT-full

With that change the results of the external speediest improved however the results of the test of internal network speed seems to be slightly lower than in the previous round (maybe it just shows its variability).

Tamosoft test (test laptop connected to Audience via WIFI, "server PC" connected to Audience on ether2):

SerAudienceEther2_vlan10_CliWIFI2_vlan10_2.png

Speedtest over WIFI:

Speedtest_WIFI2.png

Having said that it's still quite far from the wired external speed test, And I'm not sure if that's the best I can expect from 5GHz ac network? Or if I should be worried about all that pocket loss?

For reference, these are the results of the wired speedtest:

Speedtest_Wired.png

I'll now proceed with the below step to check how it compares with the tests to the server connected to Audience:

and also a test from a wireless client to a server connected to ether2 of hEX PoE

ias · Mon Dec 30, 2024 8:31 pm

OK, so the test with the Tamosoft server connected to hex poe ether2 left me somewhat surprised. I don't know what I expected, but I guess I expected that the results will be worse, or for some mysterious reason they will be better. I surely didn't expect the mix. I performed the test twice with lot of time in between to make sure it wasn't some momentary, non-replicable condition but both tests concluded with very similar results.

So here is the test where the Tamosoft server is connected to hex poe ether2 and the test laptop is still connected via WIFI to the same Audience as in previous tests (this means that there is now Audience, hex and hex poe between the client and the server):

SerHex1Ether2_vlan10_CliWIFI2_vlan10_2.png

What I find strange is that average TCP speeds are significantly lower here than in the previous test (where server was connected directly to ether2 on Audience) the UDP Down speed is slightly lower (but comparable) but the UDP Up average speed is significantly higher than in the previous test. While I don't quite understand this, I think it indicates that the wireless connection is not the main influencing factor here as in both tests the client is connected to the same AP, is in the same distance from the AP (less than 1m actually) and is subject to the same interference.

Anyway, thanks sindy for your insights so far. You already identified the reason behind the sub 100 Mbps speeds, and since then I've already noticed this issue twice: first on the link between hex poe and hex, and later on the link between hex poe and other Audience (to be clear I checked that link after your initial post and it was 1 Gbps but later I found it reverted to 100 Mbps). I'll monitor this some more to check if it's something that happens regularly on my network.

mkx · Tue Dec 31, 2024 7:11 pm

On networks with any kind of problems (real or perceived, e.g. packet loss or large RTT), TCP and UDP connections behave very much differently.

For UDP connections packet drop is fine and stats show high throughput with high packet loss. Usually throughput shown on Tx side reflects throughput of first link leg ... Tx stack does wait for Tx buffers to flush properly ... if there's a bottleneck on link legs beyond the first one, then the nearer device has to drop packets which overflow the buffer. So receiving end shows thtoughput, achieved by narrowest bottleneck on the whole link.

For TCP connections packet drop is not an option, there are mechanizms which ensure packet delivery (retransmissions) and to avoid excessive triggering of those mechanizms, there's Tx window size which effectively reduces rate of transmission (trying to reduce throughput below the packet drop threshold). Since they're driven by feedback from receiver, these mechanizms reflect conditions of the whole link between transmitter and receiver (all legs) and sometines also the reverse direction (which can be even more important in case of very asymmetrical link legs or half-duplex link legs). Due to these (and a few other) reasons it's very hard to achieve wire (or "wire" in case of wifi) speed using TCP connections ... even on a perfect link one can expect to achieve results a few percent below full capacity.

I don't know what exactly the results shown on screenshot are telling. I'm more used to iperf3 (I guess tamosoft adds a fancy GUI to iperf3 CLI and tries to make dispkay of results nicer, but along the way some important stuff gets hidden). Running iperf3 it's possible to directly compare results from both ends (server and client) which allows to diagnose point of the probkem a bit easier. E.g. it's not clear what "UDP up:" resukt means ... is it end-to-end result, reported by receiver (server)? Or is it report from sender (client) ... which then shows thtoughput of first link leg (wifi) which in turn means that there's another bottleneck further beyond wifi which causes packet drop?
Iperf3 also allows to set Tx rate for UDP ... setting it to values below expected bottleneck bandwidth allows to verify that packet drop is indeed caused by low bandwidth of a bottleneck rather than poor link leg quality (e.g. faulty UTP cable or noisy/congested wifi connection) which would drop random packets regardless the data rate.

Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Re: Troubles with performance of CAPsMAN-managed WIFI on RoS 7.16.2 with vlans

Who is online