Mikrotik hAP ax3 - slow download speed through wired connection

trmns · Tue Mar 26, 2024 7:46 pm

I had already posted about my issue on Reddit, but I am also trying my luck here, hoping someone knows whats up and can help.
I am using a hAP ax3 and I noticed that some types of connections are behaving irrationally. (Everything which isn't TLS it seems)
I have a 1gE fiber connection and in online speedtests I achieve those speeds: I just ran three tests while writing this post and I get

fast.com: [810 Mbps, 880 Mbps]
speedtest.net: [911 Mbps, 924 Mbps]
speed.cloudflare.com: [877 Mbps, 889 Mbps]

The same holds true for game updates/downloads through e.g. Steam. The issue I have is that on non-TLS connection (but still TCP), these speeds are not achievable.
If I test the performance with iperf or through SSH (scp), things look very differently. The upload performs somewhat normally as can be seen below
(iperf3 -c <host>):

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  95.1 MBytes   797 Mbits/sec    0   3.94 MBytes
[  5]   1.00-2.00   sec  98.2 MBytes   824 Mbits/sec    0   3.94 MBytes
[  5]   2.00-3.00   sec   102 MBytes   852 Mbits/sec    0   3.94 MBytes
[  5]   3.00-4.00   sec  98.6 MBytes   827 Mbits/sec    0   4.14 MBytes
[  5]   4.00-5.00   sec   111 MBytes   929 Mbits/sec    0   4.14 MBytes
[  5]   5.00-6.00   sec   109 MBytes   918 Mbits/sec    0   4.14 MBytes
[  5]   6.00-7.00   sec   110 MBytes   927 Mbits/sec    0   4.14 MBytes
[  5]   7.00-8.00   sec   105 MBytes   883 Mbits/sec    0   4.14 MBytes
[  5]   8.00-9.00   sec   112 MBytes   935 Mbits/sec    0   4.14 MBytes
[  5]   9.00-10.00  sec   112 MBytes   934 Mbits/sec    0   4.14 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.03 GBytes   883 Mbits/sec    0             sender
[  5]   0.00-10.01  sec  1.03 GBytes   880 Mbits/sec                  receiver

As you can see things totally change when I Download (iperf3 -c <host> -R)

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  19.4 MBytes   162 Mbits/sec
[  5]   1.00-2.00   sec  25.5 MBytes   214 Mbits/sec
[  5]   2.00-3.00   sec  18.2 MBytes   153 Mbits/sec
[  5]   3.00-4.00   sec  15.4 MBytes   129 Mbits/sec
[  5]   4.00-5.00   sec  23.6 MBytes   198 Mbits/sec
[  5]   5.00-6.00   sec  24.9 MBytes   209 Mbits/sec
[  5]   6.00-7.00   sec  24.6 MBytes   207 Mbits/sec
[  5]   7.00-8.00   sec  21.8 MBytes   182 Mbits/sec
[  5]   8.00-9.00   sec  24.0 MBytes   201 Mbits/sec
[  5]   9.00-10.00  sec  25.1 MBytes   211 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   226 MBytes   189 Mbits/sec  465             sender
[  5]   0.00-10.00  sec   222 MBytes   187 Mbits/sec                  receiver

The same happens when I use scp to transfer a file from that host to my machine. My ping to the machine is between 6 and 8ms

scp host:tmp/2g_random .
2g_random              22%  455MB  37.2MB/s   00:42 ETA

If I upload via SCP, everything is fine (or at least good)

$ scp 2cg_random host:tmp/
2cg_random                    41%  859MB 100.5MB/s   00:11 ETA

Of course one could say that the host does not support the bandwidth needed, but the host has a 10gbit fiber connection and when I download a test file through HTTPS speeds are nice:

wget https://host/2g_random
2g_random                         11%[====>            ]   220M   108MB/s

I tried the latest stable OS + firmware and also the latest development version + firmware. It would be nice if anyone has an idea how to debug or fix this.

My config:

# 2024-03-26 21:08:12 by RouterOS 7.15beta8
# software id = S3S2-FI0P
#
# model = C53UiG+5HPaxD2HPaxD
/interface bridge
add admin-mac=48:A9:8A:B8:BC:9A auto-mac=no comment=defconf name=bridge port-cost-mode=short
/interface ethernet
set [ find default-name=ether1 ] mac-address=64:D1:54:6D:6E:40
/interface wifi
set [ find default-name=wifi1 ] channel.band=5ghz-ax .frequency=5180-5480 .skip-dfs-channels=disabled .width=20/40/80mhz configuration.antenna-gain=5 .country=Netherlands .mode=ap .ssid="5ghz" .tx-power=20 disabled=no \
    security.authentication-types=wpa2-psk,wpa3-psk
set [ find default-name=wifi2 ] channel.band=2ghz-ax .frequency=2412-2472 .skip-dfs-channels=disabled .width=20/40mhz configuration.antenna-gain=3 .country=Netherlands .mode=ap .ssid="2.4ghz" .tx-power=13 disabled=no \
    security.authentication-types=wpa2-psk,wpa3-psk
/interface wireguard
add listen-port=13231 mtu=1420 name=wg0
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=dhcp interface=bridge lease-time=10m name=defconf
/ip smb users
set [ find default=yes ] disabled=yes
/interface bridge port
add bridge=bridge comment=defconf interface=ether2 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether3 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether4 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=ether5 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=wifi1 internal-path-cost=10 path-cost=10
add bridge=bridge comment=defconf interface=wifi2 internal-path-cost=10 path-cost=10
/ip firewall connection tracking
set udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=all
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/interface wireguard peers
add allowed-address=10.0.0.4/32 comment=peer1 interface=wg0 name=peer2 public-key="1"
add allowed-address=10.0.0.3/32 comment=peer2 interface=wg0 name=peer3 public-key="2"
add allowed-address=10.0.0.2/32 comment=peer3 interface=wg0 name=peer4 public-key="3"
add allowed-address=10.0.0.5/32 comment=peer4 interface=wg0 name=peer5 public-key="4"
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0
add address=10.0.0.1/24 comment=wireguard interface=wg0 network=10.0.0.0
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward disabled=yes nth=4,1 protocol=udp
add action=drop chain=forward disabled=yes nth=4,3 protocol=udp
add action=drop chain=forward disabled=yes nth=4,2 protocol=udp
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input comment="accept WireGuard connections" dst-port=13231 protocol=udp
add action=accept chain=input comment="allow access to RouterOS's DNS server via WireGuard" dst-port=53 in-interface=wg0 protocol=udp
add action=accept chain=input comment="allow access to RouterOS mobile app via WireGuard" dst-port=8291 in-interface=wg0 protocol=tcp
add action=accept chain=input comment="allow access to RouterOS webinterface via WireGuard" dst-port=80 in-interface=wg0 protocol=tcp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat comment=https://help.mikrotik.com/docs/display/ROS/NAT#NAT-HairpinNAT dst-address=192.168.88.0/24 dst-port=22 out-interface-list=LAN protocol=tcp src-address=192.168.88.0/24
add action=dst-nat chain=dstnat comment="This rule changes the IP of all packets which are from the external network and directed to tcp:<external_ip>:22 to tcp:192.168.88.124:22" dst-port=22 in-interface-list=WAN protocol=tcp \
    to-addresses=192.168.88.124 to-ports=22
add action=dst-nat chain=dstnat comment="This rule changes the IP of all packets which are from the internal network and directed to tcp:<external_ip>:22 to tcp:192.168.88.124:22" dst-address=<external_ip> dst-port=22 \
    in-interface-list=LAN protocol=tcp to-addresses=192.168.88.124 to-ports=22
add action=dst-nat chain=dstnat comment="This rule changes the IP of all packets which are from the internal network and directed to tcp:<external_ip>:2222 to tcp:192.168.88.60:22" dst-address=<external_ip> dst-port=2222 \
    in-interface-list=LAN protocol=tcp to-addresses=192.168.88.23 to-ports=22
add action=dst-nat chain=dstnat comment="This rule changes the IP of all packets which are from the external network and directed to tcp:<external_ip>:2222 to tcp:192.168.88.60:22" dst-port=2222 in-interface-list=WAN protocol=tcp \
    to-addresses=192.168.88.23 to-ports=22
add action=dst-nat chain=dstnat disabled=yes dst-port=5201 in-interface-list=WAN protocol=tcp to-addresses=192.168.88.124 to-ports=5201
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api-ssl disabled=yes
/ip smb shares
set [ find default=yes ] directory=/pub
/ip ssh
set host-key-size=4096 host-key-type=ed25519 strong-crypto=yes
/ipv6 address
add from-pool=v6-pool interface=bridge
/ipv6 dhcp-client
add add-default-route=yes interface=ether1 pool-name=v6-pool pool-prefix-length=56 rapid-commit=no request=address,prefix use-peer-dns=no
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN

Skibbi · Mon Oct 28, 2024 7:30 pm

I'm fighting with pretty similar issue. My hAP ac (ROS 7.16.1) cannot download more than 10-15MB/s scp on my 300Mbit/s link. Of course directly linking my computer to ISP modem solves the issue.
Have you found a solution?

anav · Mon Oct 28, 2024 9:40 pm

Your config looks pretty basic, so these are try it just in case, or normal things to do.

1. Change this to NONE
/interface detect-internet
set detect-interface-list=NONE

2. I noted that this config line is in red? Lets modify it.
From:
add action=masquerade chain=srcnat comment=https://help.mikrotik.com/docs/display/ ... HairpinNAT dst-address=192.168.88.0/24 dst-port=22 out-interface-list=LAN protocol=tcp src-address=192.168.88.0/24

TO:
add action=masquerade chain=srcnat comment="Hairpin NAT" dst-address=192.168.88.0/24 src-address=192.168.88.0/24

3. It would appear you are playing some silly games with respect to servers on the LAN network. Please explain as they make no sense to me.
Explain it in term of user traffic flow. To be clear I am talking about your dst nat rules!!
How do user get to server from external and internal ............

4. firewall rules are a bit clunky.
Take these and simplify.. and keep in mind, why should the whole lan have access either !!!

From:
add action=accept chain=input comment="allow access to RouterOS's DNS server via WireGuard" dst-port=53 in-interface=wg0 protocol=udp
add action=accept chain=input comment="allow access to RouterOS mobile app via WireGuard" dst-port=8291 in-interface=wg0 protocol=tcp
add action=accept chain=input comment="allow access to RouterOS webinterface via WireGuard" dst-port=80 in-interface=wg0 protocol=tcp

TO:
/ip firewall
add action=accept chain=input comment="admin access" src-address-list=Authorized

/ip firewall address-list ( static set dhcp leases where applicable )
add address=10.0.0.2 list=Authorized comment="admin laptop"
add address=10.0.0.3 list=Authorized comment="admin smartphone/ipad"
add address=192.168.88.X list=Authorized comment="admin PC on local network"
add address=192.168.88.YY list=Authorized comment=" admin wifi on local network"

Note I didnt include wireguard peers 10.0.0.4 or 10.0.0.5 as I assumed maybe they were simply users that need access to the LAN but are not admin so dont need access to config router.

++++++++++++++++++++++++++++
Thus your input chain would look like:

/ip firewall filter
{default rules to keep}
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
{admin rules}
add action=accept chain=input comment="accept WireGuard connections" dst-port=13231 protocol=udp
add action=accept chain=input comment="admin access" src-address-list=Authorized
add action=accept chain=input comment="users to services" in-interface-list=LAN dst-port=53 protocol=udp
add action=accept chain=input comment="users to services" in-interface-list=LAN dst-port=53 protocol=tcp
add action=drop chain=input comment="drop all else" { put this in last or you will lock yourself out }

infabo · Mon Oct 28, 2024 9:51 pm

use /tools/profile when under load and bad throughput. I guess there is something running over cpu.

Skibbi · Tue Oct 29, 2024 11:07 am

use /tools/profile when under load and bad throughput. I guess there is something running over cpu.

In my case CPU tops at 20% max during download. Also connections are fasttracked.
I've also installed pfSense on minipc and I get better download speeds ~20MB/s on scp and 37MB/s via http(s). So definitely there is something in Mikrotik that throttles the speeds.

mkx · Tue Oct 29, 2024 11:20 am

use /tools/profile when under load and bad throughput. I guess there is something running over cpu.
In my case CPU tops at 20% max during download.

Is that CPU load on single CPU or average? When doing scp (or single threaded iperf), there's a single connection and by design, all packets belonging to same connection are processed by single CPU core (and hAP ax3 has 4 CPU cores). So if you see a single CPU core being highly utilized, then that's the reason. You can verify that this is indeed bottleneck by running iperf with option "-P 4", which will cause iperf to use 4 parallel streams (and each will, hopefully, peg different CPU core on your hAP ax3).

Skibbi · Tue Oct 29, 2024 4:55 pm

In my case CPU tops at 20% max during download.
Is that CPU load on single CPU or average? When doing scp (or single threaded iperf), there's a single connection and by design, all packets belonging to same connection are processed by single CPU core (and hAP ax3 has 4 CPU cores). So if you see a single CPU core being highly utilized, then that's the reason. You can verify that this is indeed bottleneck by running iperf with option "-P 4", which will cause iperf to use 4 parallel streams (and each will, hopefully, peg different CPU core on your hAP ax3).

I have hAP ac but I have the same issue as OP. And hAP ac has one CPU core only.

Mikrotik hAP ax3 - slow download speed through wired connection

Mikrotik hAP ax3 - slow download speed through wired connection

Re: Mikrotik hAP ax3 - slow download speed through wired connection

Re: Mikrotik hAP ax3 - slow download speed through wired connection

Re: Mikrotik hAP ax3 - slow download speed through wired connection

Re: Mikrotik hAP ax3 - slow download speed through wired connection

Re: Mikrotik hAP ax3 - slow download speed through wired connection

Re: Mikrotik hAP ax3 - slow download speed through wired connection

Who is online