Community discussions

MikroTik App
  4. RouterOS
  5. General

Very Slow output for traffic passing through CHR

Post Reply
 
Alupis
just joined
Topic Author
Posts: 23
Joined: Wed Feb 29, 2012 6:30 pm

Very Slow output for traffic passing through CHR

Fri Apr 22, 2022 2:41 am

UPDATE IN BELOW POST - No longer think this is a GRE Tunnel issue

Basic Setup:

CHR P10 licensed & activated, hosted on Vultr cloud hosting provider running 7.2 Stable, 1000Mbps/1000Mbps
CCR hardware running 6.48.6 Long Term, 1000Mbps / 45Mbps

Topology:

Office Workstation (Private: 10.0.0.5/27)
|
V
CCR (Public: 1.1.1.1/29, Private: 10.0.0.1/27, Tunnel: 172.16.1.1/30)
|
V
TUNNEL (GRE, EOIP, or IPIP with IPsec)
|
V
CHR (DHCP assigned fake-static Public: 2.2.2.2/23, Private: 10.1.0.1/27, Tunnel: 172.16.1.2/30)
|
V
Virtual Server (Private: 10.1.0.5)

I initially created a GRE tunnel with IPsec Secret between the two endpoints. The Tunnel connects, and IPsec Active Peers show up as expected. After adding Routes, I can pass traffic as expected. Everything looked good until I attempted to pass actual workloads through the tunnel... and discovered they are extremely slow, less than 1Mbps and often in the Kbps range.

I disabled IPsec Secret and tried just a plain tunnel, same thing.

I tried the same configuration, but used EOIP and IPIP instead. Same thing with both.

If I do a BTest between both mikrotiks, I can achieve reasonable throughput (300-700Mbps+ down, depending if IPsec is enabled or not).

If I bypass the CHR and talk directly to a public IP on the cloud Virtual Server, I can achieve maximum throughput as well, which would indicate it's not the virtual server, perhaps.

The problem only occurs when passing traffic through the CHR either direction.

Sniffing the connection, I see a lot of retransmissions and duplicate packets - unsure if that is the problem or standard for a tunnel like this. Seems suspicious though.

I've attached cleaned/trimmed exports from each side. I tried to remove things that were obviously unrelated, such as our "road warrior" IKEv2 configs. If something doesn't make sense, let me know and I'll check the config.

CHR:
Code: Select all
# apr/21/2022 17:25:19 by RouterOS 7.2
# software id = 
#
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
set [ find default-name=ether2 ] comment=\
    "MTU 1450 per Vultr VPC Documentation" disable-running-check=no mtu=1450
/interface gre
add allow-fast-path=no mtu=1426 name=gre-tunnel2 remote-address=1.1.1.1
/disk
set sata1 disabled=no
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec profile
set [ find default=yes ] dh-group=modp1024 enc-algorithm=aes-128
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc lifetime=8h pfs-group=\
    modp4096
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=gre-tunnel2 list=LAN
add list=LAN
/interface ovpn-server server
set auth=sha1,md5
/ip address
add address=10.1.0.1/27 interface=ether2 network=10.1.0.0
add address=172.16.1.2/30 interface=gre-tunnel2 network=172.16.1.0
/ip dhcp-client
add interface=ether1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=10.1.0.4 name=db.example.net
add address=10.1.0.5 name=cloud.example.net
/ip firewall address-list
add address=1.1.1.1/29 list=office
add address=10.0.0.1/27 list=office
add address=172.16.1.0/30 list=gre_tunnel
/ip firewall filter
add action=accept chain=input comment="Accept all GRE Tunnel Traffic" \
    protocol=gre src-address-list=office
add action=accept chain=input comment="Accept all GRE Tunnel Traffic" \
    dst-address-list=gre_tunnel protocol=gre
add action=accept chain=input comment=\
    "VPN IPSec Encapsulating Security Payload (ESP)" protocol=ipsec-esp
add action=accept chain=input comment=\
    "VPN IPSec Encapsulating Security Payload (AH)" protocol=ipsec-ah
add action=accept chain=input comment="Allow ICMP" protocol=icmp
add action=accept chain=input comment=\
    "Allow LAN to access Services (DNS, etc)" in-interface-list=LAN
add action=accept chain=input comment="Allow Established & Related" \
    connection-state=established,related
add action=accept chain=input comment="Allow All Traffic from Office" \
    in-interface-list=WAN src-address-list=office
add action=drop chain=input comment="Drop Everything Else" log=yes \
    log-prefix=DROP
add action=accept chain=forward comment="Allow All Traffic from Office" \
    out-interface-list=LAN src-address-list=office
add action=accept chain=forward comment="Allow All Traffic to Office" \
    dst-address-list=office out-interface-list=WAN
add action=accept chain=forward comment="Allow Established & Related" \
    connection-state=established,related
add action=accept chain=forward comment="Allow LAN to Talk to Each Other" \
    connection-state=new in-interface-list=LAN
add action=drop chain=forward comment="Drop Everything Else" log=yes \
    log-prefix=DROP
/ip firewall nat
add action=masquerade chain=srcnat comment="Default Masquerade" \
    out-interface-list=WAN
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
/ip ipsec policy
set 0 disabled=yes
/ip route
add disabled=no distance=1 dst-address=10.0.0.1/27 gateway=gre-tunnel2 \
    pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=America/Los_Angeles
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.us.pool.ntp.org
add address=1.us.pool.ntp.org
add address=2.us.pool.ntp.org
/tool sniffer
set file-limit=5000KiB file-name=CAPTURE2.cap memory-limit=2000KiB
CCR:
Code: Select all
# apr/21/2022 17:26:04 by RouterOS 6.48.6
# software id = 
#
# model = CCR1009-8G-1S-1S+
# serial number = 
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1588 speed=100Mbps
set [ find default-name=ether2 ] disabled=yes l2mtu=1588 speed=100Mbps
set [ find default-name=ether3 ] disabled=yes l2mtu=1588 speed=100Mbps
set [ find default-name=ether4 ] disabled=yes l2mtu=1588 speed=100Mbps
set [ find default-name=ether5 ] disabled=yes l2mtu=1590 speed=100Mbps
set [ find default-name=ether6 ] disabled=yes l2mtu=1590 speed=100Mbps
set [ find default-name=ether7 ] disabled=yes l2mtu=1590 speed=100Mbps
set [ find default-name=ether8 ] l2mtu=1590 speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] advertise=\
    10M-full,100M-full,1000M-full l2mtu=1590
set [ find default-name=sfp1 ] advertise=10M-full,100M-full,1000M-full \
    disabled=yes l2mtu=1590
/interface gre
add allow-fast-path=no mtu=1426 name=gre-tunnel2 remote-address=\
    2.2.2.2
/interface list
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=CCR1
/ip ipsec profile
set [ find default=yes ] dh-group=modp1024 enc-algorithm=aes-128
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc lifetime=8h pfs-group=\
    modp4096
/ip pool
add name=dhcp_pool1 ranges=10.0.0.2-10.0.0.30
/ip dhcp-server
add address-pool=dhcp_pool1 authoritative=after-2sec-delay interface=sfp-sfpplus1 \
    name=dhcp1
/interface list member
add interface=sfp-sfpplus1 list=LAN
/ip address
add address=1.1.1.1/29 interface=ether1 network=1.1.1.0
add address=10.0.0.1/27 interface=sfp-sfpplus1 network=10.0.0.0
add address=172.16.1.1/30 interface=gre-tunnel2 network=172.16.1.0
/ip dhcp-server network
add address=10.0.0.0/24 dns-server=10.0.0.1 gateway=\
    10.0.0.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip firewall address-list
add address=10.0.0.0/24 comment="Our LAN Subnet" list=LAN
add address=2.2.2.2 list=CLOUD-SERVERS
add address=10.1.0.0/27 list=CLOUD-SERVERS
add address=172.16.1.0/30 list=GRE-TUNNEL
/ip firewall filter
add action=accept chain=input comment="Accept all GRE Tunnel Traffic" \
    protocol=gre src-address-list=CLOUD-SERVERS
add action=accept chain=input comment="Accept all GRE Tunnel Traffic" \
    dst-address-list=GRE-TUNNEL protocol=gre
add action=accept chain=input comment="Established & Related Connections" \
    connection-state=established,related
add action=accept chain=input comment="Allow External Ping" protocol=icmp
add action=drop chain=input comment="Drop Invalid Connections" \
    connection-state=invalid log=yes log-prefix=INVALID
add action=accept chain=input comment=\
    "VPN IPSec Encapsulating Security Payload (ESP)" log-prefix=IPSEC- \
    protocol=ipsec-esp
add action=accept chain=input comment=\
    "VPN IPSec Encapsulating Security Payload (AH)" log-prefix=IPSEC- \
    protocol=ipsec-ah
add action=accept chain=input comment="VPN IPSec Internet Key Exchange (IKE)" \
    dst-port=500 log-prefix=IPSEC- protocol=udp
add action=accept chain=input comment="VPN IPSec NAT Traversal (NAT-T)" \
    dst-port=4500 log-prefix=IPSEC- protocol=udp
add action=accept chain=input comment="Allow LAN => Router" src-address-list=\
    LAN
add action=drop chain=input comment="Drop Everything Else" log=yes \
    log-prefix=DROP
add action=accept chain=forward comment="Established & Related Connections" \
    connection-state=established,related
add action=drop chain=forward comment="Drop Invalid Connections" \
    connection-state=invalid log=yes log-prefix=INVALID
/ip firewall nat
add action=masquerade chain=srcnat comment=\
    "Default NAT Masquerade: LAN --> WAN" out-interface=ether1
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip ipsec policy
set 0 disabled=yes
/ip route
add distance=1 gateway=1.1.1.0
add distance=1 dst-address=10.1.0.0/27 gateway=gre-tunnel2
Last edited by Alupis on Sat Apr 23, 2022 12:53 am, edited 1 time in total.
Top
 
Alupis
just joined
Topic Author
Posts: 23
Joined: Wed Feb 29, 2012 6:30 pm

Re: Very Slow CHR -> CCR GRE/IPIP/EOIP Tunnel

Sat Apr 23, 2022 12:39 am

I think I've narrowed the problem down to traffic going from any of the cloud servers through the CHR. So output from the CHR's perspective.

A MTR in both directions shows zero packet loss:

Image

Cloud Server to Cloud Server throughput is great, measured with iperf:

Image

And Office upload to Cloud Server is fine, however... Cloud Server upload to Office is terrible:

Image

I tried another Cloud Server in the Virtual Private Network (VPC in Vultr terms) and got the same results:

Image

I even tried running Speedtest.net from the Cloud Server and got similar results:

Image

A BTest from the CHR to the CCR looks reasonable for being a IPsec inside a tunnel, which means it's traffic passing through the CHR that is the issue, not traffic originating from the CHR:

Image

What is going on here? I can't seem to think of any configs in the CHR that would throttle things like this. The servers are all Rocky Linux 8.x, with a mostly vanilla setup. If it was a networking issue on the servers I would expect to see the same problems with all traffic, not just traffic going outbound through the CHR.

This makes me think this is not a GRE tunnel issue... I'm not sure what to make of this.
Top
 
mada3k
Forum Veteran
Forum Veteran
Posts: 751
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: Very Slow output for traffic passing through CHR

Sat Apr 23, 2022 8:17 pm

Hmm... MTU or fragmentation issue?

Just start count the overhead backwards from the MTU used by your ISP
Top
 
Alupis
just joined
Topic Author
Posts: 23
Joined: Wed Feb 29, 2012 6:30 pm

Re: Very Slow output for traffic passing through CHR

Tue Apr 26, 2022 1:31 am

Hmm... MTU or fragmentation issue?

Just start count the overhead backwards from the MTU used by your ISP
Thanks. I had initially suspected a possible MTU issue.

According to Vultr, all interfaces on the VPC network must bet set to 1450 MTU.

https://www.vultr.com/docs/how-to-creat ... cloud-vpc/

When initially creating the VM, if you attach it to a VPC network is auto-sets the MTU on the correct interface.

So this means,

CHR:
ether1 WAN - MTU 1500
ether2 LAN - MTU 1450
GRE-Tunnel - MTU 1426 (guessed here, matches CCR side as well)

VM:
ether1 DISABLED
ether2 LAN - MTU 1450

For traffic that is VM -> CHR -> Public, it should go MTU1450 -> ( MTU1450 -> BUFFER -> MTU1500 ) -> Public, where everything between the parentheses are inside the CHR.

From one of the cloud VM's (interface MTU 1450), doing a
Code: Select all
ping 4.2.2.2 -M do -s 1422
returns fine, but using a packet size of 1423 errors out and needs to fragment. This leaves overhead of about 28 bytes for a VM -> Public packet.

I also tried re-enabling one of the cloud VM's public interfaces (not behind the CHR, attached directly to the public) and a speedtest shows just shy of 5Gbps in both directions:

Image

So it seems to entirely be related to traffic going through the CHR. VM to VM traffic is good (about 2Gbps). VM to public traffic is good (about 5Gbps). VM traffic through CHR is bad (less than 1Mbps).

Not sure where this leaves me...
Top
 
darksoul
just joined
Posts: 2
Joined: Fri Dec 25, 2015 5:42 am

Re: Very Slow output for traffic passing through CHR

Sun Jul 24, 2022 4:16 am

Hi,

I ran into the same issue but with slow upload speeds from my VPC nodes to the outside via mikrotik.

I had reached out their tech support, we ran some tests and everything had point out to the router.

I am sure it's not 100% mikrotik issue, but it seems to me that it's a combination of the way how VPC is organized.

Had anyone had success on this problem?

Thanks.
Top
 
MikeToTheP
just joined
Posts: 1
Joined: Fri Sep 30, 2022 5:51 pm

Re: Very Slow output for traffic passing through CHR

Fri Sep 30, 2022 5:53 pm

Did you ever resolve this?

I'm having similar issues and looking for answers.
Top
 
Underdogs
just joined
Posts: 1
Joined: Thu Dec 01, 2022 3:00 am

Re: Very Slow output for traffic passing through CHR

Thu Dec 01, 2022 3:07 am

I'm having a similar problem in a Vultr environment. The virtual Mikrotik CHR (running 7.6) can upload/download over 500mb/s on the WAN port. The cloud computers connected to it on the virtual LAN can download over 500mb/s, but only upload 7-10mb/s. And even uploading to the Mikrotik directly from the cloud computers is constrained by this speed.

I will note that the problem was initially much worse, with uploads in the 250kb/s range. What helped was reinstalling the cloud Mikrotik CHR on an Intel platform vs. AMD. But the upload of 7-10mb/s is still awful. Tried many MTU settings with no change. Currently have 1450 set on the Mikrotik LAN, as recommended. On the cloud computers (running Windows) I've got the MTU set to 1422 which is optimal to avoid fragmentation.

I have another Vultr account with a nearly identical setup and on that one the uploads work fine. Anyone else run into this?
Top
 
luddite
just joined
Posts: 22
Joined: Fri Apr 06, 2012 12:09 am

Re: Very Slow output for traffic passing through CHR

Thu Dec 29, 2022 10:03 am

Same same, just found this post, after umpteen MTU adjustments and iperf tests between Windows and Linux hosts over the Vultr VPC I can only get 300-600Mbps - Mbps!
(tried lots of mtu values from 1300 up to 1450)

Like poster above said if enabling NIC connecting directly to internet get 4Gbps from Speedtest.net.

Going to give up on Vultr I think.
Top
 
User avatar
rilliam
newbie
Posts: 48
Joined: Thu Mar 12, 2009 7:34 pm

Re: Very Slow output for traffic passing through CHR

Tue Mar 26, 2024 3:00 am

I'm having same issue with vultr +mtik +server 2022. Upload speed through mtik on server 2022 is 2mbit/s max. Otherwise everything is fine.

Anybody figure this out?
Top
 
User avatar
honeyfairy
newbie
Posts: 43
Joined: Sat Nov 21, 2020 1:25 am
Contact:
Contact honeyfairy

Re: Very Slow output for traffic passing through CHR

Fri Aug 02, 2024 10:59 pm

Same issue here. Vultr support is of no help. Anybody found a solution?
Top
 
UpRunTech
Member Candidate
Member Candidate
Posts: 238
Joined: Fri Jul 27, 2012 12:11 pm

Re: Very Slow output for traffic passing through CHR

Thu Oct 24, 2024 12:57 pm

I wasted time today with this until I found this forum.

CHR with a P1 license on Vultr. Connects internally to a Debian VM which has it's public interface disabled - all traffic is routed through the CHR. The VM downloads at >800Mbit, that's fine. Uploads are in the 1Mbit range and Wireshark shows a lot of retransmissions. The posts above don't give me much hope. VPC2 didn't work well (I can't recall why, similar issues?) so I dropped back to VPC. MTU's on all interfaces are fine 1450 for CHR eth2 and the Debian VM 2nd interface. Public interfaces are 1500. Mucking with MSS mangles and dropping MTU doesn't help. This is disappointing!
Top
 
UpRunTech
Member Candidate
Member Candidate
Posts: 238
Joined: Fri Jul 27, 2012 12:11 pm

Re: Very Slow output for traffic passing through CHR

Fri Oct 25, 2024 6:22 am

Testing with iPerf3 from a Debian VM inside communicating with an external server via the CHR with a P1 license (using masquerading):

TCP: WAN->VPC Speeds are fine.
TCP: VPC->WAN Speeds are slow and erroneous connections.
UDP: WAN->VPC Speeds are fine.
UDP: VPC->WAN Speeds are fine.

I initially noticed the problem when I set up a Rustdesk server on the Debian VM and relay performance was abysmal, the opposite of what I expected. I am wondering if there is a problem with connection tracking for TCP in this particular arrangement.

I tried with VPC2 but I rediscovered the reason I don't use it is that in my case only IP traffic for the same VPC2 subnet seems to work, anything outside of that does not - eg trying to ping 1.1.1.1 from the VM though the CHR shows the traffic doesn't even make it to the CHR's VPC2 interface. Ping traffic or DNS lookups to the CHR work fine. There is some ARP weirdness going on as on the VM the ARP entry for the CHR is a completely different MAC address to what's assigned on the CHR's interface. Vultr support wasn't helpful.
Top
 
UpRunTech
Member Candidate
Member Candidate
Posts: 238
Joined: Fri Jul 27, 2012 12:11 pm

Re: Very Slow output for traffic passing through CHR

Tue Oct 29, 2024 9:35 am

I seem to have solved this problem by running a simple GRE tunnel through the VPC interfaces on the CHR and Debian VM. iperf3 now runs at expected speeds in both directions with UDP and TCP.

In this and the cases above the corruption of TCP traffic between a CHR and a VM using Vultr's VPC service is not explained but can be worked around with a tunnel as I did with GRE.
Top
Post Reply
  4. RouterOS
  5. General