Did someone test the speed available on a IPIPV6 Tunnel for RB3011, RB4011 and RB5009 routers ?
(The IPIPv6 Tunnel will be used on the SFP Interface, on a VLAN)

Actually i have a RB3011 that is limiting my IPv4 Internet access speed to around 350 mb/s. The CPU1 core is saturated at 100% by this Tunnel.

I would like to get at least 940mb/s for IPv4 through this IPIPV6 Tunnel.

I need to replace the RB3011 but i want to be sure that a RB4011 or RB5009 will solve the problem before to buy it.

Thanks

...
(The IPIPv6 Tunnel will be used on the SFP Interface, on a VLAN)

Actually i have a RB3011 that is limiting my IPv4 Internet access speed to around 350 mb/s. The CPU1 core is saturated at 100% by this Tunnel.
...

Is this speed before or after heeding the advices of the documentation's Layer2 misconfiguration part's VLAN filtering with multiple switch chips section and taking into account Bridge Hardware Offloading limitations of the QCA8337 switch chip as well as the Setup examples section of the documentation with keeping an eye out for the notes explicitly mentioning the QCA8337 such as the following

On QCA8337 and Atheros8327 switch chips, a default vlan-header=leave-as-is property should be used. The switch chip will determine which ports are access ports by using the default-vlan-id property. The default-vlan-id should only be used on access/hybrid ports to specify which VLAN the untagged ingress traffic is assigned to.

and taking into account how using SFP1 affects ports Eth6 to Eth10 on the second QCA8337?

This is very interesting but i do not think that the problem can be related to the QCA8337 switch chip setup because i'm using an RB3011.

In this router the SFP interface is not in the switch1 or switch2 group. It is directly linked to the CPU1 through a 1 Gbps link.
For LANs, i'm using the switch1 Interfaces that have a 2 Gbps link to CPU0 and CPU1. I'm not using switch2 interfaces.


If i remove the IPIP6 Tunnel and the VLAN on the SFP Wan interface, managing that on an external router, keeping exactly the same setup on LAN interfaces, i get full speed near 1 Gbs.

This is very interesting but i do not think that the problem can be related to the QCA8337 switch chip setup because i'm using an RB3011.

In this router the SFP interface is not in the switch1 or switch2 group. It is directly linked to the CPU1 through a 1 Gbps link.
For LANs, i'm using the switch1 Interfaces that have a 2 Gbps link to CPU0 and CPU1. I'm not using switch2 interfaces.


If i remove the IPIP6 Tunnel and the VLAN on the SFP Wan interface, managing that on an external router, keeping exactly the same setup on LAN interfaces, i get full speed near 1 Gbs.

Actually if the QCA8337's feature set/set up requirements and the hardware lay out which were referenced above is not taken account than the the resulting configuration is adversely affecting the throughput. I am glad that you are aware that if SFP1 is used than the second QCA8337 looses its link to CPU1 therefore CPU0 has to handle all of the "switch2 cpu" and part of the "switch1 cpu" traffic (in other words the traffic which have to leave the given switch chip as it either has to be routed or fed to the firewall).

So if I understand you correctly you are using RouterOS v7.13 or newer on your RB3011 and in your configuration you have:

• bridge1 with ether1 to ether5
• bridge2 with ether6 to ether10
• SFP1 is not part of either of bridge1 or bridge2
• there aren't any VLANs beside you are trying to use on SFP
• when adding the IPIPv6 tunnel to your configuration you included the
  Code: Select all

  clamp-tcp-mss=yes dscp=inherit

  options

Since you are CPU limited and might be also on either RB4011 or RB5009 also (therefore you may have to go with a CCR2004-1G-12S+2XS it would be good to know whether you have increased L2 MTU on each and every every interface on the RB3011 to their hardware limit ("ether1-ether5: 8156 ; ether6-ether10: 8156 ; sfp1: 8158") and have set the MTU to 8000 on all of the above mentioned (unfortunately RB3011 does not make it possible to use the nowadays de facto standard 9000 MTU), including end devices and have set the IPIPv6 tunnel to MTU 8000 as well to substantially reduce the required CPU cycles to transfer a given amount of useful data (L3 MTU minus the IP)? As one can clearly see the sizeable positive effect of increasing the MTU size on the throughput in the above referenced MikroTik test results.
By the way why do you want to add VLAN to the mix on the SFP1 as this may lead to Layer2 misconfiguration which can have adverse effect on the amount of traffic that can be handled with the RB3011?

...
So if I understand you correctly you are using RouterOS v7.13 or newer on your RB3011 and in your configuration you have:

bridge1 with ether1 to ether5
bridge2 with ether6 to ether10
SFP1 is not part of either of bridge1 or bridge2
there aren't any VLANs beside you are trying to use on SFP
when adding the IPIPv6 tunnel to your configuration you included the clamp-tcp-mss=yes dscp=inherit

No i'm not using RouterOS 7 because on the RB3011 it does not work for me. The bandwidth fall down to something like 20 mb/s. I'm not alone with this problem.

Then i reverted to RouterOS 6.49.11.

- bridge2 is not used at all.
- SFP1 is not part of switch1 neither switch2, by nature it is directly connected to CPU1. Then it is not useful to put it inside a bridge. I tried and the bandwidth was even lower.
- there is a VLAN on SFP1, i need it because it is a direct link with an Internet provider that need VLAN 836. It gives IPv6 connectivity.
- i tried the clamp-tcp-mss option on the IPIPv6 Tunnel but it does not make any difference.

- The MTU is set to 1700 for the SFP1 and VLAN interface on it, to allow for some room for the IPIPv6 Tunnel that need a 1500 internal MTU.

It is not useful to rise the MTU more than 1700 on the SFP1 and VLAN interface on it because it is for the provider Internet access limited to MTU = 1500 for IPv6 traffic. The IPIPv6 provider Tunnel traffic has a MTU a bit higher than 1500, but only to support an IPv4 MTU of 1500 inside the Tunnel.

This mean that rising the MTU of the IPv6 link to 9000 or 8000 is not useful because IPv6 traffic is Internet traffic limited to MTU=1500.

I'm using the old bridge method win VLAN interfaces inside bridges for each LAN Subnetwork. I'm using only a single interface for that on switch1. A few L2 manageable switches are behind and are are managing VLANs distribution through GVRP (Procurve) and untagging for final devices.

The CPU load is correct with this VLAN setup even at 1 gb/s upload or download if i do not use the IPIPv6 Tunnel and VLAN setup on the SFP1 (using another router to manage the VLAN and the IPIPv6 tunnel).

You are right i need a more powerful setup to reduce load on CPUs, but before to buy an RB4011 or RB5009 (or a CCR) i would like to be sure they are able to sustain a 1 gb/s bandwidth inside a IPIPv6 Tunnel with ROS 7.

I can't find this information. If somebody could test that, putting an RB4011 / RB5009 between two other more powerful routers to measure the available bandwidth on the SFP interface with a VLAN and IPIPv6 Tunnel on it, it would be nice.

By the way what throughput can you achieve when using IPIP with the undocumented allow-fast-path=yes setting instead of IPIPv6 where fast path is not supported? The other missing feature/setting in IPIPv6 that is present in IPIP(v4) is the dont-fragment one, tough this should not be an issue if ICMP(v6) type 2 (packet too big) traffic is not restricted by firewall (either by the stateless raw or the stateful filter), probably for this reason in the documentation there are separate ICMP chains.

No i'm not using RouterOS 7 because on the RB3011 it does not work for me. The bandwidth fall down to something like 20 mb/s. I'm not alone with this problem.

Then i reverted to RouterOS 6.49.11.

I see. I don't know when was the last time you tried v7 on RB3011 and whether you went with the risk averse method of Netinstall (than recreate the configuration based on previous export) or upgraded (which may have caused quite some random/inexplicable issues down the road), however one thing is sure: despite IPv6 is still a second class citizen on RouterOS it is in a way better shape on v7 than it is on v6, for example there is working NPTv6 and masquerade too.

I'm using the old bridge method win VLAN interfaces inside bridges for each LAN Subnetwork. I'm using only a single interface for that on switch1. A few L2 manageable switches are behind and are are managing VLANs distribution through GVRP (Procurve) and untagging for final devices.

The CPU load is correct with this VLAN setup even at 1 gb/s upload or download if i do not use the IPIPv6 Tunnel and VLAN setup on the SFP1 (using another router to manage the VLAN and the IPIPv6 tunnel).

You are right i need a more powerful setup to reduce load on CPUs, but before to buy an RB4011 or RB5009 (or a CCR) i would like to be sure they are able to sustain a 1 gb/s bandwidth inside a IPIPv6 Tunnel with ROS 7.

I can't find this information. If somebody could test that, putting an RB4011 / RB5009 between two other more powerful routers to measure the available bandwidth on the SFP interface with a VLAN and IPIPv6 Tunnel on it, it would be nice.

As you are using not more than 5 pieces of 8P8C (aka RJ45) ports the RB4011 would not be handicapped with its two switch chip design, however as you are using using other managed switches in your network between the two I would go with the RB5009 as using DHCP Snooping along with DHCP Option 82 to increase the security of the network does not disable hardware offloading on the RB5009 with 88E6393X unlike it does with the RB4011 with its two RTL8367. Also the IGMP(/MLD) snooping needed for a proper multicast setup with the router acting as multicast one with PIM-SM does not disable the hardware offload on the bridge on the RB5009 unlike it does on the RB4011. One thing to keep in mind is that despite the above mentioned added features the RB5009 is a bit more of a SOHO device since it does not have serial console port.

Based on a back of a napkin guestimate:

• the RB5009 is enough for your needs based on the 188.4 kpps (RB3011) vs 761 kpps (RB5009) 64 byte "routing with 25 ip filter rules" official test results, however
• the CCR2004-1G-12S+2XS is the better choice based on the 72.4 kpps (RB3011) vs 238.1 kpps (CCR2004-1G-12S+2XS) 64 byte "single tunnel AES-128-CBC + SHA1" IPsec official test results (relevant if the ipsec-secret option is used when setting up the IPIPv6 tunnel).

PS:

I'm using the old bridge method win VLAN interfaces inside bridges for each LAN Subnetwork. I'm using only a single interface for that on switch1. A few L2 manageable switches are behind and are are managing VLANs distribution through GVRP (Procurve) and untagging for final devices.

The CPU load is correct with this VLAN setup even at 1 gb/s upload or download if i do not use the IPIPv6 Tunnel and VLAN setup on the SFP1 (using another router to manage the VLAN and the IPIPv6 tunnel).

I may have misunderstood your way of setting up bridge and VLANs, anyway @DarkNate explained why single bridge (per switch chip) is the way to go currently on RouterOS.

Thanks for those advices.

I did try RouterOS 7.12 a few days ago and it was the same heavy slow down problem as i got a few months ago. Not a slight 10% or 20% slow down that seems to be seen globally with RouterOS7 specially on old devices, but an heavy slow down to around 20 mb/s download speed.

Nevertheless i did not try a NetInstall neither a full erase of the setup before to update. I did find a few threads in different forums talking about this problem. Then i concluded that the RB3011 had a serious problem with RouterOS 7.

After thinking to those problems, it would be very comfortable for next generations of low cost routers to have a programmable area in the hardware switch chips (an FPGA area), so that it could be possible to program small things like tunnels that are not actually hardware accelerated even by top of the range Mikrotik routers if i'm right.

This would permit (without changing the hardware) to add a couple new hardware accelerated functions, like tunneling for example, or to correct a part of the switch chip that could be bugged, replacing it by a custom function programmed in a small FPGA area of the switch chip.

Thanks for those advices.

I did try RouterOS 7.12 a few days ago and it was the same heavy slow down problem as i got a few months ago. Not a slight 10% or 20% slow down that seems to be seen globally with RouterOS7 specially on old devices, but an heavy slow down to around 20 mb/s download speed.

Nevertheless i did not try a NetInstall neither a full erase of the setup before to update. I did find a few threads in different forums talking about this problem. Then i concluded that the RB3011 had a serious problem with RouterOS 7.

The slowdown is mostly for use cases which greatly benefitted from the route cache which Linux feature is available on the kernel used by the v6 branch, however is not in the one used by the v7 versions.

Such a large performance drop which you have experienced may warrant a clean slate approach with Netinstall and rebuilding from scratch as it is not normal. By the way people starting with v6 running devices may have Netinstalled (and rebuilt based on info in the export) third time already because of the substantial changes that may caused issues (first time the upgrade to v7 than midway to solve pesky storage issue causing loosing parts of the configuration and latest with the WiFi and CAPsMAN focused changes,).
A thing that one needs some getting used to is that the testing releases more often than not have higher stability than the latest available stable release (because of the continuos development, and the approach to fix the stable's issues in the next beta, tough in this regard having a point release in the current stable release is welcome news).

I tried to disable the IPv4 route cache under RouterOs 6.49.11 on this RB3011.

For IPv4 Internet access from a PC browser to a bandwidth test site, here is what i get on the IPIPv6 tunnel :

With route cache enabled :
download 371 mb/s
upload 390 mb/s

With route cache disabled :
303 mb/s
324 mb/s

This is around 17% to 18% speed loss regardless the traffic direction.

This mean that Linux IPv4 route cache has a significative influence for Internet downloads/uploads from/to a single target.
The situation is probably different for a router used at a provider site with many different routes and client destinations. But the truth is that many small Mikrotik routers are used only for Internet access with simple setups using a single default route where route cache is beneficial. For those routers, specially the older and less powerful ones, it is probably better to keep RouterOs 6.

This simple test show us that L3 hardware acceleration and / or heavy CPU / software optimization will become even more important in the futur. We can clearly see here that an RB3011 device, that does not have L3 hardware acceleration, cannot sustain a simple IPIPv6 tunnel at full 1 gb/s link speed.

In my country we are seeing since a few years 2 gb/s and 10 gb/s fiber Internet links offers for general public. Provider router boxes are optimized for those bandwidth, they can deliver full speed (when the EPON branches are not overloaded). Replacing them needs quite high end Mikrotik routers.

I'll try an RB5009 that will permit at the same time to suppress one manageable switch in my installation thanks to the POE outputs on every ports.

I tried to disable the IPv4 route cache under RouterOs 6.49.11 on this RB3011.
...
This is around 17% to 18% speed loss regardless the traffic direction.

The results are in the expected range of performance hit in this usage scenario.

Were you able to test IPIP(v4) usage scenario where there is Fast Path allow-fast-path=yes to increase the performance?

Is the resource hog PPPoE used for authentication by your ISP?

This simple test show us that L3 hardware acceleration and / or heavy CPU / software optimization will become even more important in the futur. We can clearly see here that an RB3011 device, that does not have L3 hardware acceleration, cannot sustain a simple IPIPv6 tunnel at full 1 gb/s link speed.

No current MikroTik equipment has accelerated IPIPv6 (same goes for PPPoE).

Were you able to test IPIP(v4) usage scenario where there is Fast Path allow-fast-path=yes to increase the performance?

No, i do not have an IPIP(v4) tunnel to try that. My provider is IPv6 native, with an IPIP6 Tunnel for IPv4 Internet access.

Nevertheless i did test the underlying native IPv6 speed that carry the IPv4 Tunnel, the speed here is around 600 Mb/s download. Without the RB3011, if i use the provider Internet Box, i get around 940 mb/s. This IPv6 slow down is probably caused by the VLAN interface on top of the SFP. The tagging / untagging on the RB3011 SFP port is using the CPU too unfortunately.

Is the resource hog PPPoE used for authentication by your ISP?

No, this provider is using only the Ethernet MAC address for authentication and native Ethernet IPv6 with an MTU slightly higher than 1500 to allow for an IPIP6 tunnel with an internal MTU of 1500. A simple spoof of the provider box MAC address gives access.
Eventually from here we can get the IPv6 prefix and default IPV6 route through a DHCPv6 client request. It's working i did try it. But as there is only a single /60 to get and a default route, and because that never change, i did the same setup manually. A global IPv6 address on the router is not even necessary to get IPv6 Internet access, the FE80 link local address is enough. But an IPv6 global address on the router WAN port is necessary to mount the IPIP6 tunnel. This address is forged from from the /60 prefix, adding 0:ffff:ffff:0 to it at the end. It is the same method for all clients of this provider.

The setup is really simple : VLAN 836, native IPv6 on it with a /60 that can be split in /64 prefixes (allow to make 15 IPv6 LAN sub networks), then an IPIP6 tunnel on VLAN 836 interface for IPv4 access giving a /32 public IPv4 address, then client IPv4 NAT for client LANs.

Would have been even simpler if this provider would have given native IPv4 too, but unfortunately it is not the case. I suppose that they decided to remove IPv4 from their distribution infrastructure to reduce costs and management.

I got an RB5009. I did put Ros 7.14beta7 on it and i did test an IPIPv6 tunnel on a vlan interface on the SFP.

The internal MTU of the tunnel is 1500. (MTU 1700 in the parent VLAN)

The result is not impressive. CPU2 is near 100% and 875 mb/s is the maximum speed i can get inside the tunnel.

I thought i would have been able to get around 940 mb/s, the same i'm able to get if i use the provider box that this router has replaced.


But nevertheless it is a large improvement compared to the RB3011, that gaves 2.35 times less speed (370 mb/s).