SLOW SPEEDS WITH ETHERNET BRIDGES

cmrp · Thu May 30, 2019 3:30 pm

We have this scenario, (click on image to view in wide)

we have connected the switch and the routers with the with S+DA0003 - SFP DACs (https://mikrotik.com/product/SplusDA0003), the switch only have 4 bridges with 3 interface in each one, all is configured by default, no networks or firewall or nating rules and basically is working a a simple switch with this 4 bridges, nothing more.

all ISP have 1Gbps symmetric links.

when we make a speed test of this setup, the D/U speed is not over than 300Mbps, ever. Now, if we connect directly the ONT of the ISP to the routers the speed of D/U is 1Gbps, ever.

this is a hardware limitation or a misconfiguration? we can not understand why the performance is too poor if is working like a switch/bridge.

Ros: 6.44.3
Switch model: CRS328-4C-20S-4S+RM
Routers model: CCR1036-12G-4S

UPDATE: fixed image url

sindy · Thu May 30, 2019 3:42 pm

You forgot to make the file with the drawing accessible for everyone.

Post your configuration (see the hint in my automatic signature) - maybe hardware "acceleration" (actually, direct forwarding done by the switch chip) is disabled on your bridges?

cmrp · Thu May 30, 2019 4:01 pm

Image URL Fixed

this is the config applied after a full reset and removing the default configuration after the boot

/interface bridge
add name=bridge-01
add name=bridge-02
add name=bridge-03
add name=bridge-04

/interface list
add name=LANs
add name=WANs

/interface list member
add interface="combo4" list=LANs  ### only to access with winbox to configure (no have ip address)
add interface=sfp2 list=WANs
add interface=sfp4 list=WANs
add interface=sfp6 list=WANs
add interface=sfp8 list=WANs
add interface=sfp10 list=WANs

/interface bridge port
add bridge=bridge-01 interface=sfp2
add bridge=bridge-01 interface=sfp1
add bridge=bridge-01 interface=sfp9
add bridge=bridge-02 interface=sfp6
add bridge=bridge-02 interface=sfp3
add bridge=bridge-02 interface=sfp11
add bridge=bridge-03 interface=sfp10
add bridge=bridge-03 interface=sfp5
add bridge=bridge-03 interface=sfp13
add bridge=bridge-04 interface=sfp8
add bridge=bridge-04 interface=sfp7
add bridge=bridge-04 interface=sfp15

/ip neighbor discovery-settings
set discover-interface-list=LANs

/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes

/ip service
set telnet disabled=yes
set ftp disabled=yes
set api disabled=yes
set api-ssl disabled=yes

sindy · Thu May 30, 2019 5:05 pm

Okay. The Port switching chapter of Manual:CRS3xx series switches says:

Warning: Currently it is possible to create only one bridge with hardware offloading on CRS3xx series devices. Use the hw parameter to select which bridge will use hardware offloading.

As you have defined 4 independent bridges, the chance that you hit the one which the RouterOS has chosen to benefit from hardware offloading is 25 %.

So to have full wirespeed, you have to use a single bridge where the individual ports will be configured as access ports to different VLANs. You'll probably want to use S-VLAN tags, and may need to use /interface ethernet switch rule configuration subtree - there is an ongoing integration of VLAN handling by bridge and switch chip, so start from configuring /interface bridge port and /interface bridge vlan for use with vlan-filtering=yes on the single bridge (with ether-type set to the default 0x8100 if you want to use C-VLANs or to 0x88a8 if you want to use S-VLANs); if it doesn't work as expected, start reading the /interface ethernet switch rule chapter.

Or take the simpler path with no VLANs at all, and use the switch chip configuration to set up port isolation on a single bridge without VLANs.

cmrp · Thu May 30, 2019 9:31 pm

Hi sindy

thanks for your answer, i've tested the second option than you mention "Or take the simpler path with no VLANs at all, and use the switch chip configuration to set up port isolation on a single bridge without VLANs." and the speed was improved, but i've detected packet losted doing pings, i've tested first only with 2 ISP (6 interfaces in one bridge with 2 groups)

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=52 time=13.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=52 time=12.8 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=52 time=12.8 ms
64 bytes from 8.8.8.8: icmp_seq=7 ttl=52 time=12.5 ms
64 bytes from 8.8.8.8: icmp_seq=8 ttl=52 time=12.6 ms
64 bytes from 8.8.8.8: icmp_seq=12 ttl=52 time=12.2 ms
64 bytes from 8.8.8.8: icmp_seq=14 ttl=52 time=13.0 ms
64 bytes from 8.8.8.8: icmp_seq=15 ttl=52 time=12.7 ms
64 bytes from 8.8.8.8: icmp_seq=16 ttl=52 time=13.3 ms
64 bytes from 8.8.8.8: icmp_seq=17 ttl=52 time=12.7 ms
64 bytes from 8.8.8.8: icmp_seq=18 ttl=52 time=13.1 ms
^C
--- 8.8.8.8 ping statistics ---
18 packets transmitted, 11 received, 38% packet loss, time 17123ms
rtt min/avg/max/mdev = 12.246/12.849/13.330/0.325 ms

that happens with that 2 ISP, intermittently you know what can be?

sindy · Fri May 31, 2019 4:58 pm

I don't but given the round-trip time and the speed, I'd say there are quite some hops between your hardware and the nearest instance of Google DNS. There is not much reason why a mere L2 network with hardware forwarding should have such a high drop rate, so if I was to debug it, I'd connect one PC instead of ISP 1 and try pinging it from the one from which you pinged 8.8.8.8. If you can see the same drop rate, something is wrong with the switches themselves or, much more likely, with the interconnections between them (dirty or ill-crimped cables in cause if copper connections, dirty connectors or too bent cables in case of optical connections). If there are no drops, the switches and their interconnections are OK and the issue is somewhere between the uplink port of the switch and the Google DNS instance, so in the next step I'd connect the test PC directly to the ISP path using the same cable previously used to connect the switch. If the above is impossible due to different interface types, a microscope, and optical power meter and cleaning wipes in case of optical connections, and crimping tool and a bag of new 8p8c connectors or just a bag of new patchcords are the next tools you need.

Error counters on switch interfaces can tell you where a cabling issue is, unless the packets get malformed on the uplink between the switch and the ISP as you can't see ISP device's counters.

cmrp · Fri Jun 21, 2019 11:59 am

Hi sindy, thanks for your answer, I've can't test before for other thing than i'm working.

the ONTs in fact are SFP GPON ONU Sticks with the ISP fibre attached directly and the interconnections between the routers and the switch are mikrotik DACs fiber cables S+DA0003 as i've mentioned before.

to discard problems with the uplinks and know if the problems are related to the switch, i've tested pings and i've generated traffic in the next scheme

PC1 ------utp cat7-----------> ROUTER ----------DAC SFP cable -----------> PC2 as bridge + SFP GPON ONU Stick ----------- ISP Fibre ---------------> OLT (ISP side)

like this no packages lost and any performance issue.

when i've replace the PC2 with the switch feel the same issue mentioned before (packets lost, slow performance)
PC1 ------utp cat7-----------> ROUTER ----------DAC SFP cable -----------> SWITCH + SFP GPON ONU Stick ----------- ISP Fibre ---------------> OLT (ISP side)
i've reset the configurations in the switch again and i've configured following your second recommendation (without VLANs) the problem persist.

I feel than something is missing or bad configured on the switch, this is very weird.

cmrp · Thu Aug 29, 2019 2:29 pm

all the problem was solved reconfiguring all again in the bridge switch

Current config:

1 - only one bridge with all ports inside and Hardware of OffLoad enabled
2 - Isolate the ports than we want E.g:

port1 + port2 + port9 = line ISP1 and trunk for each router
port3 + port4 + port11 = line ISP2 and trunk for each router
port5 + port6 + port13 = line ISP3 and incoming for each router
etc...

thanks.

PhoeKhwar · Mon Jan 25, 2021 10:33 pm

all the problem was solved reconfiguring all again in the bridge switch

Current config:

1 - only one bridge with all ports inside and Hardware of OffLoad enabled
2 - Isolate the ports than we want E.g:

port1 + port2 + port9 = line ISP1 and trunk for each router
port3 + port4 + port11 = line ISP2 and trunk for each router
port5 + port6 + port13 = line ISP3 and incoming for each router
etc...

thanks.

Can you please show me your config ?
I have the same situation like you

SLOW SPEEDS WITH ETHERNET BRIDGES

SLOW SPEEDS WITH ETHERNET BRIDGES

Re: SLOW SPEEDS WITH ETHERNET BRIDGES

Re: SLOW SPEEDS WITH ETHERNET BRIDGES

Re: SLOW SPEEDS WITH ETHERNET BRIDGES

Re: SLOW SPEEDS WITH ETHERNET BRIDGES

Re: SLOW SPEEDS WITH ETHERNET BRIDGES

Re: SLOW SPEEDS WITH ETHERNET BRIDGES

Re: SLOW SPEEDS WITH ETHERNET BRIDGES

Re: SLOW SPEEDS WITH ETHERNET BRIDGES