Unstable CRS326-4C+20G+2Q

deadpete · Sun Apr 14, 2024 9:33 pm

Hi folks,

I replaced an old HPE 1950 department switch with a CRS326-4C+20G+2Q+RM a few days ago. It was more or less a plugin replacement, where all the ethernet ports had the same VLAN assignments as in the old switch. The main difference is the connection to the upstream switch (a Mikrotik CRS510-8XS-2XQ-IN), which is using a DAC cable XQ+DA0001 to get a 40 Gbit connection from the upstream switch. I have got a similar arrangement in another location, but the department switch in that case is Mikrotik CRS354-48P-4S+2Q. That setup works perfectly. Not a single hiccup.

The CRS326-4C+20G+2Q is very unstable. I upgraded the firmware to 7.14.2, which is the latest stable version, but that did not help. The problems still persist. The interfaces subsequently stop forwarding packets, and after a few hours (2 - 3), the switch essentially stops communicating, the interfaces with the least traffic stop first. When I connect a freshly rebooted Linux PC with VLAN access to the bridge interface via Winbox (using Wine), I can connect to the switch, but still , most of the other ports are not forwarding anything. There is nothing in the log indicating any problems. There are no Layer 3 firewall rules or something similar interfering, just standard Layer 2 VLAN stuff.

The interfaces have got one of 2 PVIDs, Default VLAN 1 or 10. 6 of the ports also have got tagged traffic, besides untagged traffic. 3 of the ethernet interfaces have got 11 VLANs, but in that case, more than 2 or 3 are seldom active from the client side. One of those interfaces have got 7 VLANS active (almost no traffic, NTP). First, I setup the switch using dynamic VLANs from the PVID on the ethernet ports. As there were problems, I assigned untagged VLANS to each ethernet interface, but that did not help either. I have also scanned the network for potential loops, but there were no changes in the network with respect to topology, or similar. Just a change of switches.

The switch is a production switch for a construction department, so I must get to the customer tomorrow morning at 5 a.m. to put back the old HP switch, so they can do some meaningful work.

There is no question of misconfiguration. Other switches in the network are configured according to the same rules, and they work without any glitches.

I would be very grateful for some input on this problem. I have spent the better part of 3 days to try to get it working, but without any positive results.

Best regards,

Peter

erlinden · Sun Apr 14, 2024 9:43 pm

Can you share the config:

/export file=anynameyoulike

Remove serial and any other private info. And post between code tags by using the </> button.

Did you upgrade firmware as well?

deadpete · Sun Apr 14, 2024 10:06 pm

Hi erlinden,

Yes, I have upgraded the firmware twice. First directly after the initial configuration to 7.14.1, then when to 7.14.2 to see if it did any difference (which it did not).

I will post the configuration tomorrow morning before I decommission the switch.

Best regards,

Peter

erlinden · Sun Apr 14, 2024 10:14 pm

Just to be sure...you know the difference between an OS upgrade and a firmware upgrade?

patrikg · Sun Apr 14, 2024 11:16 pm

In Mikrotik linguistics:

Name                        Mikrotik World     PC World            
---------------------------------------------------------------------
Operating System            RouterOS           Windows 11        
Firmware                    RouterBoot         Bios/UEFI

deadpete · Mon Apr 15, 2024 9:44 am

Hi erlinden,

Sorry for the confusion. I upgraded the RouterOS from the factory installed to 7.14.1 and then to 7.14.2.

Best regards,

Peter

deadpete · Mon Apr 15, 2024 9:57 am

Hi,

Below is the current configuration. I had to set the combo mode manually, as RouterOS was not able to detect the type.

I cannot see any errors in the configuration.

Best regards,

Peter

# 2024-04-15 06:17:33 by RouterOS 7.14.2
# software id = KCBF-PET5
#
# model = CRS326-4C+20G+2Q+
/interface bridge
add name=main pvid=40 vlan-filtering=yes
add frame-types=admit-only-vlan-tagged name=mgmt pvid=190 vlan-filtering=yes
/interface ethernet
set [ find default-name=combo1 ] combo-mode=copper
set [ find default-name=combo2 ] combo-mode=copper
set [ find default-name=combo3 ] combo-mode=copper
set [ find default-name=combo4 ] combo-mode=sfp
set [ find default-name=ether1 ]
set [ find default-name=ether2 ]
set [ find default-name=ether3 ]
set [ find default-name=ether4 ]
set [ find default-name=ether5 ]
set [ find default-name=ether6 ]
set [ find default-name=ether7 ]
set [ find default-name=ether8 ]
set [ find default-name=ether9 ]
set [ find default-name=ether10 ]
set [ find default-name=ether11 ]
set [ find default-name=ether12 ]
set [ find default-name=ether13 ]
set [ find default-name=ether14 ]
set [ find default-name=ether15 ]
set [ find default-name=ether16 ]
set [ find default-name=ether17 ]
set [ find default-name=ether18 ]
set [ find default-name=ether19 ]
set [ find default-name=ether20 ]
set [ find default-name=ether21 ]
set [ find default-name=qsfpplus1-1 ] comment="Trunk in"
/interface vlan
add interface=main name=mgmtnet vlan-id=40
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/port
set 0 name=serial0
/interface bridge port
add bridge=mgmt frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether21 pvid=190
add bridge=main interface=ether1
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether2
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether3
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether4
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether5
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether6 pvid=10
add bridge=main ingress-filtering=no interface=ether7
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether8 pvid=10
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether9 pvid=10
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether10 pvid=10
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether11 pvid=10
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether12 pvid=10
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether13 pvid=10
add bridge=main interface=ether14
add bridge=main interface=ether15
add bridge=main interface=ether16
add bridge=main interface=ether17
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether18 pvid=10
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether19 pvid=90
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether20 pvid=90
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=combo1 pvid=90
add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=combo2 pvid=90
add bridge=main interface=combo3 pvid=190
add bridge=main frame-types=admit-only-vlan-tagged interface=combo4 pvid=989
add bridge=main comment="Trunk in" frame-types=\
    admit-only-vlan-tagged interface=qsfpplus1-1 pvid=997
add bridge=main interface=qsfpplus2-1
/ipv6 settings
set disable-ipv6=yes
/interface bridge vlan
add bridge=mgmt tagged=mgmt untagged=ether21 vlan-ids=190
add bridge=main tagged=qsfpplus1-1,combo4 untagged=\
    ether1,ether2,ether3,ether4,ether5,ether7,ether14,ether15,ether16,ether17 \
    vlan-ids=1
add bridge=main tagged=\
    ether1,ether7,ether15,ether16,ether19,ether20,combo1,qsfpplus1-1 \
    untagged=ether6,ether8,ether9,ether11,ether12,ether13,ether18 vlan-ids=10
add bridge=main tagged=\
    ether1,ether7,ether14,ether15,ether16,ether17,combo4,qsfpplus1-1 \
    vlan-ids=20
add bridge=main tagged=\
    ether1,ether7,ether15,ether16,combo3,combo4,qsfpplus1-1 vlan-ids=40
add bridge=main tagged=\
    ether1,ether7,ether14,ether15,ether16,ether17,combo4,qsfpplus1-1 \
    vlan-ids=60
add bridge=main tagged=ether1,ether7,ether15,ether16,combo4,qsfpplus1-1 \
    untagged=ether19,ether20,combo1,combo2 vlan-ids=90
add bridge=main tagged=\
    ether1,ether7,ether14,ether15,ether16,ether17,qsfpplus1-1 vlan-ids=110
add bridge=main tagged=ether1,ether7,ether15,ether16,combo4,qsfpplus1-1 \
    vlan-ids=120
add bridge=main tagged=ether7,ether15,ether16,qsfpplus1-1 vlan-ids=130
add bridge=main tagged=ether7,ether15,ether16,qsfpplus1-1 vlan-ids=150
add bridge=main tagged=ether1,ether7,ether15,ether16,qsfpplus1-1 vlan-ids=190
add bridge=main tagged=ether1,ether7,ether15,ether16,qsfpplus1-1 vlan-ids=220
/ip address
add address=192.168.4.43/24 interface=mgmtnet network=192.168.4.0
/system identity
set name=What you like
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=192.168.4.98
/system routerboard settings
set boot-os=router-os enter-setup-on=delete-key

erlinden · Mon Apr 15, 2024 10:08 am

I'm far from VLAN expert, but...

I red that using multiple bridges is not necessary (and not recommended).
It is recommended not to set VLAN id to a bridge.

I'll continu reading your config...

Update
With the CRS3xx you can use the switch. There is a great help page on this type of switch:
https://help.mikrotik.com/docs/display/ ... p+features

loloski · Mon Apr 15, 2024 10:12 am

I don't think if this will help you or not, but can you make a single vlan aware bridge and remove a bridge dedicated for the management?

deadpete · Mon Apr 15, 2024 10:38 am

Hi erlinden,

No, I'm not using multiple bridges. The only bridge that is active is the bridge named main (management through VLAN 40), where all the switched ports are connected. The mgmt bridge is intended for management, but I could as well remove it. To the mgmt bridge is only connected the management port (which anyway is unusable for normal traffic as it's a 100 Mbit port). Some Mikrotik switches have got 2 switch chips, and the management port is connected to the basic switch chip, and the switched ports to the fast chip. What I have seen everywhere in the Mikrotik documentation, this is the standard way to manage the switch via a VLAN and I know it's a bad idea to use multiple bridges.

Best regards,

Peter

deadpete · Mon Apr 15, 2024 10:39 am

Hi loloski,

Thanks for your input. I forgot to add you in my answer, but that includes you input as well.

Best regards,

Peter

loloski · Mon Apr 15, 2024 11:29 am

add bridge=main tagged=qsfpplus1-1,combo4 untagged=\
    ether1,ether2,ether3,ether4,ether5,ether7,ether14,ether15,ether16,ether17 \
    vlan-ids=1

for some vlans your ether1 is a tagged port but for vlan-ids=1 it's an access port for can you disabled this first and try again? not unless this is hybrid port which I seldom use, can you check if you do have H flagged port all the bridge port, I'm just worried that they don't have H flag (hardware offloaded) otherwise this must be something else... please update your config to reflect a single bridge vlan aware.

DNAT · Mon Apr 15, 2024 11:40 am

In my understanding, ether21 ist also the first combo port of that switch. The 1st one, you're using on the mgmt-bridge:

add bridge=mgmt frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether21 pvid=190

the 2nd one, you're using on the main bridge:

add bridge=main frame-types=admit-only-untagged-and-priority-tagged \
    interface=combo1 pvid=90

I dont't think that's a good idea.

regards, DNAT

deadpete · Mon Apr 15, 2024 11:56 am

Hi loloski,

Tagged vlan-ids=1 is used for 2 trunk ports qsfpplus1-1 and combo4, the untagged ports are standard switch ports for devices. That's absolutely normal. All interfaces on the switch are flagged with H.

I also have some hybrid ports, but those are mostly used to access different VLANs in the network.

The switch works normally, until it starts to become unresponsive.

Best regards,

Peter

deadpete · Mon Apr 15, 2024 12:08 pm

Hi DNAT,

ether21 is the management port and it does not equal combo1. Combo1 is either used as the standard RJ-45 port 1T, or SFP+ port 1F.

For combo1 I use the RJ-45 port 1T. It needs to be set as untagged for PVID 90. It's completely normal VLAN switching. I could skip admit-only-untagged-and-priority-tagged, as there are no VLANS on that connection, just a dumb device in the other end. It wouldn't make any difference.

Best regards,

Peter

deadpete · Mon Apr 15, 2024 12:19 pm

Hi folks,

I discovered 2 errors in my configuration. Those errors made the switch unmanageable over VLAN 40:

"add name=main pvid=40 vlan-filtering=yes" should be
"add frame-types=admit-only-vlan-tagged name=main pvid=40 vlan-filtering=yes"

and in

"add bridge=main tagged=\
ether1,ether7,ether15,ether16,combo3,combo4,qsfpplus1-1 vlan-ids=40"

i forgot to add the bridge itself. It should look like:

"add bridge=main tagged=\
ether1,ether7,ether15,ether16,combo3,combo4,qsfpplus1-1,main vlan-ids=40"

Now, the switch is manageable over VLAN 40. It should however, not result in the misbehavior I have reported.

Best regards,

Peter

deadpete · Mon Apr 15, 2024 8:07 pm

Hi folks,

I think I've nailed the culprit that's causing instability.

I have got 2 almost identical setups with one CRS510-8XS-2XQ-IN serving as upstream switch, and then in this case a CRS326-4C+20G+2Q, and in the other setup a CRS354-48P-4S+2Q. In both cases the upstream switch, and the department switches are connected with a DAC-cable XQ+DA0001. For both upstream switches I had to set the link speed to manual 40GbitCR on the upstream switches. Later on I also set the link speed to manual 40GbitCR on the department switches. This is maybe redundant, but it does not hurt.

Both department switches are each connected to old HPE OfficeConnect 1920 switches (16-port with 2 SFP interfaces). In the working setup the trunk is connected with UTP cable to an RJ-45 port on the HPE 1920. In the setup where I have got instability, the connection to the HPE 1920 is via optics. In both ends 10Gbit SFP+ optical modules are used, though the link is only 1 Gbit.

During the problem solving, I needed to supply connectivity to a number of workstations, so I installed yet another switch, a 10 Gbit switch CRS312-4C+8XG to make sure the workstations had connectivity, and leave trivial connections to the unstable switch. I also moved the link from the troublesome switch to the CRS312-4C+8XG. I noted that it was tricky to get a connection over the optical link, but after a couple of reboots of the HPE 1920 switch, the link worked. Everything was working for a couple of hours, and then the CRS312-4C+8XG started to display similar behavior as the unstable CRS326-4C+20G+2Q, which worked without any problems during this time.

After some trouble, I had a look in the log of the CRS312-4C+8XG, and there were no particular messages except for one warning: "interface, warning
combo3 excessive or late collission, link duplex mismatch?". This message coincided in time with the time when the switch started to ill behave. I had a sense that I had seen something similar in the log of the CRS326-4C+20G+2Q also. I had a look at internet about this warning, and it seems to be caused by bad cables, or bad hardware. There was also som posts about this happening when link speed autodetection was active. I checked in the CRS312-4C+8XG, and autodetection was active. I fixed the link speed to 1GbaseX in the CRS312-4C+8XG. After that no more problems with establishing a link.

And most gratifyingly, no more switch problems at all!

So, dear Mikrotik. This warning should be promoted to an error, and colored red!

I don't know why the switches start to behave badly after this, and the error handling should really take care of this, instead of starting to playing tricks with the switch.

Problem solved after several days of stress, and sens of futility.

Best regards,

Peter

Unstable CRS326-4C+20G+2Q [SOLVED]

Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q

Re: Unstable CRS326-4C+20G+2Q [SOLVED]

Who is online