Community discussions

MikroTik App
  4. RouterOS
  5. Beginner Basics

CRS310-1G-5S-4S+ Random reboot

Post Reply
 
User avatar
loloski
Member
Member
Topic Author
Posts: 423
Joined: Mon Mar 15, 2021 9:10 pm

CRS310-1G-5S-4S+ Random reboot

Mon Jul 29, 2024 8:24 am

Code: Select all
# 2024-07-29 13:12:15 by RouterOS 7.15.1
# software id = 1ZQI-INIS
#
# model = CRS310-1G-5S-4S+
# serial number = HDF0860Q6TP
/interface bridge
add add-dhcp-option82=yes admin-mac=18:FD:74:FE:43:7C auto-mac=no \
    dhcp-snooping=yes frame-types=admit-only-vlan-tagged mvrp=yes name=\
    DISTRIBUTION vlan-filtering=yes
/interface vlan
add interface=DISTRIBUTION name=VLAN7 vlan-id=7
add interface=DISTRIBUTION name=VLAN13 vlan-id=13
/ip pool
add name=dhcp_pool0 ranges=10.100.12.2-10.100.12.254
add name=dhcp_pool1 ranges=192.168.7.2-192.168.7.254
/ip dhcp-server
add address-pool=dhcp_pool1 interface=VLAN7 name=dhcp2
/port
set 0 name=serial0
/interface bridge port
add bridge=DISTRIBUTION comment="Hybrid Port" interface=ether1 pvid=13 \
    trusted=yes
add bridge=DISTRIBUTION comment=EDGE frame-types=admit-only-vlan-tagged \
    interface=sfp-sfpplus1 pvid=4094 trusted=yes
add bridge=DISTRIBUTION comment=BRAS frame-types=admit-only-vlan-tagged \
    interface=sfp-sfpplus2 pvid=4094 trusted=yes
add bridge=DISTRIBUTION comment=OLT frame-types=admit-only-vlan-tagged \
    interface=sfp-sfpplus3 pvid=4094
add bridge=DISTRIBUTION interface=sfp-sfpplus4
add bridge=DISTRIBUTION interface=sfp1
add bridge=DISTRIBUTION interface=sfp2
add bridge=DISTRIBUTION interface=sfp3
add bridge=DISTRIBUTION interface=sfp4
add bridge=DISTRIBUTION interface=sfp5
/interface ethernet switch l3hw-settings
set ipv6-hw=yes
/ip neighbor discovery-settings
set discover-interface-list=all
/interface bridge vlan
add bridge=DISTRIBUTION tagged=DISTRIBUTION,sfp-sfpplus3 vlan-ids=7
add bridge=DISTRIBUTION tagged=DISTRIBUTION,sfp-sfpplus3,ether1 vlan-ids=9
add bridge=DISTRIBUTION tagged=DISTRIBUTION,sfp-sfpplus1,sfp-sfpplus2 \
    vlan-ids=11
add bridge=DISTRIBUTION tagged=DISTRIBUTION,sfp-sfpplus2,sfp-sfpplus3 \
    vlan-ids=20
add bridge=DISTRIBUTION tagged=DISTRIBUTION,sfp-sfpplus3,sfp-sfpplus1 \
    untagged=ether1 vlan-ids=13
/interface ethernet switch
set 0 l3-hw-offloading=yes
/ip address
add address=192.168.7.1/24 interface=VLAN7 network=192.168.7.0
add address=10.100.13.254/24 interface=VLAN13 network=10.100.13.0
/ip dhcp-server network
add address=192.168.7.0/24 gateway=192.168.7.1
/ip dns
set servers=8.8.8.8
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=10.100.13.1 routing-table=main \
    suppress-hw-offload=no
/ip service
set telnet disabled=yes
set ftp disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Asia/Manila
/system identity
set name=RQUE-POP1-R1-ACCSW
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=time.nist.gov
/system routerboard settings
set auto-upgrade=yes boot-os=router-os enter-setup-on=delete-key
/tool romon
set enabled=yes
Some people from the field is reporting that our edge switch CRS310-1G-5S-4S+ is randomly rebooting at random times this past 2 weeks there were no power outage reported inside the DC and this is the only mikrotik gear is exhibiting issues in this regen site, there where no supout.rif is being generated after reboot so we can't report this to MT.

Can anyone share their troubleshooting steps if they do encounter similar situation? I'm tempted to update ROS but it's very risky and 100 miles away from where i am and I don't see any changelogs that warrant to upgrade pertains/related to this device that might help or alleviate the situation and I don't really know the problem to begin with :(

This is the first time encounter rebooting issues with MT gear over the years so this is uncharted water for me thanks in advance
Top
 
peternz
just joined
Posts: 6
Joined: Wed Jul 03, 2024 4:09 am

Re: CRS310-1G-5S-4S+ Random reboot

Mon Jul 29, 2024 12:59 pm

The first step I would take is to confirm that the switch is rebooting. Use /system/resources/print and look at the uptime to confirm how long ago it rebooted.
Top
 
User avatar
loloski
Member
Member
Topic Author
Posts: 423
Joined: Mon Mar 15, 2021 9:10 pm

Re: CRS310-1G-5S-4S+ Random reboot

Mon Jul 29, 2024 2:11 pm

The switch was indeed had been rebooted it was confirm with our NMS and the actual logs from the switch, the switch is running fine the traffic is around 4gb+ during the outage as per the zabbix graph and we have around 2 dozens of this switch in the field running 24x7 having almost the same traffic pattern but it doesn't exhibit this behavior that's why I'm lost, this can't be factory defect because it's running fine for almost 6 months now starting from ROS 7.12 up to recent 7.15.1

I don't want to file an RMA because the switch is running in production and the performance is ok
Last edited by loloski on Mon Jul 29, 2024 2:19 pm, edited 1 time in total.
Top
 
holvoetn
Forum Guru
Forum Guru
Posts: 6825
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: CRS310-1G-5S-4S+ Random reboot

Mon Jul 29, 2024 2:14 pm

What does log say right after reboot ?
Kernel failure ? Something else ?

Potential problem with power supply ? Any chance to have that replaced by a spare (you should have a spare if it's critical equipment) ?
Top
 
User avatar
loloski
Member
Member
Topic Author
Posts: 423
Joined: Mon Mar 15, 2021 9:10 pm

Re: CRS310-1G-5S-4S+ Random reboot

Mon Jul 29, 2024 2:21 pm

Potential problem with power supply ? Any chance to have that replaced by a spare (you should have a spare if it's critical equipment) ?
Well that's next in my TODO list yes this is critical i'm going to send someone in the DC I just hate the ton of paperwork just to pull this out and replace :)

This is the logs
You do not have the required permissions to view the files attached to this post.
Top
 
holvoetn
Forum Guru
Forum Guru
Posts: 6825
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: CRS310-1G-5S-4S+ Random reboot

Mon Jul 29, 2024 2:49 pm

You are 200% sure nobody "used" the power socket for that device for other purposes for some time ?

If there was no real other power outage there (no other devices being reported down ?), then I would definitely first check that PSU.
Top
 
User avatar
loloski
Member
Member
Topic Author
Posts: 423
Joined: Mon Mar 15, 2021 9:10 pm

Re: CRS310-1G-5S-4S+ Random reboot

Mon Jul 29, 2024 3:07 pm

Yes 200% because it's a regen site so network engineer was station there in any shape or form the roving guard stay outside the premises, I have no choice we have to replace the switch this weekend and continue triaging this issue in the sidelines, thanks for your time looking into I just hope there's another way but time is the essence here so I can't play with it in as much as I want to.

As soon as I retrieved this switch I'm going to strip it down to get the bottom of it
Top
 
User avatar
loloski
Member
Member
Topic Author
Posts: 423
Joined: Mon Mar 15, 2021 9:10 pm

Re: CRS310-1G-5S-4S+ Random reboot

Wed Aug 07, 2024 11:43 pm

After we recall this switch from production, we were able to identify what cause the reboot but we don't know how to reproduce this at will, the culprit is L3 hardware offload if we turn it off the switch is performing well In the lab this switch is for days now running without incident, we believed the issue is not traffic load dependent at least afaict
Top
Post Reply

Who is online

Users browsing this forum: Hemhems and 31 guests
  4. RouterOS
  5. Beginner Basics