MikroTik

Posted: **Thu Jul 04, 2019 12:39 pm**

Hi all,

First post here, so please feel free to redirect if required

I've just received my CRS328-24P-4S+RM which looks excellent; however, as others have noted here the "typical" fan noise is not "house" friendly. I'm therefore planning to replace the existing two fans with Noctua NF-A4x20 PWMs.

The installed fans are Sunford AD4028V12BLBGF's. I've not been able to find the exact specs for this fan, but from what I can deduce from the Sunford AD4028GF series specs the closest match is the AD4028V12_LWGF with 11.61 CFM airflow and 12.15 mmH2O static pressure. By comparison, the Noctua NF-A4x20 PWMs have 5.5 CFM airflow and (only) 2.26 mmH2O static pressure.
The static pressure is my main concern here, as my PoE usage will be limited and with low wattage requirements. Accordingly, I'm planning to replace the existing Sunfords and install two further fans (4 in total) in the fan3 and fan4 slot.

My question regards the direction (push or pull) that I should install fan3 and fan4. fan1 and fan2 as currently configured are operated in pull mode (air pulled from inside the case to outside). My inclination is therefore to install fan3 and fan4 in push mode, so that they draw cool air into the case, and in general the airflow is maintained right to left across the unit.

Does anybody know in which direction fan3 and fan4 are intended to operate?

Thanks, Paul

Posted: **Thu Jul 04, 2019 2:29 pm**

My inclination is therefore to install fan3 and fan4 in push mode, so that they draw cool air into the case, and in general the airflow is maintained right to left across the unit.

If you do that, you could create a 'circling' path for the air: fan3 and fan4 suck air in, it passes from right ot left, fan1 and fan2 push it outside, where (part of it) passes from left to right and fan3 and fan4 suck it in (again) ...
The problem can become serious if space behind the RB is not really open (e.g. inside some smaller equipment rack or if router is mounted between longer servers) which makes harder for hot air to move away from the back side.

Most of gear I've seen maintains single direction of air across the device and usually that's front-to-back (most of rack installations assume cold zone in front and hot zone at the back). So if you find additional fans necessary, I'd go for the same air flow direction (front to back).

Posted: **Thu Jul 11, 2019 6:44 pm**

Hi Metod,

Thanks for the reply. The fans in the CRS328-24P-4S+RM are located on each side of the unit, so the airflow is right to left (when looking at the front of the switch). There are no rear fans or vents, so no "circling" I think.

I had a response from Mikrotik support which corroborates my intent;

"you should put them in "reverse" mode. Since 1&2 push air from the case, 3&4 fans should push air into the case."

Images of the before and after here:

Before
After

And all seems to be well - note the switch is running slightly warm as it's in a cupboard

> /system health print
     cpu-temperature: 49C
   power-consumption: 10.5W
  board-temperature1: 35C
        psu1-voltage: 26.4V
        psu2-voltage: 52.7V
        psu1-current: 0A
        psu2-current: 0.2A
          fan1-speed: 1590RPM
          fan2-speed: 1605RPM
          fan3-speed: 1530RPM
          fan4-speed: 1545RPM

Posted: **Sun Oct 06, 2019 8:57 pm**

Hello Paul,

Can you tell me if it worked? Is it better?
I have to peaces in house but one of them made a lot of noise. So i will replaced the fans but my question is wich fan did you use? And where did you get it?

Kind regards,
Thetmar Wiegers

Posted: **Sun Oct 06, 2019 10:20 pm**

To be honest if you're running RouterOS then make sure you're running the latest version and you won't have to swap out any fans.

Whatever changes they made means mine is practically silent with fans running at 270RPM instead of 4000RPM.

Of course this is at the expense of temperature which is now 58 degrees instead of somewhere in the 40s.

Screenshot from 2019-10-06 20-20-15.png

Screenshot from 2019-10-06 20-19-33.png

Posted: **Fri Oct 11, 2019 8:09 am**

Hello Paul,

Can you tell me if it worked? Is it better?
I have to peaces in house but one of them made a lot of noise. So i will replaced the fans but my question is wich fan did you use? And where did you get it?

Kind regards,
Thetmar Wiegers

What version are you using and can you post screenshots of /system health?
As @elbob2002 have said - in latest versions there were improvements which affects fan speed based on temperature.

Posted: **Fri Oct 11, 2019 2:01 pm**

Can you tell me if it worked? Is it better?
I have to peaces in house but one of them made a lot of noise. So i will replaced the fans but my question is wich fan did you use? And where did you get it?

Hi Thetmar,

It certainly operates well and is quiet. By placing my hand next to the fan there is good air flow through the unit, but I haven't carried out an exhaustive flow/temperature survey within the unit

The fans I used were Noctua NF-A4x20 PWMs. I used four in total as set out above; also see the attached pics for orientation of the fans to keep the wiring tidy(ish). I purchased the fans through Amazon, here https://www.amazon.co.uk/gp/product/B07DXRNYNX. Frankly they're not cheap, but they are quiet.

I'm currently running 6.45.6 (stable) and the system health is:

/system health print              
     cpu-temperature: 57C
   power-consumption: 9.8W
  board-temperature1: 33C
        psu1-voltage: 26.4V
        psu2-voltage: 52.7V
        psu1-current: 0A
        psu2-current: 0.1A
          fan1-speed: 1635RPM
          fan2-speed: 1740RPM
          fan3-speed: 1530RPM
          fan4-speed: 1515RPM

Hope that helps

Best

Paul

Posted: **Wed Aug 19, 2020 4:21 pm**

Can you tell me if it worked? Is it better?
I have to peaces in house but one of them made a lot of noise. So i will replaced the fans but my question is wich fan did you use? And where did you get it?
Hi Thetmar,

It certainly operates well and is quiet. By placing my hand next to the fan there is good air flow through the unit, but I haven't carried out an exhaustive flow/temperature survey within the unit :)

The fans I used were Noctua NF-A4x20 PWMs. I used four in total as set out above; also see the attached pics for orientation of the fans to keep the wiring tidy(ish). I purchased the fans through Amazon, here https://www.amazon.co.uk/gp/product/B07DXRNYNX. Frankly they're not cheap, but they are quiet.

I'm currently running 6.45.6 (stable) and the system health is:
Code: Select all
/system health print              
     cpu-temperature: 57C
   power-consumption: 9.8W
  board-temperature1: 33C
        psu1-voltage: 26.4V
        psu2-voltage: 52.7V
        psu1-current: 0A
        psu2-current: 0.1A
          fan1-speed: 1635RPM
          fan2-speed: 1740RPM
          fan3-speed: 1530RPM
          fan4-speed: 1515RPM
Hope that helps

Best

Paul

I'm guessing you don't have something that measures decibels, do you? I'm considering this instead of a Ubiquiti equivalent, but need it to be "near-silent" as it's going in a rack in a bedroom. I appreciate that this is nearly a year later than your thread

Posted: **Wed Aug 19, 2020 6:43 pm**

I'm guessing you don't have something that measures decibels, do you? I'm considering this instead of a Ubiquiti equivalent, but need it to be "near-silent" as it's going in a rack in a bedroom. I appreciate that this is nearly a year later than your thread

@SolidProfession - I can take a reading with a decibel meter on my iPhone to give you an idea. Will need to isolate the switch so will have to shutdown the remainder of my comms cabinet. Do you need it urgently?

Posted: **Wed Aug 19, 2020 7:08 pm**

Hey - Could I ask how you accessed those graphs ? Is it within the RouterOS Software or some other 3rd party?

To be honest if you're running RouterOS then make sure you're running the latest version and you won't have to swap out any fans.

Whatever changes they made means mine is practically silent with fans running at 270RPM instead of 4000RPM.

Of course this is at the expense of temperature which is now 58 degrees instead of somewhere in the 40s.

Screenshot from 2019-10-06 20-20-15.png
Screenshot from 2019-10-06 20-19-33.png

Posted: **Wed Aug 19, 2020 9:02 pm**

I'm guessing you don't have something that measures decibels, do you? I'm considering this instead of a Ubiquiti equivalent, but need it to be "near-silent" as it's going in a rack in a bedroom. I appreciate that this is nearly a year later than your thread
@SolidProfession - I can take a reading with a decibel meter on my iPhone to give you an idea. Will need to isolate the switch so will have to shutdown the remainder of my comms cabinet. Do you need it urgently?

Hi,

I appreciate your reply, and no it's not urgent at all. I'm looking to essentially consolidate what I currently have (A fanless Cisco switch that does PoE, that's connected into a HP switch) Eight of those ports are providing PoE to access points, IP cameras, and a SIP base thing

May I ask what you've got plugged in, that requires PoE and how many devices use PoE from the Mikrotik switch you have? If this switch is near silent, then that definitely makes it a contender and not having to swap out for a Ubiquiti one instead!

Posted: **Thu Aug 20, 2020 10:50 am**

Hey - Could I ask how you accessed those graphs ? Is it within the RouterOS Software or some other 3rd party?

Sure - I'm using LibreNMS. SNMP monitoring of RouterOS.

Unfortunately if you're using SwOS then SNMP is quite a bit neutered.

Posted: **Thu Aug 20, 2020 1:12 pm**

May I ask what you've got plugged in, that requires PoE and how many devices use PoE from the Mikrotik switch you have?

Hi @SolidProfession, I currently have two UniFi AP-nanoHD's connected and the rest of my used ports (0-7) are non-PoE devices. I will also connect a UniFi AP-AC-Mesh, but I need to run some further cabling for that (running SWA Cat 6 through an old farmhouse is non-trivial).

The current draw is as follows:

/interface ethernet poe monitor numbers=0,1,2,3,4,5,6,7 once 
               name: eth1     eth2     eth3       eth4       eth5     eth6     eth7     eth8
            poe-out: off      off      auto-on    auto-on    off      off      off      off
        poe-voltage: auto     auto     auto       auto       auto     auto     auto     auto
     poe-out-status: disabled disabled powered-on powered-on disabled disabled disabled disabled
    poe-out-voltage:                   52.2V      52.2V                                 
    poe-out-current:                   124mA      97mA                                  
      poe-out-power:                   6.4W       5W

So, not masses! One thing I noted is that internally (see pictures earlier in this thread) the PoE is split into three banks for ports 0-7, 8-15, 16-23, so I've consolidated "all" my devices into the first bank. I hadn't really considered whether splitting the devices across the banks would be beneficial from a thermals perspective.

I'll get back to you with the dB readings in due course - I assume measured at 1m would be appropriate?

Posted: **Thu Aug 20, 2020 1:18 pm**

May I ask what you've got plugged in, that requires PoE and how many devices use PoE from the Mikrotik switch you have?
Hi @SolidProfession, I currently have two UniFi AP-nanoHD's connected and the rest of my used ports (0-7) are non-PoE devices. I will also connect a UniFi AP-AC-Mesh, but I need to run some further cabling for that (running SWA Cat 6 through an old farmhouse is non-trivial).

The current draw is as follows:
Code: Select all
/interface ethernet poe monitor numbers=0,1,2,3,4,5,6,7 once 
               name: eth1     eth2     eth3       eth4       eth5     eth6     eth7     eth8
            poe-out: off      off      auto-on    auto-on    off      off      off      off
        poe-voltage: auto     auto     auto       auto       auto     auto     auto     auto
     poe-out-status: disabled disabled powered-on powered-on disabled disabled disabled disabled
    poe-out-voltage:                   52.2V      52.2V                                 
    poe-out-current:                   124mA      97mA                                  
      poe-out-power:                   6.4W       5W    
So, not masses! One thing I noted is that internally (see pictures earlier in this thread) the PoE is split into three banks for ports 0-7, 8-15, 16-23, so I've consolidated "all" my devices into the first bank. I hadn't really considered whether splitting the devices across the banks would be beneficial from a thermals perspective.

I'll get back to you with the dB readings in due course - I assume measured at 1m would be appropriate?

Thanks. That's rather something. I'll be changing my Arubas for UnIFi too. I largely just wanted to "learn Aruba" but once I get the UnIFI controller hosted in Azure, then I'll swap them out for UniFis . Also, it'd reduce the electricity bill!

1m would absolutely be appropriate

Posted: **Wed Aug 26, 2020 12:31 pm**

So, I managed to test the noise level this morning.

To be fair the fans are running quite slowly (~1000rpm) and I can't find a way to control the speed to get a better (worst case) test.

/system health> print
     cpu-temperature: 58C
   power-consumption: 10.2W
  board-temperature1: 33C
        psu1-voltage: 26.5V
        psu2-voltage: 52.7V
        psu1-current: 0A
        psu2-current: 0.1A
          fan1-speed: 1080RPM
          fan2-speed: 1140RPM
          fan3-speed: 990RPM
          fan4-speed: 915RPM

The measurements are as follows:

- Ambient room noise 30dB
- Switch noise 32dB@1m (38dB@0.1m)

I'll see if I can find a way to "fix" the fan speed to get a better test.

Summary, at these load levels / fan speeds, it is silent.

Posted: **Wed Aug 26, 2020 12:34 pm**

So, I managed to test the noise level this morning.

To be fair the fans are running quite slowly (~1000rpm) and I can't find a way to control the speed to get a better (worst case) test.
Code: Select all
/system health> print
     cpu-temperature: 58C
   power-consumption: 10.2W
  board-temperature1: 33C
        psu1-voltage: 26.5V
        psu2-voltage: 52.7V
        psu1-current: 0A
        psu2-current: 0.1A
          fan1-speed: 1080RPM
          fan2-speed: 1140RPM
          fan3-speed: 990RPM
          fan4-speed: 915RPM
The measurements are as follows:

- Ambient room noise 30dB
- Switch noise 32dB@1m (38dB@0.1m)

I'll see if I can find a way to "fix" the fan speed to get a better test.

Summary, at these load levels / fan speeds, it is silent.

Thanks for that, regardless. A "worst case" would certainly be good, but even without that, that's still pretty good! Makes me want to actually go ahead and buy one, given that it is ultimately cheaper than other stuff and also lower power than equivalent equipment

Posted: **Sun Sep 27, 2020 12:16 am**

So, I managed to test the noise level this morning.

To be fair the fans are running quite slowly (~1000rpm) and I can't find a way to control the speed to get a better (worst case) test.
Code: Select all
/system health> print
     cpu-temperature: 58C
   power-consumption: 10.2W
  board-temperature1: 33C
        psu1-voltage: 26.5V
        psu2-voltage: 52.7V
        psu1-current: 0A
        psu2-current: 0.1A
          fan1-speed: 1080RPM
          fan2-speed: 1140RPM
          fan3-speed: 990RPM
          fan4-speed: 915RPM
The measurements are as follows:

- Ambient room noise 30dB
- Switch noise 32dB@1m (38dB@0.1m)

I'll see if I can find a way to "fix" the fan speed to get a better test.

Summary, at these load levels / fan speeds, it is silent.

Thought you (and the rest of the community) may like to know that I've managed to get even better results with Noctua's new fans. They're about half the size of the ones you have, which probably helps. I've likely got a later version of the Mikrotik switch AND a later version of the Noctua fans (That much I can say for sure, given the tiny size of these)

You can see my "progress" , and my temperatures here:

https://imgur.com/a/EgynIc6

Another thing I really like about this switch is the metal thing at the back of the switch, where the power goes. It makes it impossible to pull the power lead!

@Thetmar
@elbob2002
@TheComputerTech

^ Tagging you guys too, in case you're interested as you posted in this thread.

Posted: **Sun Sep 27, 2020 10:25 am**

Very interesting!

Mine has been behaving itself with all recent updates.. The nosiest things in my room now are a couple of HP Proliants. Not much I can do about them though!

Posted: **Sun Sep 27, 2020 12:48 pm**

Very interesting!

Mine has been behaving itself with all recent updates.. The nosiest things in my room now are a couple of HP Proliants. Not much I can do about them though!

Why can't you do much about them? Are they like the fans in my NAS where you can't get to them?

Posted: **Sun Sep 27, 2020 1:11 pm**

At least 1U proliants have plenty of small-diameter fans, all running hard. Making those servers really loud beasts.

Posted: **Sun May 23, 2021 12:15 am**

Thanks for this. I replaced the two fans with the Noctua NF-A4x10 PWM and it reduced the noise level dramatically. I actually can't hear it now and it made our home peaceful.

I've ordered two more to put in slots 3 and 4.

Fans: https://www.amazon.com/gp/product/B07DXRNYNX

Current CPU temps are 47C with the fans running at 1300 RMP. With the noisy fans I had a CPU temp of 33C with a single fan running at 7400 RPM. I'm hoping the fans I'll place in slots 3 and 4 lower the temp some but I don't think 47C is much to be worried about. I run around 10 PoE devices (two access points and 8 cameras)

This contains videos of before and after:
https://imgur.com/a/CrP4YvW

Posted: **Wed Jun 09, 2021 4:24 pm**

Hi,

I have a problem that the fans are very loud and the processor temperature is not very high 65C where the housing is 35C. Anyone have an idea where to look for a reason why windmills work like this?

cpu-temperature: 65C
power-consumption: 7W
board-temperature1: 38C
psu1-voltage: 26.5V
psu2-voltage: 52.9V
psu1-current: 0A
psu2-current: 0.1A
fan1-speed: 5865RPM
fan2-speed: 6210RPM
fan3-speed: 4980RPM
fan4-speed: 5745RPM

Posted: **Thu Jun 10, 2021 10:44 am**

Thanks for this. I replaced the two fans with the Noctua NF-A4x10 PWM and it reduced the noise level dramatically. I actually can't hear it now and it made our home peaceful.

I've ordered two more to put in slots 3 and 4.

Fans: https://www.amazon.com/gp/product/B07DXRNYNX

Current CPU temps are 47C with the fans running at 1300 RMP. With the noisy fans I had a CPU temp of 33C with a single fan running at 7400 RPM. I'm hoping the fans I'll place in slots 3 and 4 lower the temp some but I don't think 47C is much to be worried about. I run around 10 PoE devices (two access points and 8 cameras)

This contains videos of before and after:
https://imgur.com/a/CrP4YvW

That's fantastic! I think even the 20mm ones are decent too

Posted: **Thu Jun 10, 2021 10:47 am**

Hi,

I have a problem that the fans are very loud and the processor temperature is not very high 65C where the housing is 35C. Anyone have an idea where to look for a reason why windmills work like this?

cpu-temperature: 65C
power-consumption: 7W
board-temperature1: 38C
psu1-voltage: 26.5V
psu2-voltage: 52.9V
psu1-current: 0A
psu2-current: 0.1A
fan1-speed: 5865RPM
fan2-speed: 6210RPM
fan3-speed: 4980RPM
fan4-speed: 5745RPM

Have you changed your fans inside? Are they Noctua ones? Where is your switch placed? Did you get these temperatures prior to the latest heatwave? Where about in the world are you based?

I live in the UK, so the kit doesn't get much sunlight. Even now, it's somewhat protected behind a glass door, but I expect that the kit will likely have warmed up a tad.

Posted: **Fri Oct 29, 2021 11:52 am**

Dear Friends,

I jump on this issue to find a suitable solution.
The CRS312-4C+8XG is sitting in my living room and installed with RouterOS 7.1rc5.
Still, from time to time, ever with very little network activity, the fans are turning with some noise.
The noise is less than with swOS but still it is unacceptable in a living room.

Apparently, there are four fans.
Will this make less noise if I change them?

I am quite surprized that Mikrotik does not offer a "quiet" mode to choose fan mode.
For example, in Synology NAS, there are three modes : quiet, normal and cool mode.
Is there a to-do list for RouterOS where to ask for a feature?

Posted: **Fri Oct 29, 2021 9:53 pm**

Full model designation of CRS312-4C+8XG ends with -RM ... meaning rack mount. I don't think MT developers were considering noise level for a device intended to be mounted in a rack (which often implies data center with high noise levels). For use in living room you should be considering a passive-cooled switch, but there aren't many passive cooled 10Gbps RJ switches because UTP transmitters tend to run hot when operating at higher speeds.

Posted: **Fri Oct 29, 2021 10:11 pm**

The CRS312-4C+8XG is sitting in my living room and installed with RouterOS 7.1rc5.
Still, from time to time, ever with very little network activity, the fans are turning with some noise.
The noise is less than with swOS but still it is unacceptable in a living room.

Apparently, there are four fans.
Will this make less noise if I change them?

I also have this switch acting as my 10gig core in the corner of my living room, which I use as an office area. I have tried the latest stable revs of both SwitchOS 2.13 and RouterOS 6.49 as they relate to fan noise. The CRS312 seems to run the fans at a relatively stable but audible rate when running switch code. It reminds me of the way a Brocade sounds, but not as enthusiastic. RouterOS code will allow the fans to fluctuate more. I have my core fully-loaded, and when running router code the device goes silent at times.

I think the CPU dictates the fan speed on this model when running router code, so when it hits 58C or so the fans come on until it reaches it's bottom-end target temp. I have found the RouterOS fan management to be more pleasing to me personally. I had intended to put this switch under my entertainment center where my old n-gear core switch was, but it ended up being too loud for that area.

I really don't mind the sound that much now with the CRS in the new location and running RouterOS, but I am still curious how the Noctua fans would work since I am working most days right next to the device. I don't want to install $60 worth of fans, but even so, I couldn't find anything comparable to this switch that was quieter for less than ~$900. With a fan investment, it's still a pretty great deal in my opinion.

I've been researching aftermarket fan options for this device, and I have also been considering purchasing 3x 20mm thick and 1x 10mm thick Noctua FLX 12v fans. I had even ordered them from Amazon but I canceled when I decided to run 2 cables to move the core from under the TV and entertainment center area and into the corner "office."

Every video I've watched it seems that the Noctua fans are producing a fault, but I can't determine if that's due to folks using 4-pin pwm fans because it's a 4-pin socket when they should be using 3-pin DC. If anyone figures this out before I do, please let me know. If I figure this out, I'll post.

Posted: **Sat Oct 30, 2021 2:58 am**

Apparently, there are four fans.
Will this make less noise if I change them?

A friend of mine just swapped her factory fans out today for the 40x20mm Noctua 12Vs and she has experienced a noise reduction of over 10 decibels! The difference is astounding.

Posted: **Sun Nov 28, 2021 11:48 am**

Dear all,

I am going to give a last try changing fans before I dump the CRS312-4C+8XG.
I want to go silent in the living room and I am prepared to invest for it.

Could you confirm what fan model I should buy: how many fans I should buy, I guess 4 pin (adjustable speed) and thickness.

Is there any advantage in adding more fans than originally to run silently?
For example on my Synology NAS there are two large fans but they are both spinning very silently.

Thank you in advance,
Kind regards,

Posted: **Sat Feb 03, 2024 10:53 pm**

Sorry to necro this post but it came up when I searched on the original fan reference.

Just to add that I have just swapped the fans in a CRS328 (the old ones bearings were shot - but then the internal date stamps are 2017...)

I fitted 4 of these ARCTIC S4028-6K (which are cheaper than the noctua ones), and it is pretty quiet with the fans at 1000RPM, at present I have no POE load on it and the fans are actually off...

5 fans for £26 - bargain..

https://www.amazon.co.uk/gp/product/B09 ... UTF8&psc=1

David

MikroTik

CRS328-24P-4S+RM installation of fan3 and fan4

CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4

Re: CRS328-24P-4S+RM installation of fan3 and fan4