I am trying to setup Securityonion to make a SOC in my home network. I use a Topton N100 firewall PC with 4x 2.5 gbit/s Intel i226-v network cards for Securityonion. The management interface is connected to Port9 (1 gbit/s), and the mirror interface is connected to the SFP1 port (2.5 gbit/s). I am using the MikroTik S+RJ10 for SFP module + shielded cat 6 RJ45 cables.
The SwOS version is 2.17 (the newest).
The switch is configured with these settings:
When I download with approx. 700 mbit/s from the Internet (the router is on port 1, and the client is on port 24), the packet loss on the mirror port is approx. 52% (it is mentioned as 52% capture loss in Securyonion). Because there are no Zeek Loss or Suricata Loss in Securityonion, the high packet loss is not due to Securityonion (https://docs.securityonion.net/en/2.4/g ... pture-loss). Even with low download speed (under 30 mbit/s), the packet loss is high (over 20%).
I have tried the following:
- Limit the mirror ingress and mirror egress port in the Forwarding tab
- Disabled (unmark) mirror ingress and mirror egress on all ports in the Forwarding tab, and enabled (mark) "mirror" on all VLANs in the VLANs tab
I dont know how to move on from here, and what I may have done wrong in the configuration.
Packet loss on mirror port on CRS326-24G-2S+ Rev. 2
You do not have the required permissions to view the files attached to this post.
Re: Packet loss on mirror port on CRS326-24G-2S+ Rev. 2
I have no experience with Securityonion, so I'm just speculating here ... Are you sure that the mini PC is able to process in real time whatever software requires? Unlike actual HTTP/FTP/etc protocol between client and server, where any of parties can slow down the transfer, your "sniffer" has to process packets as they arrive, it has no way of slowing the transfer down. If it's not ready to receive next packet due to still processing previous packets, that packet is lost ... and not due to switch not pushing it towards your PC.
A test: run tcpdump (that's CLI variant of wireshark), capture packets into a file and run short transfer between normal computer and internet. See what Securityonion has to say about it. Then open capture file using wireshark and see if it also detects lost packets. If both Securityonion and wireshark give you the ssme impression about loct packet rate, then this indicates tgat indeed switch might be dropping packets. If both ways of analyzing packets disagree, then this indicates software problems.
A test: run tcpdump (that's CLI variant of wireshark), capture packets into a file and run short transfer between normal computer and internet. See what Securityonion has to say about it. Then open capture file using wireshark and see if it also detects lost packets. If both Securityonion and wireshark give you the ssme impression about loct packet rate, then this indicates tgat indeed switch might be dropping packets. If both ways of analyzing packets disagree, then this indicates software problems.
Re: Packet loss on mirror port on CRS326-24G-2S+ Rev. 2
Thanks for the reply. You gave me alot to think about (in a positive way).
Regarding using the Topton firewall:
When I use the htop software in Securityonion, the CPU usage was in the range of approx. 20 - 90 % on all cores. Sometimes 1 CPU core hit 100% usage. The load average (1min) was never above 3.5 (between 2.1 and 3.5 when downloading files from the Internet at approx. 700 - 900 mbit/s). But the CPU frequency was changing between 800 MHz and 2.9 GHz on all cores (3,5 GHz is max frequency for the N100 CPU), maybe due to some power limit. The temperature is not an issue (under 50°C the whole time).
The TCPDUMP test:
I used my ThinkPad X390 with an USB AX88179 Gigabit Ethernet dongle and Debian Bookworm as OS. I connected the laptop to port 16 in the switch, and got these results:
Port 16 configured as VLAN2 member:
Port 16 configured as VLAN4 member:
I downloaded diffrent GNU/Linux ISOs from different websites and mirrors.
The higher loss in SecurityOnion could be due to other network traffic, but it is much lower than approx. 52% I got to begin with.
The test with my gamer PC (the client from my first post):
I also tested the packet loss with the client from my first post. The PC has the following configuration: AMD Ryzen 7700X cpu, ASRock B650M Pro RS with Realtek 8125BG network card, 64 GB ram, RTX 4060 Ti graphic card, and Windows 11 23H2.
I used the pktmon (packet monitor) software included in Win 11, and got these results:
Port 16 configured as VLAN2 member:
Port 10 configured as VLAN2 member (the original port used in my first post):
I was downloaded a bunch of different ISO files as in the TCPDUMP test.
I dont know, why the packet loss is higher, when I download from my gamer than my laptop.
I think my next step could be to buy an Intel I226-V 2.5 gbit/s network card and try to install SO on a more powerful PC (a PC with an AMD Ryzen 5600G CPU + 64GB ram) to see if the Topton computer is the limited factor.
Or do you have any other suggestions on what the next step could be?
Regarding using the Topton firewall:
When I use the htop software in Securityonion, the CPU usage was in the range of approx. 20 - 90 % on all cores. Sometimes 1 CPU core hit 100% usage. The load average (1min) was never above 3.5 (between 2.1 and 3.5 when downloading files from the Internet at approx. 700 - 900 mbit/s). But the CPU frequency was changing between 800 MHz and 2.9 GHz on all cores (3,5 GHz is max frequency for the N100 CPU), maybe due to some power limit. The temperature is not an issue (under 50°C the whole time).
The TCPDUMP test:
I used my ThinkPad X390 with an USB AX88179 Gigabit Ethernet dongle and Debian Bookworm as OS. I connected the laptop to port 16 in the switch, and got these results:
Port 16 configured as VLAN2 member:
- SecurityOnion (SO): 0.7% packet loss
- X390: 0.05% packet loss
Port 16 configured as VLAN4 member:
- SO 0.6% packet loss
- X390 0.02% packet loss
I downloaded diffrent GNU/Linux ISOs from different websites and mirrors.
The higher loss in SecurityOnion could be due to other network traffic, but it is much lower than approx. 52% I got to begin with.
The test with my gamer PC (the client from my first post):
I also tested the packet loss with the client from my first post. The PC has the following configuration: AMD Ryzen 7700X cpu, ASRock B650M Pro RS with Realtek 8125BG network card, 64 GB ram, RTX 4060 Ti graphic card, and Windows 11 23H2.
I used the pktmon (packet monitor) software included in Win 11, and got these results:
Port 16 configured as VLAN2 member:
- SO 12.7% to 33.9% packet loss
- Gamer: 0.01% packet loss
Port 10 configured as VLAN2 member (the original port used in my first post):
- SO: 7.4% to 13.7% packet loss
- Gamer: <0.01% packet loss
I was downloaded a bunch of different ISO files as in the TCPDUMP test.
I dont know, why the packet loss is higher, when I download from my gamer than my laptop.
I think my next step could be to buy an Intel I226-V 2.5 gbit/s network card and try to install SO on a more powerful PC (a PC with an AMD Ryzen 5600G CPU + 64GB ram) to see if the Topton computer is the limited factor.
Or do you have any other suggestions on what the next step could be?
Re: Packet loss on mirror port on CRS326-24G-2S+ Rev. 2
Excellent analysis.
The thing is: processing data, which can't be rate controlled, calls for real-time OS. And vast majority of modern OSes (linux, macOS, windows included) are not real-time OSes. These can only process such data if their hardware is seemingly overspecced for the task ... because that gives them extra capacity to cover any processing spikes. And that means that average resource consumption should stay well bellow certain threshold (for CPU when processing network data streams I'd say bellow 50%).
When it comes to router the hardware constraint is a bit less stringent because router can (slightly) control data rate ... delayed forwarding of packets is common way of rate limiting (think queues). So lack of resources on router/firewall generally reduces throughput without inducing too much of packet loss.
So I think that your plan to try with more powerful computer for running SO is a good plan.
Before you go and shell out money you may want to try to configure your Topton to run at "high performance" (usually BIOSes have some similar setting) which should force computer to run at higher CPU frequencies most of time. The problem with (aggressive) CPU frequency down-switching is that it takes a while to up-switch as well ... and that delay may cause software to skip a beat or two. But if such a setting exists, it might give just enough boost for SO to stop missing packets.
My guess: it's likely that your gamer is pretty much faster than your laptop, so it could ACK packets with considerably lower latency ... and hence use (a bit) more of available bandwidth. If the "sniffing" computer is on the edge of performance, every tiny increase of processing can push it over the edge. And CPU utilization up to 90% indicates that SO is at the edge.I dont know, why the packet loss is higher, when I download from my gamer than my laptop.
The thing is: processing data, which can't be rate controlled, calls for real-time OS. And vast majority of modern OSes (linux, macOS, windows included) are not real-time OSes. These can only process such data if their hardware is seemingly overspecced for the task ... because that gives them extra capacity to cover any processing spikes. And that means that average resource consumption should stay well bellow certain threshold (for CPU when processing network data streams I'd say bellow 50%).
When it comes to router the hardware constraint is a bit less stringent because router can (slightly) control data rate ... delayed forwarding of packets is common way of rate limiting (think queues). So lack of resources on router/firewall generally reduces throughput without inducing too much of packet loss.
So I think that your plan to try with more powerful computer for running SO is a good plan.
Before you go and shell out money you may want to try to configure your Topton to run at "high performance" (usually BIOSes have some similar setting) which should force computer to run at higher CPU frequencies most of time. The problem with (aggressive) CPU frequency down-switching is that it takes a while to up-switch as well ... and that delay may cause software to skip a beat or two. But if such a setting exists, it might give just enough boost for SO to stop missing packets.
Re: Packet loss on mirror port on CRS326-24G-2S+ Rev. 2
Thanks again for a good reply.
I bought an Intel I226-v network card for testing, so I can use a more powerful computer than the Topton computer. I will post the result in mid February (the netword card was not in stock).
I bought an Intel I226-v network card for testing, so I can use a more powerful computer than the Topton computer. I will post the result in mid February (the netword card was not in stock).