Page 1 of 1
CCR2004 packet loss over 10G Interfaces
Posted: Tue Nov 10, 2020 9:44 am
by captainproton
Hallo,
I have a new CCR2004 here and been testing in different constellations.
We only used 10G connection. We connected a Juniper Router, a DWDM System, a Server and a CCR1016.
In any setup we saw significat PL in all directions. The only counter that was running and looked suspicious was the TX Drop on the according outgoing interface. But why are those packets dropped?
We were using hardware-queues only, but the ports did not nearly ran at full capacity and the PL continiously occured anyway. eg. at 100 MBit load
What I consider very sange is that this is the deault setting for a SFP+ Port. This leads to the port links at 10G even though the rate is not selected and behaves ever weirder.
default setting:
also funny, paket statistics for packets > 1023 missing
Something is very wrong with this router. Don't buy before you hear that this is fixed.
Re: CCR2004 packet loss over 10G Interfaces
Posted: Tue Nov 10, 2020 3:04 pm
by server8
Same here
viewtopic.php?f=3&t=167512 opened ticket 4 weeks ago witout any solution from mikrotik.
Re: CCR2004 packet loss over 10G Interfaces
Posted: Tue Nov 10, 2020 10:14 pm
by captainproton
I have seen your post but I thought it was more specific about traffic from 10G to 1G interfaces.
Re: CCR2004 packet loss over 10G Interfaces
Posted: Tue Nov 10, 2020 10:34 pm
by torstorm
Hallo,
I have a new CCR2004 here and been testing in different constellations.
We only used 10G connection. We connected a Juniper Router, a DWDM System, a Server and a CCR1016.
In any setup we saw significat PL in all directions. The only counter that was running and looked suspicious was the TX Drop on the according outgoing interface. But why are those packets dropped?
We were using hardware-queues only, but the ports did not nearly ran at full capacity and the PL continiously occured anyway. eg. at 100 MBit load
What I consider very sange is that this is the deault setting for a SFP+ Port. This leads to the port links at 10G even though the rate is not selected and behaves ever weirder.
default setting:
also funny, paket statistics for packets > 1023 missing
Something is very wrong with this router. Don't buy before you hear that this is fixed.
I also have this router and all is looking normal, dont have any drop packets . I also use 10Gbit sfp+ modules from Finisar.
Re: CCR2004 packet loss over 10G Interfaces
Posted: Wed Nov 11, 2020 8:28 am
by server8
I also have this router and all is looking normal, dont have any drop packets . I also use 10Gbit sfp+ modules from Finisar.
Ros version?
Re: CCR2004 packet loss over 10G Interfaces
Posted: Wed Nov 11, 2020 1:25 pm
by markonen
I took counter samples from three of the seven CCR2004s I have in production and the TX drop rates for the busiest interfaces were:
* 1/23536 packets dropped (0.0042%)
* 1/76566 packets dropped (0.0013%)
* 1/104768 packets dropped (0.00095%)
These are all on 10G ports with 10G fiber links and ROS 6.47.2 or 6.47.3.
Re: CCR2004 packet loss over 10G Interfaces
Posted: Wed Nov 11, 2020 8:06 pm
by torstorm
I also have this router and all is looking normal, dont have any drop packets . I also use 10Gbit sfp+ modules from Finisar.
Ros version?
ROS version is 6.47.7, but I think this router is meant for ROS 7 when it will be stable. ARM64 is better in 5.6.x kernel... :)
Re: CCR2004 packet loss over 10G Interfaces
Posted: Wed Nov 11, 2020 10:38 pm
by server8
I am checking deep and I have discovered that packet loss is only on the SFP+ port s, using 10 giga module on the SFP28 ports we have no packet loss.
According with the block diagran
https://i.mt.lv/cdn/product_files/CCR20 ... 200459.png the SFP+ and SFP28 ports have different bus to the PIPE
Re: CCR2004 packet loss over 10G Interfaces
Posted: Wed Nov 11, 2020 11:31 pm
by mkx
ARM64 is better in 5.6.x kernel... :)
I think it was mentioned that ROS v7 will be running kernel 4.19. But I guess this can change before stable v7 gets rolled out, new linux kernel LTS was announced after the rumour circulated (5.4).
Re: CCR2004 packet loss over 10G Interfaces
Posted: Thu Nov 12, 2020 6:35 am
by torstorm
ARM64 is better in 5.6.x kernel... :)
I think it was mentioned that ROS v7 will be running kernel 4.19. But I guess this can change before stable v7 gets rolled out, new linux kernel LTS was announced after the rumour circulated (5.4).
I unpack ros7.1 b2 , but looks like there is kernel 5.6.3, maybe i am wrong?
Re: CCR2004 packet loss over 10G Interfaces
Posted: Thu Nov 12, 2020 9:36 am
by captainproton
We were using ROS 6.46.8 for our testing.
Upgrading to 6.47.8 would be worth a trial, but somehow I am sceptical...
Re: CCR2004 packet loss over 10G Interfaces
Posted: Thu Nov 12, 2020 10:45 am
by morf
I took counter samples from three of the seven CCR2004s I have in production and the TX drop rates for the busiest interfaces were:
* 1/23536 packets dropped (0.0042%)
* 1/76566 packets dropped (0.0013%)
* 1/104768 packets dropped (0.00095%)
These are all on 10G ports with 10G fiber links and ROS 6.47.2 or 6.47.3.
I'm sorry if I'm off topic. I can't find any real test results in CCR2004 anywhere. Please tell me how much traffic you have going through the most loaded CCR2004 and what is the load on the CPU ?
Re: CCR2004 packet loss over 10G Interfaces
Posted: Thu Nov 12, 2020 2:38 pm
by markonen
I'm sorry if I'm off topic. I can't find any real test results in CCR2004 anywhere. Please tell me how much traffic you have going through the most loaded CCR2004 and what is the load on the CPU ?
Our CCR2004s currently do BGP (full table) and have a simple Raw firewall (conntrack disabled). So with BGP running and no fasttrack/fastpath, we can saturate the main 10G egress interface with web traffic (a few thousand simultaneous flows). One of the CPUs is pegged (the one doing BGP, this is ROS6) and the rest are well-utilized on traffic peaks, but not at 100%. Sorry, don't have more exact figures at the moment.
Overall we can just about serve our workload with these, and that's pretty amazing considering the price point. But really looking forward to testing a CCR2016 and the rest.
Re: CCR2004 packet loss over 10G Interfaces
Posted: Thu Nov 12, 2020 7:36 pm
by torstorm
I am checking deep and I have discovered that packet loss is only on the SFP+ port s, using 10 giga module on the SFP28 ports we have no packet loss.
According with the block diagran
https://i.mt.lv/cdn/product_files/CCR20 ... 200459.png the SFP+ and SFP28 ports have different bus to the PIPE
Thats strange.... I use Finisar FTLX8571D3BCL sfp+ modules, test 10Gbit bandwith and no errors. I also use complex firewall with multiple chains and processor is not loaded even 1/3 of 100%.
On SFP+ modules I use auto negotiation.
For switch i use Mikrotik CRS358 series switch. Maybe you have problems not on router, then on switch side. What switch do you have?
Re: CCR2004 packet loss over 10G Interfaces
Posted: Thu May 06, 2021 8:04 am
by captainproton
Hey there,
as Mikrotik released v6.49beta27 with a change described as
"switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device"
I gave it another try.
Our Core Router <---Mikrotik-10G-DirectAttatchCable---> CCR2004 <---10G-SM-SFP+---> our Ookla Speedtest Server
After a night of operation and ~3 TB of data speedtested, I couldn'd see any TxDrops on both interface, no more PL and no obvious performance issues. This looks very promissing.
I really hope this router is really fixed, can anyone confirm?
Maybe my test scenario was not complex enough since others report PL only with higher CPU load. But in my initial testing several months ago, PL even massively occured, when CPU load was very very low. So this still looks good to me.
Re: CCR2004 packet loss over 10G Interfaces
Posted: Tue May 18, 2021 3:55 pm
by biziiac
Hi,
For me even the beta version has packet loss. But only when I have traffic over 1Gbps.
Hopefully this issue will be resolved soon.
Re: CCR2004 packet loss over 10G Interfaces
Posted: Sat Aug 19, 2023 2:50 pm
by rextended
