Community discussions

MikroTik App
  4. Other topics
  5. RouterBOARD hardware

CCR2004 packet loss over 10G Interfaces

Post Reply
 
captainproton
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 72
Joined: Sat Jan 12, 2008 3:01 pm

CCR2004 packet loss over 10G Interfaces

Tue Nov 10, 2020 9:44 am

Hallo,

I have a new CCR2004 here and been testing in different constellations.

We only used 10G connection. We connected a Juniper Router, a DWDM System, a Server and a CCR1016.
In any setup we saw significat PL in all directions. The only counter that was running and looked suspicious was the TX Drop on the according outgoing interface. But why are those packets dropped?
Image

Image

We were using hardware-queues only, but the ports did not nearly ran at full capacity and the PL continiously occured anyway. eg. at 100 MBit load

What I consider very sange is that this is the deault setting for a SFP+ Port. This leads to the port links at 10G even though the rate is not selected and behaves ever weirder.
default setting:
Image

also funny, paket statistics for packets > 1023 missing
Image

Something is very wrong with this router. Don't buy before you hear that this is fixed.
Top
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss over 10G Interfaces

Tue Nov 10, 2020 3:04 pm

Same here viewtopic.php?f=3&t=167512 opened ticket 4 weeks ago witout any solution from mikrotik.
Top
 
captainproton
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 72
Joined: Sat Jan 12, 2008 3:01 pm

Re: CCR2004 packet loss over 10G Interfaces

Tue Nov 10, 2020 10:14 pm

I have seen your post but I thought it was more specific about traffic from 10G to 1G interfaces.
Top
 
torstorm
just joined
Posts: 5
Joined: Mon Sep 21, 2020 9:50 pm

Re: CCR2004 packet loss over 10G Interfaces

Tue Nov 10, 2020 10:34 pm

Hallo,

I have a new CCR2004 here and been testing in different constellations.

We only used 10G connection. We connected a Juniper Router, a DWDM System, a Server and a CCR1016.
In any setup we saw significat PL in all directions. The only counter that was running and looked suspicious was the TX Drop on the according outgoing interface. But why are those packets dropped?
Image

Image

We were using hardware-queues only, but the ports did not nearly ran at full capacity and the PL continiously occured anyway. eg. at 100 MBit load

What I consider very sange is that this is the deault setting for a SFP+ Port. This leads to the port links at 10G even though the rate is not selected and behaves ever weirder.
default setting:
Image

also funny, paket statistics for packets > 1023 missing
Image

Something is very wrong with this router. Don't buy before you hear that this is fixed.
I also have this router and all is looking normal, dont have any drop packets . I also use 10Gbit sfp+ modules from Finisar.
Top
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss over 10G Interfaces

Wed Nov 11, 2020 8:28 am

I also have this router and all is looking normal, dont have any drop packets . I also use 10Gbit sfp+ modules from Finisar.
Ros version?
Top
 
markonen
newbie
Posts: 31
Joined: Tue Aug 11, 2020 4:28 pm

Re: CCR2004 packet loss over 10G Interfaces

Wed Nov 11, 2020 1:25 pm

I took counter samples from three of the seven CCR2004s I have in production and the TX drop rates for the busiest interfaces were:

* 1/23536 packets dropped (0.0042%)
* 1/76566 packets dropped (0.0013%)
* 1/104768 packets dropped (0.00095%)

These are all on 10G ports with 10G fiber links and ROS 6.47.2 or 6.47.3.
Top
 
torstorm
just joined
Posts: 5
Joined: Mon Sep 21, 2020 9:50 pm

Re: CCR2004 packet loss over 10G Interfaces

Wed Nov 11, 2020 8:06 pm

I also have this router and all is looking normal, dont have any drop packets . I also use 10Gbit sfp+ modules from Finisar.
Ros version?
ROS version is 6.47.7, but I think this router is meant for ROS 7 when it will be stable. ARM64 is better in 5.6.x kernel... :)
Top
 
server8
Long time Member
Long time Member
Posts: 592
Joined: Fri Apr 22, 2011 1:27 pm

Re: CCR2004 packet loss over 10G Interfaces

Wed Nov 11, 2020 10:38 pm

I am checking deep and I have discovered that packet loss is only on the SFP+ port s, using 10 giga module on the SFP28 ports we have no packet loss.

According with the block diagran https://i.mt.lv/cdn/product_files/CCR20 ... 200459.png the SFP+ and SFP28 ports have different bus to the PIPE
Top
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 13033
Joined: Thu Mar 03, 2016 10:23 pm

Re: CCR2004 packet loss over 10G Interfaces

Wed Nov 11, 2020 11:31 pm

ARM64 is better in 5.6.x kernel... :)

I think it was mentioned that ROS v7 will be running kernel 4.19. But I guess this can change before stable v7 gets rolled out, new linux kernel LTS was announced after the rumour circulated (5.4).
Top
 
torstorm
just joined
Posts: 5
Joined: Mon Sep 21, 2020 9:50 pm

Re: CCR2004 packet loss over 10G Interfaces

Thu Nov 12, 2020 6:35 am

ARM64 is better in 5.6.x kernel... :)

I think it was mentioned that ROS v7 will be running kernel 4.19. But I guess this can change before stable v7 gets rolled out, new linux kernel LTS was announced after the rumour circulated (5.4).
I unpack ros7.1 b2 , but looks like there is kernel 5.6.3, maybe i am wrong?
Top
 
captainproton
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 72
Joined: Sat Jan 12, 2008 3:01 pm

Re: CCR2004 packet loss over 10G Interfaces

Thu Nov 12, 2020 9:36 am

We were using ROS 6.46.8 for our testing.
Upgrading to 6.47.8 would be worth a trial, but somehow I am sceptical...
Top
 
morf
Member Candidate
Member Candidate
Posts: 183
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CCR2004 packet loss over 10G Interfaces

Thu Nov 12, 2020 10:45 am

I took counter samples from three of the seven CCR2004s I have in production and the TX drop rates for the busiest interfaces were:

* 1/23536 packets dropped (0.0042%)
* 1/76566 packets dropped (0.0013%)
* 1/104768 packets dropped (0.00095%)

These are all on 10G ports with 10G fiber links and ROS 6.47.2 or 6.47.3.
I'm sorry if I'm off topic. I can't find any real test results in CCR2004 anywhere. Please tell me how much traffic you have going through the most loaded CCR2004 and what is the load on the CPU ?
Top
 
markonen
newbie
Posts: 31
Joined: Tue Aug 11, 2020 4:28 pm

Re: CCR2004 packet loss over 10G Interfaces

Thu Nov 12, 2020 2:38 pm

I'm sorry if I'm off topic. I can't find any real test results in CCR2004 anywhere. Please tell me how much traffic you have going through the most loaded CCR2004 and what is the load on the CPU ?
Our CCR2004s currently do BGP (full table) and have a simple Raw firewall (conntrack disabled). So with BGP running and no fasttrack/fastpath, we can saturate the main 10G egress interface with web traffic (a few thousand simultaneous flows). One of the CPUs is pegged (the one doing BGP, this is ROS6) and the rest are well-utilized on traffic peaks, but not at 100%. Sorry, don't have more exact figures at the moment.

Overall we can just about serve our workload with these, and that's pretty amazing considering the price point. But really looking forward to testing a CCR2016 and the rest.
Top
 
torstorm
just joined
Posts: 5
Joined: Mon Sep 21, 2020 9:50 pm

Re: CCR2004 packet loss over 10G Interfaces

Thu Nov 12, 2020 7:36 pm

I am checking deep and I have discovered that packet loss is only on the SFP+ port s, using 10 giga module on the SFP28 ports we have no packet loss.

According with the block diagran https://i.mt.lv/cdn/product_files/CCR20 ... 200459.png the SFP+ and SFP28 ports have different bus to the PIPE
Thats strange.... I use Finisar FTLX8571D3BCL sfp+ modules, test 10Gbit bandwith and no errors. I also use complex firewall with multiple chains and processor is not loaded even 1/3 of 100%.
On SFP+ modules I use auto negotiation.
For switch i use Mikrotik CRS358 series switch. Maybe you have problems not on router, then on switch side. What switch do you have?
Top
 
captainproton
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 72
Joined: Sat Jan 12, 2008 3:01 pm

Re: CCR2004 packet loss over 10G Interfaces

Thu May 06, 2021 8:04 am

Hey there,

as Mikrotik released v6.49beta27 with a change described as
"switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device"
I gave it another try.

Our Core Router <---Mikrotik-10G-DirectAttatchCable---> CCR2004 <---10G-SM-SFP+---> our Ookla Speedtest Server

After a night of operation and ~3 TB of data speedtested, I couldn'd see any TxDrops on both interface, no more PL and no obvious performance issues. This looks very promissing.
I really hope this router is really fixed, can anyone confirm?

Maybe my test scenario was not complex enough since others report PL only with higher CPU load. But in my initial testing several months ago, PL even massively occured, when CPU load was very very low. So this still looks good to me.
Top
 
biziiac
just joined
Posts: 2
Joined: Mon Nov 30, 2020 12:13 pm

Re: CCR2004 packet loss over 10G Interfaces

Tue May 18, 2021 3:55 pm

Hi,

For me even the beta version has packet loss. But only when I have traffic over 1Gbps.
Hopefully this issue will be resolved soon.
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12580
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: CCR2004 packet loss over 10G Interfaces

Sat Aug 19, 2023 2:50 pm

Unfortunatelly,
SPAM
Top
Post Reply

Who is online

Users browsing this forum: tornadoro and 21 guests
  4. Other topics
  5. RouterBOARD hardware