Not even so so..

Today made necessary change for collabsing hardware (Xeon), but will return it immediately.
CCR1016-12G latency jitter is just terrible from constant ,3ms to following on picture
1880mangles, 350firewalls, 200Mbit traffic. Unbalanced load of CPUs touching by one of those all the time about 85percent, but average 20percent

This is ping thru router on local network gbit network, before it was constant ,3ms
--- 10.10.9.253 ping statistics ---
947 packets transmitted, 947 received, 0% packet loss, time 948371ms
rtt min/avg/max/mdev = 0.280/1.756/23.996/2.478 ms

Just ogrish jitter

change made hlaf past one..

What is wrong ??

Yesterday I purchased CCR1016, hope for a positive result.
Why mikrotik promises do not match the reality?

Not even so so..

Today made necessary change for collabsing hardware (Xeon), but will return it immediately.
CCR1016-12G latency jitter is just terrible from constant ,3ms to following on picture
1880mangles, 350firewalls, 200Mbit traffic. Unbalanced load of CPUs touching by one of those all the time about 85percent, but average 20percent

This is ping thru router on local network gbit network, before it was constant ,3ms
--- 10.10.9.253 ping statistics ---
947 packets transmitted, 947 received, 0% packet loss, time 948371ms
rtt min/avg/max/mdev = 0.280/1.756/23.996/2.478 ms

Just ogrish jitter

change made hlaf past one..

What is wrong ??

What hardware were you using before CCR?

Not even so so..

Today made necessary change for collabsing hardware (Xeon), but will return it immediately.
CCR1016-12G latency jitter is just terrible from constant ,3ms to following on picture
1880mangles, 350firewalls, 200Mbit traffic. Unbalanced load of CPUs touching by one of those all the time about 85percent, but average 20percent

This is ping thru router on local network gbit network, before it was constant ,3ms
--- 10.10.9.253 ping statistics ---
947 packets transmitted, 947 received, 0% packet loss, time 948371ms
rtt min/avg/max/mdev = 0.280/1.756/23.996/2.478 ms

Just ogrish jitter

change made hlaf past one..

What is wrong ??

What hardware were you using before CCR?

Some Xeon 2GHz

any more details than "Some Xeon 2GHz"

any more details than "Some Xeon 2GHz"

gosh.. keep on subject, what it has to do with CCR?
XEON 2 cores 2,9GHz, 2GB RAM with built in 4 Gbit ports What exactly is that I dont know.. I can take a look once returning it instead of CCR

any more details than "Some Xeon 2GHz"

gosh.. keep on subject, what it has to do with CCR?
XEON 2 cores 2,9GHz, 2GB RAM with built in 4 Gbit ports What exactly is that I dont know.. I can take a look once returning it instead of CCR

I haven't purchased a CCR yet and the question I was asking what hardware would CCR replace or put another way should I be considering other hardware than CCR - a difficult question and very much depending on application but sometimes it's better to get a older product with a proven track record of reliability and average performance than a new product with high performance but has low reliability record – this comment applies to all new products and not just the CCR?

We have one CCR for testing. It died and we get it repaired by MT.
Since then it runs flawless running in our OSPF core holding one minor connection.

We put much hope in this beast at it has the hw to hold 4 licensed links per site
and should be fast enough to replace our gigabit switches by simply bridging ports.

At the moment we proceed slowly as we are not sure ROS6 is stable enough and
how the MTBF of this HW is.

Any comments from MikroTik ? Or silent overcome again ?

OK, nothing new under sun..

And now try to guess, which part of graph is CCR 16core running 25 percent top load and what is Xeon working in on 75percent top load..
Guys do the CPU balance better way, then I believe it could be really bulls eye, until that I spent 3K USD not the right way

I repeat it is approximately 400firewall rules, 250 NAT rules and 1890mangles for queue tree

Any comments from MikroTik ? Or silent overcome again ?

Comments on what exactly? Please clarify the question. This topic includes a wide variety of observations.

Any comments from MikroTik ? Or silent overcome again ?

Comments on what exactly? Please clarify the question. This topic includes a wide variety of observations.

I mean: Why is your CCR routes so cute and fun.... NO. I mean what is the problem, that your platform brings me such heavy latency jitter and huge latency at all. I presume, that the problem is bonded directly with a instant load of 1 of cores on CCR. The load is pretty unbalanced,where average load is 25 percent, but 1 or 2 cores are running 70-90percent of load randomly, while rest other 2 cores are having load 1-5 percent.

Any comments from MikroTik ? Or silent overcome again ?

Comments on what exactly? Please clarify the question. This topic includes a wide variety of observations.

I mean: Why is your CCR routes so cute and fun.... NO. I mean what is the problem, that your platform brings me such heavy latency jitter and huge latency at all. I presume, that the problem is bonded directly with a instant load of 1 of cores on CCR. The load is pretty unbalanced,where average load is 25 percent, but 1 or 2 cores are running 70-90percent of load randomly, while rest other 2 cores are having load 1-5 percent.

+1
When there is a load impossible to manage through the CCR Winbox

I read this topic, and the issue is still not clear. Basically your issue is that sometimes, one ping reaches latency of 20ms?

I read this topic, and the issue is still not clear. Basically your issue is that sometimes, one ping reaches latency of 20ms?

Basicly say yes.
If I will compare it with x86 platform there is no latency jitter at all - under all circumstances thruput thru router takes about 100us up to 95 percent load under same setting and 300-350Mbit load.

Which is normall and for example RB1200 which we are having on one location where we need it to change as soon as possible because of 100CPU time to time load, but latency grows fluently approaching 100percent of CPU time, but the jitter is still like 1-2ms (long term average 2 ms, but jitter makes it 2,3 -4 ms randomly, and under heavy load 7-9 ms randomly).

By CCR with a same setting as X86 it behaves strange way.
As a regular user I cant see differences, but once I have latency statistics i got miserable latency jitter like 10-30ms. So instead of fluent 1ms ping I am getting results as in ms is:
1
4
3
2
1
4
13
2
15
25
2
4
3
1
4
3
4
etc..

The average load of all CPU cores is 25 percent, but once you look on resources and CPU use, and you will allign CPUs in row, you will see, that 70 percent of load are splitted between 3-4 cores and rest is idling about 5-30percent. An as you watch the single largest core load, every time it jumps toward 80 percent on single core, latency jitter appears. So it has to do with load balance between cores i believe. Once you will sack off mangles, it stabilises, but only because lower overal load.. By other words, CCR has good enough result up to 15 percent average load of all CPU..

It is no secret, and it has been discussed in other topics, that some few features are not yet optimized for multicore. Some specific firewall matchers etc. Upcoming versions will solve this.

If one packet needs to be processed by another core, this will happen as you see it. Adding full multicore optimisation for the remaining few things will fix your problem.

It is no secret, and it has been discussed in other topics, that some few features are not yet optimized for multicore. Some specific firewall matchers etc. Upcoming versions will solve this.

If one packet needs to be processed by another core, this will happen as you see it. Adding full multicore optimisation for the remaining few things will fix your problem.

Hi
Did you fix problem with multicore load balancing?

Hello,
I think this discussion is exactly about the problem I'm dealing with.
My CCR started to be unstable and after come crash and reboot, there was only "Starting services" on the LCD, so i had to replace it for a while with my backup device UBNT EdgeMAX PRO router.
Can you please explain the difference between pings? It is just a direct 1Gbps line. One meter of cable fron the server to the router. Same server, same cable.
Now I have the router back, but I'm about to leave the UBNT router as a main router and the CCR as a dissaster replacement.
As I can see, I'm not the only one, who can see the problem with delays.

The CCR performs poorly with virtual networks but performs very fast on physical networks. Ping times significantly increase using virtual networks (internal bridge routing, VPN, etc) while physical routing remains really fast. Even VPN is unstable at full loads and disconnects every few minutes when at full load. Its almost as is if the mesh network of the TILE CPU has high latency when the latencies on the specsheets are in nanoseconds.

The CCR performs poorly with virtual networks but performs very fast on physical networks. Ping times significantly increase using virtual networks (internal bridge routing, VPN, etc) while physical routing remains really fast. Even VPN is unstable at full loads and disconnects every few minutes when at full load. Its almost as is if the mesh network of the TILE CPU has high latency when the latencies on the specsheets are in nanoseconds.

The picture shows the ping delays on gigabit ethernet with no VPN or virtual lan. Firewall just says "Ok, I'm here." CCR is just responding slower than my grandma.

@ SystemErrorMessage

In your experience does the CCR have issues with bridged physical ports?

@ SystemErrorMessage

In your experience does the CCR have issues with bridged physical ports?

Hi, the issue is not related to the physical ports. I received a new/replaced peace of HW and the behavior is the same. I don't use bridges.
You can see on the picture, that the pings are high for Mikrotik. During the time, UBNT EdgeRouter PRO was there, tha pings were just ok.
I can provide you with my configuration, if needed.

jaroslavhala

I do not have any insight into this issue but I have a similar observation from ubiquiti wireless devices vs mikrotik wireless. Attached is a smoking graph where the switch from a Nanostation M5 to a Sextant5 is apparent.

jaroslavhala

I do not have any insight into this issue but I have a similar observation from ubiquiti wireless devices vs mikrotik wireless. Attached is a smoking graph where the switch from a Nanostation M5 to a Sextant5 is apparent.

Screen Shot 2014-12-01 at 12.40.16 AM.png

Thank you for you comment, but there is a little missunderstandig.
The pings go from Linux gentoo server(Dell) to the router. The pings were much lower(expectable), when I was using x86 MikroTik(half year ago). When I purchased the CCR to replace it, the pings went this high. The configuration did not change. Once the CCR broke(few weeks ago), I replaced it for the time being with the cheaper UBNT router. The pings were just ok until I placed back the MikroTik CCR.
My question is, if there is someone else experiencing this kind of issue and how to fix it.?

Are the responses going THRU the router also affected? Why are pings to the router so important?

Are the responses going THRU the router also affected? Why are pings to the router so important?

That is the point!
Pings to the router are mentioned because they should be clean in every case.
All the pings are affected.
Please take a look at the ping to the provider's GW.
(The peak ping after change to the UBNT router is caused by lack of shaping policy on the router. It took me some time to configure it properly.)

Have you all taken a read of:

http://mum.mikrotik.com/presentations/RU14/megis.pdf

I remember reading it just after Mikrotik released it and it was so useful for me planning my CCR Configs and letting me know some of the limitations / design input that helped me.

Also I see references to Queue Trees in your comments, I am not sure if that has been fixed however the Queue Trees was one of the features that was clearly identified as not multicore clean / optimised.

As the PDF above points out each queue has to be assessed and if the queue tree has a parent queue then every packet would need to be handled by the core dealing with that parent queue and that would provide a bottleneck.

CCR's work amazingly for me in my network and give me great results. However as with anything it needs planning and deployment care to make best use of the system.

The TILE Architecture is a major departure from the x86 you have replaced and with the code as it is currently the best performance needs a config designed for it.

Regards
Alexander

I am in the same boat as Alexander.

Our CCR's are performing well, and we are pushing them really hard (MPLS, VPLS, OSPF, iBGP, eBGP, Hundred of Thousands PPS, VRF's)

It is normal that traffic to a router interface will have random latency. We see this on our Cisco and Juniper routers too. It is the traffic going through the router that is important, and through the CCR's the latency is extremely low.

The main issue we face with the CCR is the inability to forward TCP traffic at greater than 1Gbit per "connection".

Bridging works fine entirely and is a bit funny on how many different things you can attach to it. VPN needs a lot of work to get working fast and stable.
Latency from bridging one physical port to another is minimal and really good but bridge does use CPU and is subject to random spikes.
Because my network relies on a lot of virtual layers firewall generalisation is important. Ive had to spend a lot of time trying to adapt really good firewall rules and examples just to work with my config. I dont have such a thing as WAN/up port for example. Many routerOS config examples are made for edge networks and not core networks.

As long as the latency from the CCR is equal or better than my servers dedicated intel NICs latency when using linux than i am happy.

I am in the same boat as Alexander.

Our CCR's are performing well, and we are pushing them really hard (MPLS, VPLS, OSPF, iBGP, eBGP, Hundred of Thousands PPS, VRF's)

It is normal that traffic to a router interface will have random latency. We see this on our Cisco and Juniper routers too. It is the traffic going through the router that is important, and through the CCR's the latency is extremely low.

The main issue we face with the CCR is the inability to forward TCP traffic at greater than 1Gbit per "connection".

How much % of the CPU do you get to utilise for your networking? While a single connection may have a limit it may be advantageous when facing attacks. How many PPS before your CPU usage is full? The specs on mikrotik say some MPPS but i am curious to how fast it works in a heavy practical environment. In my case it is mainly limited by virtualisation and logic.

It was also funny stress testing the CCR with different things. At times windows networking would fail till reboot and doesnt support load balancing.

I do hope mikrotik would at least get metarouter working on CCRs even if it would use one core per VM and have each script that runs using a different core. I tried running 2 different scripts doing loads of simple math and it would not use more than 1 core for both scripts. Scripts can use more than 1 core when utilising other components but the scripts themselves all are stuck to the same core. I know mikrotik doesnt implement power saving so i see no harm in keeping all the cores a little busy rather than idling most of them.

Have to admin i have been getting strange ICMP's on my ccr1036. my monitoring machine zabbix pings through these to different devices.

I can see 10ms latency .... using fping send 2 avg result (thats not good).

I am pretty sure it was a bit more stable at 6.19 running 6.22

Have to admin i have been getting strange ICMP's on my ccr1036. my monitoring machine zabbix pings through these to different devices.

I can see 10ms latency .... using fping send 2 avg result (thats not good).

I am pretty sure it was a bit more stable at 6.19 running 6.22

Exactly. I need to monitor the wireless network. I can see no delay ok 433 RBs I have, but CCR makes the ping monitoring really unreadable.

Hello guys, there is a huge I'm sorry i need to tell you here!
I need to appologise for my doubts.
The server did gather many information and data from MikroTik, what was not in case of UBNT router. The server had a serious issues with IO after the replacement of SCSI disks with SATA disks. The load caused by IOwait was the reason for high pings.
Thank you all for your time. I'm sorry I missused it for something like this.

Hello guys, there is a huge I'm sorry i need to tell you here!
I need to appologise for my doubts.
The server did gather many information and data from MikroTik, what was not in case of UBNT router. The server had a serious issues with IO after the replacement of SCSI disks with SATA disks. The load caused by IOwait was the reason for high pings.
Thank you all for your time. I'm sorry I missused it for something like this.

Awesome that you took the time to report back!

jaroslavhala

I do not have any insight into this issue but I have a similar observation from ubiquiti wireless devices vs mikrotik wireless. Attached is a smoking graph where the switch from a Nanostation M5 to a Sextant5 is apparent.

Screen Shot 2014-12-01 at 12.40.16 AM.png

Hello kozmonov,

could you tell us more detailed information on what setup you had on the Sextant and what on the Nanostation (wireless protocol,chains, width,distance,bridge/routing)?