We were in desperate need of additional ports at one of our smaller POP's and decided to give the CCR2004-16G-2S+PC a shot to replace a CCR1009-7G-1C-1S+. We're trying - hard - to move away from Cisco but running into capacity limitations far short of Mikrotik's published test results.

We perform very minimal NAT'ing on this router (<1mbps), have 0 filter rules and no queuing. It handles basic OSPF routing for about 1.5Gbps (peak)of traffic and the CPU of the new CCR2004-16G-2S+PC is running close to 90% CPU compared to the CCR1009 running at near idle. The attached graph shows the CPU usage transition from CCR1009 to CCR2004 at approximately 06:00 Friday and with connection tracking disabled (as a test) on the CCR2004 on Sunday @ 12:00.

It's not a true apples to apples comparison as this is looking at one CPU core (CPU0) but the average usage has jumped from roughly 20% to nearly 90% when making this transition. I've seen the articles regarding route-cache being disabled and causing higher CPU but this seem excessive and makes our transition to any new ARM hardware running v7 a downgrade in terms of performance, even on new hardware tailored for v7.

The 2nd screenshot shows a comparison in usage of 4 cores (# of cores in the CCR2004) - the other 5 cores in the CCR1009 were discarded when our NMS system recognized the new CCR2004, however the CCR1009 balanced CPU usage almost perfectly across all cores while the CCR2004 is spiking CPU0 to near full capacity (and limiting throughput) while other cores are running at 30%. What are we missing? How is Mikrotik getting 22Gbps of routing performance through this with no routing rules (512 byte packets)?

Mikrotik: Totally off subject but you've got to release DC power supplies for all your units that have modular power supplies. No WISP I know runs AC power for any remotely critical site.

Hello
you didnt provide us any info about what the router does, apart a very short description.
wich is a configuration (export with hide-sensitive)
a brief network diagram.
we dont have any info about your setup to discuss with.

tools profile
cpu resource print

What are we missing? How is Mikrotik getting 22Gbps of routing performance through this with no routing rules (512 byte packets)?

It's probably in your config. If you post it I can't guarantee I'd see it, but if you don't I will gladly guarantee that I will not see it.

When going from Cisco to MikroTik one thing to embrace is that a minor change in configuration on a MikroTik device can lead to you losing all hardware acceleration on the packet path without warning. MikroTik switches for example use general purpose ASICs and embedded ARM or MIPS chipsets to provide a feature set that's very close to what Cisco devices can do. A Cisco Catalyst 9k switch is essentially an x86 powered server with a dedicated ASIC (Cisco calls it UADP) for almost all networking related tasks. So it is vastly more powerful, costly and energy hungry.
Talking about routing, MikroTik accelerates that in the Switch ASIC. But most switch ASICs cannot contain firewall logic in their pipelines, or have specific requirements of how ports need to be structured to support the hardware acceleration. If you now make one change that needs to process even one packet outside of the ASICs supported pipelines (and RouterOS will let you do that, in Cisco devices unsupported configurations are often not valid or possible) the whole pipeline will move to the CPU and be constrained both by it's speed and connectivity back to the switching fabric. The way MikroTik combines the features from switch ASICs and CPUs (eg. crypto and packet filter acceleration in CPU, routing acceleration in ASIC) can very quickly lead to situations where you lose at least one aspect of an accelerated pipeline. On the other side, Cisco's UADP for example has programmable pipelines which is why they were able to add features such as MPLS to the ASIC layer after the fact. And from there on your switch could do MPLS at line rate.
Maybe that helps you understand where the differences are.

I'm almost certain your configuration is very different from what MikroTik uses for benchmarking. The way I understand MikroTik benchmarks is to compare different MikroTik devices against each other. I have found using that data to draw anything more than a rough guesstimate for real world use cases unfruitful.