I have several routers that seem to hit a brick wall around 30 megs of TCP traffic using the bandwidth test tool between routers to test. CPU utilization never peaks above 40%. It makes no difference what type of Interface queue is being used. NAT, connection tracking, enabled or disabled seems to make no difference. The only notable difference I see is if I increase the connection count to 100 I can get the speed up to around 40-45 megs. UDP connection shows right around 100 megs each direction. Very straight forward configuration - DHCP, couple static NAT's, no shaping, layer 7 or firewall rules.

I've seen this problem crop up over and over again but never any real resolution that I can find. Has anyone figured this out before? Any ideas?

Don't use btest on the router itself. It is cr*p. The CPU in most routers is not suitable for all the tasks required to test TCP speeds.

If you need to do a decent test just get 2 ordinary PC's with a modern OS like Windows 7 or better, Linux with 3.x kernel or Mac OS X 10.9 and do a bandwith test using 'iperf'.

That would hold water if the CPU was spiking anywhere near the maximum but it isn't.

That would hold water if the CPU was spiking anywhere near the maximum but it isn't.

Maybe.. still, try iperf. Routers are always better in handling traffic going through than traffic to/from the router itself.
And of course, that is what you want to know the performance of.