We are using two 3 Ghz Quad Core machines with 2 Gb of RAM as our border routers. Each machine has two interfaces, which are PCI-E Intel Gigabit cards. CPU Load never exceeds 30%. However we are unable to get more than about 650 Mbit/s full duplex on the links. We are unsure if this is our issue or issue on the side of upstream provider. So this post it to confirm that it is possible to push more than 650 Mbit full duplex through a machine like described. Any replies appreciated.

OK, not a single person is using that setup? If you are using any other setup for high-bandwidth routing which is it?

quad core? I think, 30% means that one core is used fully (25%), and that process is limiting the bandwidth.

MikroTik plans to add multicore optimizations in v5 or v6 of ROS

for now, I think, you should move some QoS/Firewalling to another router(s)

quad core? I think, 30% means that one core is used fully (25%), and that process is limiting the bandwidth.

MikroTik plans to add multicore optimizations in v5 or v6 of ROS

v5 or v6

i

Thank you for the reply. There is no QOS on these routers and firewalling is very rudimental. But here
http://www.mikrotikrouter.com/speedtests.asp they claim to do almost 6 Gbps (okay, that is in one direction on one card, for us it would be 1.5 Gbps both directions on two cards - still far more than enough)? And RB1000 claims to do 400,000 pps which is also more than enough in our case as well?

p4 style supermicro, intel pcie nics, full gigabit both ways:

http://h1x.com/mt/gigabit/gigabit.html
and
http://h1x.com/mt/gigabit/gigabit2.html

Turn off connection-tracking and see if that helps. Also disable USB and any other onboard devices you are not using. The above was done with 2.9.24.

Thank you for the input. This is however only half of what is needed - as we need full gigabit both ways on two cards, not just one.

quad core? I think, 30% means that one core is used fully (25%), and that process is limiting the bandwidth.

MikroTik plans to add multicore optimizations in v5 or v6 of ROS

I started to look into that suggestion and amazingly it seems that this is true. Router caps exactly when CPU reaches 25%. However starting some other activity (e.g. disable/enable all rules in firewall - even though there are not many) increases CPU usage beyond that point, which means it happens on another core. However all routing is done on one core.

We got RB1000 for a test today. Put it in place of one of the border routers, disabled conntrack, empty firewall, queues, etc. The device was only able to pull around 55000 pps or 350 mbps, less than our Quad-Core machines and far less than the specified 400000 pps. Screenshot is attached. I am not sure what we can do now - there some 3.6 ghz CPUs out there, maybe we should try one of them.

Can somebody confirm or deny my findings?

far less than the specified 400000 pps

I think, it's more bandwidth limitation, not pps. on one of our core router we have ~800Mbps, 100kpps. but when with ~400 Mbps load I run bandwidth test with small udp packets to this router, I can easily get ~200kpps. just haven't tested more =)

far less than the specified 400000 pps

I think, it's more bandwidth limitation, not pps. on one of our core router we have ~800Mbps, 100kpps. but when with ~400 Mbps load I run bandwidth test with small udp packets to this router, I can easily get ~200kpps. just haven't tested more =)

What are the specs of that router? What functionality is enabled on it?

Core2Duo, integrated Intel NIC, a few firewalling (conntrac is enabled)

There is a lot more to it than just simple CPU usage. PCI buss design and limitiations, IRQs, testing application, etc. So just buy putting some hardware togeather, don't mean its a good design. There is lots more to it than just build it and it will work!

We user Intel server motherboards (SH3200), Quad core 3ghz CPU's, Intel PCI-E NIC's. What could possibly be wrong here hardware wise?

There is nothing wrong with what you have setup however things like which network adapter or which motherboard can make a difference. The OS underneath is linux and unless mikrotik has changed anything in the drivers for performance you're going to find that different cards/drivers will have better performance. Some drivers for the motherboard will be better then others. A server cpu may be better for what you're doing. Instead of using a core2duo you could use a single core xeon or something that will give you more power in a single cpu

So, to rephrase you, we could get more out of 3.6 Ghz P4 (if I'am not mistaken 3.6 Ghz is the fastest one can get)?

In short yes. If you are having issues with maxing out your cpu then getting a faster cpu will help you but it may not help you enough. I think you need to look at the bigger picture and that is you need to figure out where you bottlenecks are. It could be one of many things like cpu, memory, motherboard. If I were you I would ditch the P4 and go with a much beefier processor like Xeon or the newer i5 or i7 processors and in the process you'll get a performance boost also because you'll probably be running a faster motherboard with faster ram.

Try to run more than one BT on the same router, open 4-6 console terminal and run Bw test in each console, whith this test on ethernet you can see maximum speed for your router.

Core2Duo, integrated Intel NIC, a few firewalling (conntrac is enabled)

Could you tell you conntrac configuration? Do you use NAT on mikrotik?

hello, i have one question if i have: Bandwith 30 Mb > RB 450G > switches with management > clients ; do you think this configuration support more than 100-200 clients without crashes ? ( RB 450G with firewall, bandwith limited over mac, dhcp server, and other configuration). you guys think it will work ?
Thank you and sorry for my bad english.

Core2Duo, integrated Intel NIC, a few firewalling (conntrac is enabled)

Could you tell you conntrac configuration? Do you use NAT on mikrotik?

no NAT at all

Do you use conntrack for traffic flow?

650 Mbit/s full duplex on the links

That is the value I've seen in all my (netio-)tests with cheap Realtek or Yukon chipsets under Linux.

There was an extremely interesting presentation on multi 10gigabit routing using Linux based routers given at the 2010 New Zealand Network Operators Group conference.

The presentation covers chipsets, architectures, advantages of NUMA, Kernel status in regards to routing throughput.

Most of this is relevant to RouterOS on x86 and is clearly presented by two highly knowledgeable researchers.

It is the first video on http://www.onfarm.net.nz/nznog.html



Regards,



Andrew

Do you use conntrack for traffic flow?

conntrack is enabled, and we are collecting NetFlow information from our routers

I hope, I'll soon test LACP on 2-port GigE Intel card - will see if it can bring the throughput on higher level =)

Do you use ospf on your NAS?

I did not realise that LACP is supported by RouterOS...

Do you use ospf on your NAS?

sure

I did not realise that LACP is supported by RouterOS...

but it is, although my previous try (early v3 and some old Zyxel switch) was unsuccessful =)

What configuration do you use? Do you using NSSA like describe here or like on mikrotik wiki and manual using networks and passive ospf for users connects?

omg, just saw it! direct routing, no nat, a few firewall rules (one is matching against address list with ~1000 items), NetFlow:

Chupaka, I am confused ? What have you just seen?

Looks like some nice throughput....

exactly - that router have only one NIC =)

damn, thats impressive.