We've got a routing situation where we need to route multi-100Mbps traffic.

Anyone doing this?

We are currently pushing 350Mbps peak and our Alteon and Cisco are giving us problems (needing reboots and failing to handle massive DOS attacks etc).

I've been pleased with the 100Mbps routing of Mikrotik in some of our stuff, but haven't pushed it through a GigE connection yet.

Just looking for comments on actual experience from people pushing/pulling more than 100Mbps.

we are running at peak about 420Mbit on our portwell based MT routers.
keep in mind if you want this kind of traffic...
use 64bit pci xpress intel networkcards
and xeon cpu's

it's this one

http://www.portwell.com.tw/ca-product.a ... l=nar-5072

we would like to deploy this one , but as MT doesn't do dual cpu...
we are still hoping they will soon..

http://www.portwell.com.tw/ca-product.a ... l=nar-7080
http://www.portwell.com.tw/ca-product.a ... =nar-7070e

ps we are not portwell resellers, we just love the combination MT + portwell... and the belgium companies we resell them to do to..;

Thanks for the info.

About how much are those boxes?
Does MT support the 2 line LED display on those boxes (not that it's important, but it would be cool!)

Does anyone know if MT works on AMD 64 bit processors?

from 2.9.8 if you go to console... you can find the Led display driver there...
not in winbox yet... there is a small bug with one letter... but this is getting fixed by MT.
select the correct driver there... and do ther rest in winbox if you like..
a fully configured one like we run costs about 1300 eur...
so in my opinion thats pretty cheap...
for running 2Gb fibers + 4Gb ethernet...
or 7Gb ethernet...

flash 128Mb 512Mb ram and a 2.8P4 cpu included...
where you from... maybe we can order to gether, so we can split shipping...
still hopeing that MT enables Dual CPU...
then we can kick out some other junipers and replace those to with custom Portwell MT routers...

That platform does sound very nice!

We are located in the United States (Utah) though.

Have you ever linked up a couple of these boxes and run TCP Bandwidth tests from one or two machines connected through them on either side?

That would be a good test I think of how much total traffic systems like this could handle.

I wish someone at Mikrotik would do this and make a report on Gigabit per second max throughput.

we indeed did some tests...
our findings are the following...

we did a bandwithtest from MT to MT both where NAR-5071-710 with 7 64 bit pci express networkcards... crossovercable.
we got 102Mbit stable traffic so we where verry disapointed...
we noticed that both boxes where running at 100% cpu load
so we touth damm no good.
then we did the test again but we on both boxes a dell server with gigabit networks cards connected with a crossovercable...
both running the MT bandwith test software...
now we where running about 350Mbit from dell to dell.
both where running server and client, and we did a tx test with UDP
for some reason if we did a duplex test from one to the other... the pc cpu would max out, and performance was bad...
so we did a TX test from A to B and a TX test from B to A
at the same time...
so we had exactly the same traffic in both directions on the firewalls....

330(350) TX and RX but cpu on the routers was still at 30%
so we made some more servers and put it with a switch on the same interface and did the test again...
we got in both directions 380Mbit , but now the pcu's where not maxing out on cpu and where only running the half of 380Mbit and still the routers where at 30-38% CPU

so we noticed this was probaly a bus problem....
but still 380Mbit in-out was pretty nice...
now we disconnected the switch and just put all the servers with crossover cables on the routers...
so every server had it's own interface...
no we where running about 840Mbit and now the cpu of both routers maxed out...

so conclusion... the gigabit interfaces max out at 380Mbit per interface...
but you can run more traffic on your router then that.
multiple interfaces make that you can use your cpu to the max...
and i'm pretty sure even more is possible with Xeon CPU's or even Dual Xeons if MT would ever support this.

also the bottle neck on the intel Gigabit cards is MT related.
i put some cards with the same chip in my linux box and i was running the same traffic until i enabled Jumbo-frames... then i got about 700Mbit interface to interface...
so if Mt would support jumbo-frames or tell me how to enable them..
i'm sure our boxes can do more...

PS keep in mind this was a routed setup, no mangles no firewall...
if you do this on your box, you'll take some cpu from the speed....

That was the information I was looking for.

Looks like we are going to look elsewhere as we are already at 350Mbps.
We need one interface to be able to do at least twice that full duplex.

Still, it is impressive that it can do 350Mbps for that price.

i'm sure it's not hardware related, it just those stupid jumbo frames...
i did the same tests with the fiber gigbit interfaces...
same values....
i just wished i knew how to get them to work...

Here is what we have tested about a year ago with a Xeon box and gig intel interfaces. The traffic was generated and received on two other Xeon boxes.

We have put 1000 rules on the box which would be the same as 1000 hotspot users basically. About 850Mb/s send and 750Mb/s receive -- 1500 byte packet, UDP. This is going through the router with two interface.

John

by the way, you will actually be able to see this test live in action at the Mikrotik User Meeting (http://www.mikrotik.com/mum)

any idea of implementing a Dual CPU based version...
to even push more traffic and Jumbo frames...

i would personaly get rit of all our junipers and cisco's in favor of MT
but with a Single xeon we still have not enouch power for all the traffic...

in 2.10 possibly

yes, you guys are great...
i can't wait for kicking out all of our junipers and cisco's...

would that be dual CPU, and jumbo frames???
or only dual CPU ??? of only jumbo frames...???

thxs guys

weren't jumbo-frames already supported on some cards (like Intel)?

Jumbo frames are only compatible on the local wire and not across the internet right? I assume if you turn on jumbo frames that you can't talk to another box using 1500 MTU ?

Sam

i had 2xintel desktop pro1000mt adapters routing between 2 networks. the mt server was a 900mhz duron with 384mb ram. i had 2 linux boxes, 1 on each network and ran iperf between them. i got 380mbps HDX throughput, mt winbox reported ~400mbps per interface, which is 800mbps total. at first i was confused because cpu was sitting nicely at 70%, then i remember i was actually doing 800mbps, which is essentially the max of the pci bus.

so i feel pretty certain i could get more than 2gbps total with a 2ghz pro and pci express bus and cards.

btw. this was running 2.8.26

Jumbo frames are "trapped" in one ethernet segment, and all devices (including switches) should support it and the "Jumbo frame rate set" should be the same.
The cheapest switch with jumbo frame I found in my country is Cnet cgs-500/e http://www.cnet.com.tw/product/cgs-500.htm which support jumbo frames up to 9.6kbytes.
Does Miktorik support jumbo frames and How I can enable it (I'm not sure but I think MTU variable will work here) ?
Will there be problems with bridging (I'm bridging 4 wireless interfaces on 4 vlans of the one ethernet and 100 mbit aren't enough).

P.S. I finnaly found Intel PRO/1000 MT (desktop version) on reasonable price and I think that soon there will be import of Marvel based chipset cars, soo I will try these 2 solutions, any experince here (especially I have interest in vlan support of those 2 cards)

Edit: Some theoretical points which should be foreseen:

Classic PCI is 32 bit bus working on 33 MHz:
32*33/8 = 132 MB/s (1056 mbit/s).
This is theoretical maximum of the PCI bus, you must count in noise and other factors and the throughput goes down.
Tip: Disable all unsused ports and peripherial on the router PC (sound cards, game ports, paraller port, USBs if you don't use them, the more devices on PCI bus the more lower throughput you will gain).

Different modifications of PCI gives the ability to run on 64 bit and 66 MHz and the maximum throughout rises up to (peripherial could be marked as PCI 64 compatiblе;
64*66/8= 528 MB/s (4224 mbit/s). But the boards which support this are pretty expensive and NICs too ...

The next step was PCI-X is 64 bit/66 MHz, and the latest revision PCI-X v2 uses both front and the rear tact of the wave (somethink like DDR a think). Soo the theoretical maximum of PCI-X v2 is
2*64*66/8= 1056 MB/s (8448 mbit/s) but with this technologie there is much bigger noise on the bus soo you will gain less performance.

Soo now the most fashion technologie is PCI Express :
The biggest priority of PCIe is the point-to-point architecture. Instead the whole bus is atached to the chipset in PCIe every port is atached to the chipset. The other important thing is that PCIe uses "highways" soo PCIe x1 uses 2 "highways" for upstream and downstream each one at 2.5 gbit/s (5 gbit/s on both directions).
Soo one card on PCIe x1 is should be faster than the whole PCI Bus. The number of slots and the type of slots (there is PCIe x1,x2,x4,x8,x16,x32) depends on the motherboard vendor and chipset in use ...
This is the resume of the PCI/PCI-X and PCIe for more info the vendor:

http://zone.ni.com/devzone/conceptd.nsf ... 37006DFDB3 (in English)

http://www.hardwarebg.com/reviews.php?read=49&page=1
(in Bulgaria, this is not an translation of the upper link this is another review)

Jumbo frames are "trapped" in one ethernet segment, and all devices (including switches) should support it and the "Jumbo frame rate set" should be the same.
The cheapest switch with jumbo frame I found in my country is Cnet cgs-500/e http://www.cnet.com.tw/product/cgs-500.htm which support jumbo frames up to 9.6kbytes.
Does Miktorik support jumbo frames and How I can enable it (I'm not sure but I think MTU variable will work here) ?
Will there be problems with bridging (I'm bridging 4 wireless interfaces on 4 vlans of the one ethernet and 100 mbit aren't enough).

P.S. I finnaly found Intel PRO/1000 MT (desktop version) on reasonable price and I think that soon there will be import of Marvel based chipset cars, soo I will try these 2 solutions, any experince here (especially I have interest in vlan support of those 2 cards)

Edit: Some theoretical points which should be foreseen:

Classic PCI is 32 bit bus working on 33 MHz:
32*33/8 = 132 MB/s (1056 mbit/s).
This is theoretical maximum of the PCI bus, you must count in noise and other factors and the throughput goes down.
Tip: Disable all unsused ports and peripherial on the router PC (sound cards, game ports, paraller port, USBs if you don't use them, the more devices on PCI bus the more lower throughput you will gain).

Different modifications of PCI gives the ability to run on 64 bit and 66 MHz and the maximum throughout rises up to (peripherial could be marked as PCI 64 compatiblе;
64*66/8= 528 MB/s (4224 mbit/s). But the boards which support this are pretty expensive and NICs too ...

The next step was PCI-X is 64 bit/66 MHz, and the latest revision PCI-X v2 uses both front and the rear tact of the wave (somethink like DDR a think). Soo the theoretical maximum of PCI-X v2 is
2*64*66/8= 1056 MB/s (8448 mbit/s) but with this technologie there is much bigger noise on the bus soo you will gain less performance.

Soo now the most fashion technologie is PCI Express :
The biggest priority of PCIe is the point-to-point architecture. Instead the whole bus is atached to the chipset in PCIe every port is atached to the chipset. The other important thing is that PCIe uses "highways" soo PCIe x1 uses 2 "highways" for upstream and downstream each one at 2.5 gbit/s (5 gbit/s on both directions).
Soo one card on PCIe x1 is should be faster than the whole PCI Bus. The number of slots and the type of slots (there is PCIe x1,x2,x4,x8,x16,x32) depends on the motherboard vendor and chipset in use ...
This is the resume of the PCI/PCI-X and PCIe for more info the vendor:

http://zone.ni.com/devzone/conceptd.nsf ... 37006DFDB3 (in English)

http://www.hardwarebg.com/reviews.php?read=49&page=1
(in Bulgaria, this is not an translation of the upper link this is another review)

Great information aviper.

http://www.belwave.com/MikroTik_GigE_Te ... edited.bmp

Here is a snapshot of a couple P4 3GHz MT routers running Intel NICs and basic 32bit Intel motherboards. This is actually a new hub of ours being turned up and therefore the reason for the editing of some specific information. Regardless it illustrates conservatively what can be achieved with the MikroTik OS and Intel hardware. This test was run from the routers themselves and as such consumed resources.

I agree the max for a 32bit PCI bus is somewhere less than 1GB. Our tests have shown anywhere from 830Mbps to 700Mbps total...we have seen varying results depending on the specific NIC used in the test.

I'm very interested in MikroTik supporting dual processors, but I was told by the ImageStream guys that dual CPUs would offer little improvement on a single connection rate. So, would the improvement be more readily realized on a multi-interface router passing traffic in multiple directions?

Best,

Brad

Soo to continue the theme:
First the term I was missing was lanes (which I wrote "highways").
Soo if you are interested in different chipset supports here is a brief of the sittuation:
PCI Chipsets:
All Via chipset uses: »VT8251, VT8237x, VIA VT8235 south bridges which supports only PCI bus.
Nvidia: Nforce 1,2,3 supports only PCI bus
Intel ich5 Soutch Bridge and earlier versions support only PCI Bus

PCIe Chipsets:
Intel ICH6 (i915)
20 lanes one x16 (or 2*x8 ) for VGA and 4 PCIe x1 (500 MB/s each)
Intel ICH7 (i945,i925,i955)
22 lanes one x16 (or 2*x8 ) for VGA and 6 PCIe x1 (500 MB/s each)
Nforce4/Nforce4Ultra
20 lanes one x16 and 3 PCIe x1
Nforce4SLI
20 lanes flexiable (depends on the MB manifacturer)
Nforce4SLI16x
For Intel 40 lanes flexiable (depends on the MB manifacturer)
For AMD 38 lanes flexiable (depends on the MB manifacturer)

PCI-X/PCI 66 Chipsets:
I saw only Intel Server Based chipsets where the varaiaty is big. I prefer Intel and Asus MB, soo if you are interested see the manifacturer sites for more info