Is anyone using MikroTik with 10Gbps connections? We're looking at possibly using Intel based 10Gbps SFP+ cards from HotLava. Is there some sort of theoretical maximum of bandwidth that Mikrotik can route or is hardware the limiting factor?

I would think it would be hardware, along with whether or not RouterOS currently supports that card or 10g at all.

The problem with 10 Gbps is that even if the system is able to use one processor core for each Network card queue there is not a lot of power dedicated to this stream.

Today (and since about 12 years) processors have only about 3 Ghz frequency, this is the limitation. As soon as you will try to do advanced filtering or tunneling (IPsec, Level 7 filtering...) bandwith will slow down.

Ask Intel to release new 100 GHz procesors, and wired speed routing at 10 Gbps will be possible with Router OS. But i'm afraid they prefer to sell DSP or FPGA chips for this market segment, instead of killing it releasing a 100 GHz x86 processor.


Today only dedicated FPGA or DSP based hardware can process 10 Gbps or more at full wire wire speed. Routerboard hardware is limited to 3.2 Gbps (ideal case) for the higher end model.

X86 seems not a very powerfull platform with Router OS, as it seems users cannot get more than 800 - 900 Mbps total throughput.


So i will say that it's difficult to use linux only based solutions for carrier class routing / filtering at 10 Gbps or more regardless if you are using Mikrotik or similar linux based solution without hardware acceleration.

If you need a router with two or more 10 Gbps ports, there are good chances you will get packet loss as soon as the trafic rise in the Gbps and more level. This is not tolerable for today carrier class routing, with VoIP and Video trafic.

http://www.iis.se/docs/10G-OS-router_2_.pdf

ROS v5 has some improvements in multicore area - you should try it on 10G =)

is SMP really going to help with a single card pulling 10gbit? I cam imagine it would help with multiple 1gbit interface where it can load balance the work but even with TCP offloading done on the card it will be putting quite a strain on the CPU

well, also RPS is going to help

I don't have the hardware to test this, but i would be curious to know if it's possible to simply transfer 10 Gbps of TCP trafic between two 10 Gbps ports and Router OS on X86 hardware.


I think as well that SMP can help for multiple 1 Gbps ports, but 10 Gbps seems really fast for 3 GHz processor core.

10 Gbps is about 1.6 GBps. This mean that one core at 3.2 GHz has only 2 clock cycle to manage 1 Byte of trafic...

I know this is a very bad approximate, because X86 processors do have SSE instruction set, and there are some helps like DMA PCI bus transfers, but this simple calculus show that it will be never possible to do advanced filtering and routing with 3.2 GHz non network optimized processors at 10 Gbps.


If someone tested this it would be fair to share it on the forum and give the model of the hardware tester set used to check packet loss.

I'm interested as well in jitter and latency results at 10 Gbps on X86 hardware.

any cheap 10Gbit cards you can recommend for testing? Maybe there is something new and popular we have missed

Seriously did I wake up and miss the years where everyone in the world got FTTH or something? Who around here is seriously pushing enough traffic to need a 10gbit NIC that is going to trust some x86 box to do it's work?

Still, why not prepare for the future ? Any recommended cards we could add support for ?

Normis are you say you can route 10gbit thru a ROS device?

Have the interface up is one thing, being able to use it is another. Is MT looking at doing custom ASIC's for routing and packet inspection now?

Anyway, take a swing at Intel's 82598EB. Its Copper based but the Intel cards already have linux drivers that try and spread the load a bit (SMP) I'd like to see the lab you use to generate 10gbit of traffic!

There is no problem for the 10 Gbps test LAB. I would recommand the new EXFO "low cost" FTB-1 tester with the Copper module FTB-860G.

You'll need a second unit with a FTB-860GL module for loopback.


It can support 10 GigE LAN or WAN by software option.



It is a wire speed 10 Gbps Ethernet tester. You will need two units to do a real bidirectionnal test.


EXFO do as weel a 100 Gbps tester. I've seen it pushing 100 Gbps of trafic without packet loss. So 10 Gbps is not a problem at all for modern testers.


The good thing with EXFO is that they support EtherSAM testing, so it is possible to test QOS directly with multiple streams at the same time.

It is really better than a simple RFC2544 test.

Whats the price bracket on those things, Seem them around downunder and the decent ones are $10k+ for gigbit testing.

I cant imagine MT dropping $20k to test 10gbit cards that about 5 clients are going to use lol

No i think the FTB-1 is under 10 000 $. And this kind of hardware can be rent by week.


Wire speed Gigabit Ethernet testers can be found for 1500 - 3000 $ on the second hand market. I bought one a few monthes ago for 800 €. (i know i was a bit lucky).

I have one main unit and one reflector unit, so i can measure round trip time and packet loss at wire speed, but only at Gbps speed.


There is no real second hand market yet for 10 Gbps testers, so renting or buying new ones are the only options except if you are very lucky in finding one second hand.

I think that the market for 10 Gbps and Router OS is really higher than 5 clients. Mediumly sized providers (1000 - 30 000 clients) can be interested in lowering their cost using Router OS (or alternatives) if they can have wire speed at 10 Gbps without packet loss and a reliable BGP engine in the router.

As soon as the number of clients rise at something higher than 4000, Internet trafic can rise higher than 1 Gbps. So you need at least one 10 Gbps link with at least one tier one or tier two provider to avoid saturations.

This is for France, in countries where "You Tube" is used by everyone, trafic can be 10 times higher...

And this will rise fastly because FTTH deployement is planned in most european countries (except ireland :=( )

Do not forget that big providers are using today 40 Gbps links, and are testing 100 Gbps links for tomorrow use in a couple years.

*If* They can have wirespeed 10gbit, Which means being able to push 10gbit of routing or packet marks or even plain old PPPoE traffic.

I get your point that the traffic is coming but I really doubt ROS is going to find a hardware platform that's going to allow that. The CPU's simply don't allow for it and SMP isn't going to help too much. You need co-processors to offload the highly parallel operations to do for things like routing or packet inspection.

Getting the link up is only the first part, no point in that if you cant push thru the router. But hey MT here's a challenge, Prove me wrong, Show us 10gbit of traffic running thru one of your routers

It will be possible in a near futur, because FPGA chips do allow this for a few $ since at least six years.


We simply need to wait that programmers learn HDL language coding, and translate Linux network code inside those high speed logic and parallel processing circuits to allow 10 Gbps wirespeed or even 100 Gbps wirespeed.


Actually, Small and talentuous companies like Mikrotik are using ready made network processors or switching circuits, this is the limitation.

But as soon as they'll be able to program themself FPGA chips for marking, filtering and routing they will be able to enter in competion with Cisco, Juniper and other Brocade or similar manufacturers, at a fraction of their cost.

Good points but still it's a heavy investment no matter how you look at it either in time or in money and if you're handling 10gbit of traffic you're really past MT's level.

After asking to EXFO, i can confirm that the FTB-1 Ethernet tester with 10 Gbps copper module 860GL is under 10 000 €.

IPv6 testing is possible as an option.

The 860G and 860GL copper modules have only one 10 Gbps port. So you'll need two testers to test a router bandwith.

For 1 Gbps testing, it's possible to use the 860G module, it has two 1 Gbps Ethernet copper ports, so the device under test can be connected between those two ports.


But for 1 Gbps testing there are cheaper alternatives on the second hand market.


The FTB-1 can be rent in europe. Can be a cheaper alternative to test a device before going to production or during development.

Interesting to note I guess in our corner of the world these things have a bigger markup

I don't think. You should verify. The FTB-1 is less costly than previous 10 Gbps testers. It's a new product just released.

We've just recently had contact from a Testing and Measurement Devices in NZ, 100mbit tester was $4k USD, 1gbit was $9k

I just did some routing tests on the RB450G, and i have serious problems with version 5.0 RC4.


I'm routing between two RB450G ports, from a 1 Gbps Ethernet tester (Sunset MTT) to a 1 Gbps Fluke responder.

The tester and responder have been tested for 0% paquet loss and 0% bit error at 1 Gbps line bandwith using direct connection between them.


After 100 Mbps through the RB450g (100 Mbps in each direction), i begin to have bursts of bit errors.


At 200 Mbps, bit errors, paquet loss (more than 1 %), and out of order paquets. processor load is 87 %.


I have no firewall rules, i have disabled ARP and all other non usefull stuff.


I will do more tests tomorrow with version 4, but i'm surprised by this result with version 5.0 RC4.

Do them in Switch mode to check its not a cabling/ethernet port problem?

You tested from Tester To Tester with one cable but you would need another one to go between the 2 RB450G.

We saw issues with link drops and errors with 1gbit cat5e cables, went away with cat6

There is no problem with cabling because i tested on the same ports with only switching.


I do not need two testers. The setup i use is :


Ethernet tester <--> RB450G port 3 // RB450G port4 <--> Loopback device.


Switching only (using slaving setup) show that the RB450G switch support full line rate at 1 Gbps between two ports without errors or packet drop.

At 1518 bytes packet size, this is about 987 Mbps in each direction.

After some new tests with RB450G and Router OS 4.13 :

I had the same packet loss problem than with version 5 RC4.

I had packet loss bursts (about 8000 packets), even at 50 Mbps about every 20 seconds.


So i disabled all non usefull packages -> same problem.


After adding static ARP rules, problems disapear. Perhaps there is an ARP stream management problem inside Router OS.


In the end, with RB450G and router OS 4.13, i can push 250 Mbps of trafic (total = 500 Mbps on each port) between two routed ports. The processor load was about 95 % at this rate for 1 Gbps total trafic through the router.


This is far less than the total rate the five 1Gbps ports can accept (10 Gbps total trafic) be is sufficient for small routing jobs.


In the End, the RB450G can be considered like a good (almost wire speed) 100 Mbps / port router, but not like a good 1 Gbps / port router.

To get wire speed on five 1 Gbps ports (10 Gbps total bandwith), we'll need about ten times the actual processor power, or 6,8 GHz instead of 680 MHz.
With Intel X86 and one 3.4 GHz core, we should be able to get theorically about 5 Gbps of total throughput.

There is no doubt that actual X86 power is not powerfull enough to manage a 10 Gbps router on more than one port at wire speed.

Very interesting test.

I think your test methods are seriously flawed, as experience speaks against your results

FIPTech, thanks for these tests.

do you see increased packet loss when you copy a file to the router's flash? or, when you backup config into a file on the router's flash?

note that these routers (450G/750G) do not have 5 ethernet ports, but merely 2 ethernet ports of which one goes through a switch. it depends between which ports you measured: port 1 i remember is direct, so to get maximum throughput (>500Mbps) the test should be between port 1 and a different port.

andy

PS: Normis, your comment is not appropriate. Many people who contribute to this forum actually spend their time to debug your product. As long as you don't test your products better, and as long as you don't document existing bugs and fixed bugs, you better shut up and contribute in a constructive way. If you claim FIPtech's test is "seriously flawed", then please would you say why you think so. THANK YOU.

No need to insult anyone. Read the forum rules.

Even RB1000 can easily throughput 3.2Gbps. We have seen much faster speeds on PC hardware. Saying that it's impossible to get more than 5Gbit through any existing hardware on the planet is just crazy.

You wanna show us a router running ROS *routing* 5gbit/sec of TCP traffic? Seriously show us, Cos i've never seen anything outside of an ASIC actually route that match TCP traffic

This is the second time you've flat out said something is not possible despite evidence to the opposite, Now I dont Ethernet test purely because we've never had any problems (And at gbit speed we use cisco) so I cant counter his testing but there is the real possibility of an issue here and weather or not it's isolated to a couple of bung 450's it still should be checked out

Who said anything about BGP? He even disabled all packages. Compare apples to apples please.

Where did I say bgp?

Outside of switch mode you either have bridge or routing and he said this issue is occurring on 2 routed ports

sorry, misread that. when you want to test maximum throughput, you use UDP. If the interface is, say, 100Mbit and you want 100Mbit TCP user traffic, you need to go back to TCP basics classes.

Normis, as beccara's boss recently expressed, it is really your product and your choice what you do with it.

Here with FIPTech you have someone spending his time to test your product using professional gear.
Why not appreciate it? Instead of flat rejecting his measurement, you could have hinted that the ports are not all equal.

And maybe there is really something relating to ARP? or about MT software that updates / grooms the MAC table in the switch chip every 30 seconds? Does this software clear all entries then write new ones? If yes then you get lost packets while the software does that. What FIPTech writes is not that impossible.

And could it be that MT does not have test equipment like FIPTech does...??

andy

Do you read my posts, or just complain?

I have nothing against testing, I only object to wild statements like 5Gbit being the maximum theoretical limit of any existing hardware in the universe (basically that's what he said).

UDP is always higher and yes I know why but most internet traffic I see is TCP

Either way, RB450G with 2 ports routed cap's out at 250mbit, FAR below what it should (TCP on a gig link should cap out around 800ish mbit if my memory is with me tonight)

So his test showed a problem, is there a flaw in it? Are you stating here that the RB450G in its most basic setup can only route 250mbit of TCP traffic? Or is there a problem that needs to be looked into?

You are starting to release a fair few gbit products now and you user base is going to be expecting to be able to route atleast 500mbit thru it if not more. Perhaps gbit RB's need to only be on the boards that have the power to route atleast x Mbit of traffic, Otherwise whats the point of all these gbit ports??

Do you read my posts, or just complain?

I have nothing against testing, I only object to wild statements like 5Gbit being the maximum theoretical limit of any existing hardware in the universe (basically that's what he said).

And I haven't see this on x86 platforms, neither has anyone I asked about it either, Switching/Bridging? Yes. Routing? no

So please prove us wrong, Show us ROS routing 5gbit or close to it. There will be a limit to what ROS can route on x86 and I bet its lower than you think

OK, your traffic consists of small packet TCP, but there are other people and other setups. Our tests are here:

http://www.routerboard.com/pdf/routerbo ... _tests.pdf

Cool, Soo on a bare bones RB going at its max you can route 2.416gbit with con-tracking off. I notice the RB450 isnt in there yet.

Also no x86, Your running PowerPC cores on the highend RB's which is capping you out at 2.4gbit @ 1.33ghz on a product you dont make anymore, RB800/1100 cap out at 1.821gbit. x86 is a little more costly to do this stuff on so you wont get much more out of it.

Once again if you wanna route 5gbit of traffic your gonna need a ASIC, So whats the maximum you've seen a system running ROS route? It's not small packet's causing it its the computing power needed.

To quote you - "Not possible"

notice the RB450 isnt in there yet.

It is. See RB400G. All series boards have same processor, so we don't separate individual models.

Will post our X86 tests in a moment.

i read your posts normis, and the complaint i made i think was fairly clear:
effectively i ask you to be polite with someone who spends his time to test your gear.
also I don't think FIPTech has claimed 5Gbps to be a limit. he has merely tested a small router, and has run into an issue.

so back to the technical subject:
has MT tested RB450G/RB750G with external test equipment like FIPTech does?

I already posted a link to the PDF above: http://www.routerboard.com/pdf/routerbo ... _tests.pdf
We use a professional test device: http://www.ixiacom.com/products/ixn2x/index.php

notice the RB450 isnt in there yet.

It is. See RB400G. All series boards have same processor, so we don't separate individual models.

Will post our X86 tests in a moment.

My bad

Have just been corrected a little tho, A old workmate of mine has pushed 10gbit thru a pair of 10gbit nics routed. CPU's sat at 80-85% on a pair of Xeon X5570's (4 cored 3ghz each) tho

I just try to make a real world test. If you want i do some tests in a precise setup, please let me know and i will do it, but it should be a real world test, with routing between two different networks.

Concerning the hardware testers, i'm using professional testers :

- Sunrise Telecom Sunset MTT with SSMTT29-L 1 Gbps Ethernet test module, advanced IP throughput test.

This tester is sending trafic to RB450G port 3, and receive through the same port trafic sent back by the reflector device connected on RB450G port 4 with a different network address.

- Fluke Linkrunner pro 1 Gbps in reflector mode (MAC + IP swaping) connected to RB450G port 4.


This test setup is able to push 1 Gbps at wire speed and give all details about packet loss, out of order packets, missing frames number and rate, runt signals, bit error, sync loss, check pattern loss and so on...


RB450G is sitting between the Ethernet tester and the reflector. He is connected through ports 3 and 4.

Tests are done with 1518 bytes frames.

The cables, testers and connections have been checked, using the same hardware setup through the RB450G, but using switching instead of routing, setting the RB450G ports 3 and 4 in switch mode (using slaving).

In switching mode, i get 2 Gbps total throughput per port, or 4 Gbps total for 2 ports, without loss and without out of order packets. This is the maximum possible speed, and show that the RB450G hardware is perfect.

With routing, the processor load at 1 Gbps total load (two ports at 2 X 250 Mbps) is about 95 %, so i think that processor power is the limiting factor. Regardless wich ports i would have used on the RB450G.

I'm testing the RB450G, not the RB1000. Normis i don't understand your comment about the RB1000 beeing able to route 3.2 Gbps. This is certainly true.

And concerning X86 hardware, it's certainly possible to get more than 5 Gbps, but with more than one core. This is supported on Router OS only since a couple monthes and there is not yet an hardware compatibility list, neither serious tests from users available on Internet showing the results.

I'd like to test a RB1000 but it's difficult to get them actually.

azg, yes i can see packet loss when i try to copy a file to the RB450G flash at the same time than pushing trafic.

I tried with 100 Mbps of trafic routed and reflected to the source, giving 400 Mbps of total trafic through two ports on the RB450G.

The CPU load was 57 % before copying the file and 0 % packet loss.
During file copy, processor load was 100 % and packet loss was 0.4 %.

If i try to make a backup configuration file from winbox, then i get a burst of 382 lost frames.


I can see packet loss bursts as well if i add a firewall rule. 4 packets lost when adding a forward permit rule at 400 Mbps total trafic load and 55 % processor load, and about 200 packets lost with 1 Gbps total trafic load and 95 % processor load.

To FIPTech: Sorry, but did you just landed from the moon and are on the way to rediscover America???

It is common knowledge that switching always work on wire speed, but bridging and routing involves CPU and memory.

For device that cost less than 100$ - even 400Mbps router throughput more than enough. Don't you agree?
400Mbps are good result, as MT itself was getting only ~750Mbps:
http://www.routerboard.com/pdf/routerbo ... _tests.pdf

writing file to the disk takes CPU time, so naturally it has impact on throughput performance

It is impossible to use state-full traffic (TCP) to check devices maximum throughput, as one point in time devices that are sending traffic will "assume" that no more speed increase is necessary, but device still can push more. UDP from other side will just spam device until it sinks in those packets.

Anyways, with one simple queue or change in queue size i can make same TCP test give 3 completely different results, while UDP will stay more or less same.

FIPTech, what about packet loss with 10Mbit flowing through the router & being reflected, using the 1518 Byte frames (similar to RTP video), or using 200 Byte frames (approximately VOiP traffic with G.711 20ms RTP)?

macgaiver: i agree completely that for less than $100 these are excellent routers. FIPTech is just measuring... right now the tests show that at 100Mbps full duplex through the router, packets are being lost. This is fine for web browsing, email and FTP, but it is not acceptable for voice and video.

at 100Mbps and 1518 byte frames one gets a frame rate of about 8200 per second. 382 frames are dropped when saving configuration in winbox, hinting that the router did not process packets for 46ms. at 10Mbps with 200Byte voice frames the 46ms translate to 287 frames, i would here too expect to see packet loss. at 10Mbps with 1518 byte frames the 46ms translate to 37 frames -- here probably the packet loss goes away because the ethernet RX ring can hold them, but the jitter increases. (FIPTech, does your tool show jitter?)

andy

Any proper Voice/video/online gaming have a mechanics to compensate for packet loss. I think i have read one topic where voice communication was still possible with 50+ % packet loss, only quality was lower, but still acceptable.

Anyway - have you tried to work with different interface queue sizes? and exactly what protocol are we talking about?

RB450G or RB750 / 750G are very good routers for their cost.

I'm using them for VoIP every days without problems in the field, but not with 100 Mbps load. The load is most of the time about 1 - 20 Mbps, with statefull filtering, queuing, natting, and all is working.

I think that those routers are usable at provider level even for VoIP, you just need to limit total throuput flowing through the router at a reasonnable value.

To avoid quality problems when using professionaly, i would recommand to buy Ethernet testers and fully test the setup at desired rate before going to production.

This is the same load problem with all software based routers, Mikrotik routerboard are good ones and are better than most other small products. I didn't do those tests to disqualify them.

During file copy, processor load was 100 % and packet loss was 0.4 %.

they are routers, not file servers. it's a known thing that CPU usage is high, when you write to the RouterBOARD NAND. that's why, RouterOS itself never writes to NAND, except when configuration is changed. otherwise it works from RAM.

Is it the same on RB1000 ?

For futur products would it be possible to have less hungry file transfers or configuration writings for writes to NAND memory ?

what and why do you need to transfer to the routerboard?

Sometimes i'm pushing big files : boot images for PXE remote boot through TFTP.

This is mainly for remote service.

Anyway, it is not a good thing that writing configuration introduce packet lost.


Wouldn't it be possible to redesign the NAND driver, so that it can write asynchronously ?

For new products, it would be nice if possible to add a small FPGA chip, where you could put a dedicated NAND controler, so that CPU is not overloaded by file transfer or configuration writing.

This FPGA could be used as well to relieve the main processor from heavy repetitive tasks like filtering and routing.

The advantage of FPGAs over Network processors is that you have total control about what you put in it, so you can update the code like main processor code.

Compared to ASICs, they are very low priced and there is no heavy cost to support like for ASIC design and manufacturing. Smaller ones begin at about 5$ by chip by quantity of 1000.

They can do almost all digital functions an ASIC can do. One drawback is that it is not possible to integrate analog circuitry in it like with an ASIC, but who needs analog circuits today for a router ? Perhaps big names to justify their high price they like to keep ASICs instead of FPGAs.

Professional VoIP (G711) needs no more than 1 % of packet lost to get a good audio quality. And 1 % is a bad value. We always try to keep packet lost at something better than 0.5 %.

We try to give our clients 0 % packet loss. This is not always easy with xDSL links, but is possible most of the time.


Do not forget than today VoIP and Video over IP replace traditional technologies like ISDN and PAL / SECAM for TV.


So we need to give at least the same quality than client did have before with those older technologies.


50 % packet loss is just impossible. Even 5 or 10 % packet loss is not tolerable at all.

Do you use limitations in your network at all? - Limitation happens by packet dropping btw - so results in same "packet loss".

Even without queues, each interface have queue on it with limited size. have you tried to increase it?

We have no problems with Routerboard hardware when using it at client rate (20 Mbps max). When we have packet drop, it is most of the time a problem on a xDSL link who need to be fixed. Most of the time noise margin because of crosstalk, bad cable, or impedance matching problems on the line between the client modem and DSLAM.

Sometimes i'm pushing big files : boot images for PXE remote boot through TFTP.

This is mainly for remote service.

Anyway, it is not a good thing that writing configuration introduce packet lost.


Wouldn't it be possible to redesign the NAND driver, so that it can write asynchronously ?

For new products, it would be nice if possible to add a small FPGA chip, where you could put a dedicated NAND controler, so that CPU is not overloaded by file transfer or configuration writing.

This FPGA could be used as well to relieve the main processor from heavy repetitive tasks like filtering and routing.

The advantage of FPGAs over Network processors is that you have total control about what you put in it, so you can update the code like main processor code.

Compared to ASICs, they are very low priced and there is no heavy cost to support like for ASIC design and manufacturing. Smaller ones begin at about 5$ by chip by quantity of 1000.

They can do almost all digital functions an ASIC can do. One drawback is that it is not possible to integrate analog circuitry in it like with an ASIC, but who needs analog circuits today for a router ? Perhaps big names to justify their high price they like to keep ASICs instead of FPGAs.

Why use FPGA for NAND, use FPGA as a stepping stone to ASIC's for routing functions and encryption off-loading. FPGA's are cheap but it's a stepping stone. ASIC's are better suited because they are designed at the most basic level for a task.

Perhaps it is time for MT to look at FPGA or ASIC's for intensive tasks on their highend products like dynamic routing/PPPoE/encryption.

Why use FPGA for NAND, use FPGA as a stepping stone to ASIC's for routing functions and encryption off-loading. FPGA's are cheap but it's a stepping stone. ASIC's are better suited because they are designed at the most basic level for a task.

Perhaps it is time for MT to look at FPGA or ASIC's for intensive tasks on their highend products like dynamic routing/PPPoE/encryption.

That's true but anyway, FPGAs are enough for small devices. The price of ASIC design and manufacturing is out of range for a small company.

ASICs can be more powerfull that's true, because they can be fully optimized and designed at hardware level, but they are very dangerous to market, if there is a bug in it, you need to reenter the design and production process, this can be a terrible financial disaster.

FPGAs are easier and less dangerous to manage, because they are tested by many clients. Only the code you put inside does change. If you have a problem, you can update the code by releasing a new firmware and let the client update the chip himself, like for a normal RISK, CISK or DSP processor.

If you have a problem with an ASIC, this can be a design problem at the hardware level. If this is the case, then you have a big problem..

You can even embed a processor inside an FPGA as software code, some FPGAs even have a hardware processor in them to avoid consumming gates for it, most of the time an ARM32.

Today a lot of products are using FPGAs, not ASICs even if they are marketed with the word "ASIC" because everyone know what is an "ASIC".
To what i've seen inside Ethernet testers, they are using FPGAs, not ASICs.

Some FPGAs do have one or more serial transmitters (serializers) in them, at 1 or 10 Gbps or perhaps more today. Those transmitters are able to synchronize themselfs to the input signal with a PLL, so communications at high speed are quite easy with those circuits. They do allow to transform 32 or 64 bits parallel data (at clock frequency of the chip) to the high speed serial bus.

So with a 10 Gbps serial stream, and 32 bits data, you need only a clock at about 300 MHz to manage the 10 Gbps stream in true real time. This is all you need to make a wire speed network filter / router with multiple 10 Gbps ports...

Last, FPGAs do allow to protect your code, so there is no problem if you need closed source.

regarding 10G on mikrotik:

i manage a video transmission system for a customer - three streams with
10-15Mbps each, from europe to the u.s. east coast. two of the three
systems have error correction, one of them gives detailed output.
i operate a total of four routers: a MT RB750G (4.13) and an old cisco
2800 (from 2004) in europe, and the same cisco/MT pair in the u.s.
the routers run ospf so that it is easy to steer the traffic over
specific links or over specific routers.

measurements this week confirm the packet loss measured by FIPTech: when
i route my 35Mbits of traffic through the two MT 750G, then the video
system's error counter shows lost packets every few seconds, and this is
with logging to memory, no config changes, static ARP, no filtering NAT
or the like, and very minimal router CPU load. the packet loss is around
150 packets per million sent. at 15Mbps (one video stream), that is a
lost packet every 5 seconds, and yes you can see dropouts in the picture
every few minutes on the one system that does not have error correction.

when i route the same traffic over the two old cisco 2800, then over one
million packets there was not a single packet lost. zero. note that this
is measured end-to-end from the LAN in europe to the LAN on the U.S.
east coast. mcgaiver, on today's networks there is NO packet loss, there
is no congestion. you pay for 100M ports, and you get 100M end to end.

before the bashing starts, please realize that these are measurements,
not guesses. it's a fact that RB750G, at 35Mbps one-way throughput,
occasionally looses a packet, while a 6-year old cisco does not.

what i'm saying however is, and here you can bash, that every tool
should be used appropriately. the MT RB750G is an incredibly good deal,
it is also easier to manage and has far more features than the cisco
2800 used above, which was probably about 60 times more expensive than
the MT. however, if you spend several thousand dollars per month in
transit and port fees, the cost of the cisco's becomes less relevant.
which brings me to my point: for me it is useless to discuss whether 10G
can be routed on a MT-based system (x86 multicore) or not. as long as
a MT750G gigabit router drops packets at 35Mbits i'm not even thinking
about routing 10G, which is a factor 300 more. i use MT for all those
places where I have up to 200M traffic & occasionally loosing a packet
is not a problem. this includes all web content and 80% of the routers
installed.

andy

azg, it's clear that it is not possible to use serial processing routers to manage high loads (between 400 Mbps and 40 Gbps) without packet loss and without high values of jitter.

Today, for carrier class 10G, 40G and 100G trafic, i think there is no alternative : use high end hardware routers. Perhaps this will change in the near futur.

Today, carrier trafic cannot tolerate packet drops. That's a fact. VoIP and Video cannot tolerate this for professional results. Clients SLA (service level agreements) are more and more tight. Packet drop, jitter, latency need to stay at very low levels.

The problem with Internet is that a packet can go through 10 or more routers to go to the final destination. So each router add an amount of latency and jitter, even if there is no packet loss on it. Without the help of parallel and wire speed processing, it is not possible to garantee a professional quality for packet delivery. Packet loss, latency and jitter needs to stay at vely low levels on each router.

So, as long as low cost routers, Mikrotik or others will stay based on serial processing without hardware ASICs, FPGAs or DSPs for routing and filtering, it will not be possible to use them for carrier trafic, except perhaps when load is very light or when trafic quality is not a concern.

Anyway, Routerboard and router OS are very good products, when they are used for what they can do. Using them at client sites is a good place for them, because most of the time the trafic load is < 20 Mbps.

It's certainly possible to use Router OS for light carrier trafic, but testing needs to be carefully done to be sure to stay inside the SLA limits in every situations (for example when there are dynamic routing updates).

Mikrotik - we have handled 600mbps of traffic on dual quad core xeons ! we have also been able to reach 3-400mbps using single quads and upwards of 100mbps on p4 based systems. I think we can easily touch upwards of 2GBPs on our equipment the way it is using GIGes..

It wont be able to touch 10g or beyond yet - we still need hardware that supports that through the bus itself without jitter so its more of a limitation on the hardware than MT.

The only isssues is the ROS stability and compatibility. You have to go with older generation hardware for compatibility or wait months for a driver to included - then find a bug and have to go back.

Despite this I am pretty happy with MT on the whole.

The limitation is not only the Ethernet devices bus. The main limitation is the processor speed and the way they are managing packets.

Serial processors (Risk or Cisk) are not able to process Ethernet packets in real time, or at least not always. So as soon as they are doing another heavy task like changing configuration or updating dynamic routing tables, they can't always come back to the packet management job fast enough. This is the reason why we see packet drops.

Even if such a high end processor is able to push 5 Gbps of trafic with a single core, there is no garantee there will be no packet drop.

To get 0 % packet drop, low latency and low jitter values, there is no other solution than hardware packet processing through special parallel processing logic circuits or signal processor circuits. Parralel processing logic circuits are FPGA or ASIC, signal processors are DSP.

So an important part of the Network Linux code need to be ported to those circuits. The console code and general management code can stay inside a classical processor for simplicity.

The main problem with parallel and signal processing are :

- programmers needs to master another programming language for logic circuits, this is very different from C++

- Linux code need to be splitted between a classical processor part for non realtime task and a logic circuits part for packet processing

- each Linux modification needs to be ported to the logic circuits part


So the cost does come from software, the hardware for logic processing is not expensive.

Today, for carrier class 10G, 40G and 100G trafic, i think there is no alternative : use high end hardware routers. Perhaps this will change in the near futur.

You've hit the nail on the head and I hope it DOESN'T change, Why dont earth would you use someone like MT to handle 10+GB of traffic? When your at those kinds of levels you need the support behind it, Cisco and Juniper are costly because if my Cisco 7609 handling 200-700gbit of traffic right now has a software problem I can call Cisco and get help, If its a bug they will have a fix out in days/weeks and if its hardware I can have a full replacement here in 4 hours.

If you're handling these levels of data you'll be wanting the ability to call the vendor up and yell until its fixed. "Nobody ever got fired for buying IBM"

Come on folks, you are asking too much from MT.
These are SOHO products and are not produced nor designed to push X number of gbit/s.

For this, there are product which sost $$$$$ and it's not for nothing.

Cheers...

That's true, but between a Cisco 7600 router able to push 750 Gbps of trafic at wire speed and 20 Mbps Soho routers, there is room for wire speed 1 - 50 Gbps low cost routers.

Today, it's possible to get about 5-10 Gbps on a low cost platform, but not at wire speed, because there is no opensource project to developpe and support code for logic circuits.

Linux need to add parallel processing functions for network functions. Because actual processors are not designed to manage a high count of small data amounts like IP packets (around 1500 bytes).

Linux developpers are too much focused on serial processors. They should watch what they can do with logic circuits (FPGA).

You've hit the nail on the head and I hope it DOESN'T change, Why dont earth would you use someone like MT to handle 10+GB of traffic?

As someone who remembers paying many thousands of dollars to get a router that could handle 45mbits properly, including a card with special chips that could handle such high speeds in hardware (This card alone was somewhere around two or three thousand USD), I find this statement to not be very future-proof.

"Nobody ever got fired for buying IBM"

I don't know who this Nobody is, but apparently they've never worked in a small business where money is very tight. (also applies to cisco and other products nobody ever supposedly got fired for buying). It depends on what management's priorities are.

The limitation is not only the Ethernet devices bus. The main limitation is the processor speed and the way they are managing packets.

One reason jumbo frames/support becomes important at 1 and 10g.

Jumbo frames are not used widely, most xDSL links do not allow for it, and most OS or applications does not use them neither.

So what you see on backbones is not jumbo frames, but normal frames.


Jumbo frames are good for processors, but they are not good for QOS.

Bigger packets means that latency and jitter rise.


As the world is now using IP networks for voice and video, it would not be a good idea to use jumbo frames.

It would be usable with high speed links, but at client side, where upload speed is most of the time < 1 Mbps, it would limit effectiveness of QOS.

This is certainly the main reason why ATM transport (53 bytes frames) is still used on ADSL links. It does allow triple play without QOS problems.

Try to do triple play on a 256 kbps Ethernet link, without ATM, you will see that it's not possible to have a good jitter value, because 1500 bytes frames are very long long to transmitt at this speed.

Using the best IP packet sheduler of the world does not help here. Because when a non prioirity packet is transmitted, you need to wait the end of his transmission to be able to transmitt a high priority packet. This delay can be too long for VoIP or video use, if receive buffers are not large enough. Using larger buffer is not always possible or desirable, because of cost, hardware implementation, or simply because it does add too much latency.

Cisco solved this problem using auto fragmentation. When big and non priority packets enter the queue buffer, they are fragmented to smaller chunks. So that priority packets can get sheduled sooner thanks to a finer graining.

Unfortunately, i'm not aware of other manufacturers implementing auto fragmentation QOS, Linux is not able do do this, neither Router OS.

Some information about traffic levels and cpu usage,

Dell PE1950, 2x Xeon 5130, 2x Dual Port Gigabit ethernet

> system resource print
uptime: 11w9h2m32s
version: "4.12"
free-memory: 1834272kB
total-memory: 1945752kB
cpu: "Intel(R)"
cpu-count: 4
cpu-frequency: 1995MHz
cpu-load: 17
free-hdd-space: 3750616kB
total-hdd-space: 3851684kB
write-sect-since-reboot: 34755872
write-sect-total: 34755872
architecture-name: "x86"
board-name: "x86"
platform: "MikroTik"

> /system resource irq print
IRQ OWNER
1 [0 0 IO-APIC-edge i8042]
3 [24 42 IO-APIC-edge ]
4 [3 1 IO-APIC-edge serial]
8 [0 0 IO-APIC-edge rtc]
9 [0 0 IO-APIC-fasteoi acpi]
12 [2 0 IO-APIC-edge i8042]
14 [0 0 IO-APIC-edge ide0]
16 [3965021910 3948232069 IO-APIC-fasteoi eth0]
16 ether1
16 ether2
16 ether3
17 [3159676871 3141868905 IO-APIC-fasteoi eth5]
17 ether4
20 [0 0 IO-APIC-fasteoi uhci_hcd:usb3]
21 [418705 16 IO-APIC-fasteoi ehci_hcd:usb1]
21 [uhci_hcd:usb2]
21 [uhci_hcd:usb4]
23 [0 0 IO-APIC-fasteoi ata_piix]


How to put all ethernet cards to different IRQ?

set different IRQ for ethernet you can only in version 5.xx
What ethernet card you use ?? and what type of bonding ?

Ethernet: HP NC360T

> /interface bonding print detail
Flags: X - disabled, R - running
0 R name="bonding1" mtu=1500 mac-address=00:1B:78:56:57:12 arp=enabled slaves=ether1,ether2,ether3,ether4 mode=802.3ad primary=none
link-monitoring=mii-type1 arp-interval=100ms mii-interval=100ms down-delay=0s up-delay=0s lacp-rate=30secs
transmit-hash-policy=layer-2-and-3

one more question - how many filter rules in chain forward and mangle ? do you use nat and enabled coonn-track ?

/ip firewall connection tracking print
enabled: no

/ip firewall filter print count-only
0

Nat disabled.

Some more information.

/ip route print count-only
4754
/ip address print count-only
81
/interface print count-only
54
/routing bgp peer print count-only
20
/routing filter print count-only
102

/interface monitor-traffic bonding1 once
rx-packets-per-second: 204374
rx-drops-per-second: 0
rx-errors-per-second: 0
rx-bits-per-second: 1286.2Mbps
tx-packets-per-second: 204359
tx-drops-per-second: 0
tx-errors-per-second: 0
tx-bits-per-second: 1285.6Mbps

Rackspace has 10Gbps routers running on x86 hardware as well as virtual routers in their cloud infrastructure with an open source solution. When you're dealing with high bandwidth links you have to really examine where and what QoS to apply. It's normally in the realm of a core network and traffic shaping should be done before it enters it. The solution of throwing more hardware at it rather than good engineering is a poor solution.

Today (and since about 12 years) processors have only about 3 Ghz frequency, this is the limitation.

Clock doubling will theoretically improve the overall performance of the machine substantially, provided the fetching of data from memory does not prove a bottleneck. In more modern processors where the multiplier greatly exceeds the rest of the architecture, the bandwidth and latency of specific memory interface circuitry and/or the bus or memory controller typically become a limiting factor.

Ask Intel to release new 100 GHz procesors, and wired speed routing at 10 Gbps will be possible with Router OS. But i'm afraid they prefer to sell DSP or FPGA chips for this market segment, instead of killing it releasing a 100 GHz x86 processor.

Since the late 1990s almost all high-performance processors run at higher speeds than their external buses, so the term "clock doubling" has lost much of its impact. Clocking a processor at 100 GHz will probably only cause damage to a CPU or other components due to overheating or even voltage break-down.

So does Mikrotik support any 10G interfaces? In a year or two we may need them. The upstream we are looking at in the datacenter only provides 100Methernet, 1Gfiber and 10Gfiber. We are thinking of a couple 1G loops to pull it out of the datacenter. So when them loops to our seperate headends start to pull more then 500 mbps each we will need more then 1G coming into the Mikrotik BGP box.

Perhaps we will never grow that big but back in 2000 who thought of ever getting a GigE Internet connection?

Clocking a processor at 100 GHz will probably only cause damage to a CPU or other components due to overheating or even voltage break-down.

That's not true. Are you working for Intel marketing staff ? If you are using the adequat technology there is no dammage at all.

Those technologies do exist since years, but big chips manaufacturers do prefer to keep actual technologies for cost reasons and because there is not much real competition in the field of high speed circuits market for mass.

If Intel did stop multi core architectures and did concentrate on speed rising, today PCs would be really faster than they are.

Multi core X86 is a technology of the past. I remember using dual processor boards with Pentium 2 PCs. This was even available before with 486 or Pentium / Pentium pro chips. It was costly and only specially compiled programs were taking benefits of this architecture.

Rising speed and using dedicated programmable logic helper circuits is the futur.

We begin to see this tendance with the use of GPU boards for non graphics computing and even for high speed routing.

Programmable logic circuits are not a good thing for X86 processor manufacturers or other processor manufacturers, because it is really a more powerfull and optimized way of managing special computing tasks like routing. Logic circuits could be considered like a market killer for X86 architecture because with it you can do the same taks but at really slower clock frequency.

intelligent design + slower clocks = lower cost

X86 / x64 = fat and costly design for most specialized tasks.

Clocking a processor at 100 GHz will probably only cause damage to a CPU or other components due to overheating or even voltage break-down.

That's not true. Are you working for Intel marketing staff ? If you are using the adequat technology there is no dammage at all.

I'm not working at any CPU maker, but I'm interested of what technology that can overcome physical laws.

Multi core X86 is a technology of the past.

If that is true, can you tell us why multi-core network processing devices have become mainstream with companies such as Freescale Semiconductor, Cavium Networks, Wintegra and Broadcom?

I am thinking that 32 bit CPU's process 32 bits at a time and 64 bit CPU's can process 64 bits at a time. So they can work with much more then 1 bit per clock cycle, I would guess.

Dealing with a 10G interface might be a struggle but I bet its certainly possible. Any driver support for 10G to try yet though? Doubt the interface cards are cheap either.

Still, why not prepare for the future ? Any recommended cards we could add support for ?

There are a number of 10G cards listed here:

http://tinyurl.com/4le8x3u

$1500 range appears to be minimum.

Clocking a processor at 100 GHz will probably only cause damage to a CPU or other components due to overheating or even voltage break-down.

That's not true. Are you working for Intel marketing staff ? If you are using the adequat technology there is no dammage at all.

I'm not working at any CPU maker, but I'm interested of what technology that can overcome physical laws.

Multi core X86 is a technology of the past.

If that is true, can you tell us why multi-core network processing devices have become mainstream with companies such as Freescale Semiconductor, Cavium Networks, Wintegra and Broadcom?

IBM have 100ghz transistors operational, There is little reason we cant ramp up clock speed these days. You just need good thermal management which is also where IBM has demo'ed micro water jet cooling on processor.

X86 is not suitable for core high end routers, It's just too fat and slow, ASIC's are what the big boys use when they need something simple done VERY fast.

Dont get me wrong tho, ASIC/PLC/Whatever you want to call them are no real threat to the X86 market

$1500 range appears to be minimum.

I found new 10 Gigabit dual port server adapters from $220 and up.

http://www.serversupply.com/NETWORKING/ ... 20GIGABIT/

Any progress?