Hi Guys,

I'm hoping you can help me with my limited knowledge.

The set up we are testing is a 3-line Fibre to the cabinet service (in the UK) using MLPPP.

The sync speed of all 3 lines is 80Mbit down and 20Mbit up (we are close to the fibre cabinet), therefore providing a maximum theoretical speed of 240/60.

Of course, there will be lots of overheads, so say around 200/50 to be safe.

The MLPPP is working fine, but the speed maxes out around 150/42. That's still great, but I'm wondering why its not going faster?

Looking at the status of the 3 ports to which the modems are connected, I can see that traffic is being split equally on all 3 lines, as follows:
As you can see the traffic is being split, but its not getting close to the 75Mbit throughput each line can take individually (tested each line on its own and it maxes at 75 down / 18 up so the lines and modems are ok.

I've tried various MTU settings etc and the local network is gigabit.

What do you think could be the culprit?

Thanks in advance,

James

I've not had the pleasure of having mlppp but i'd check a few things.

1. have you tested throughput on just one of the lines ?

2. what is your cpu usage like on the router?

3. Are you sure that server is has the capability to put out that throughput ? maybe try a few concurrent services from varying locations

Thanks for the reply rjickity.

1) Yes, each line can attain 75 down and 18 up throughput on their own.
2) CPU when maxing out the download is actually around 90% so that's pretty high I guess, could be contributing to it, however when maxing out the upload its only seeing 40% and obtaining 42Mb out of a max of around 54Mb possible.
3) Yeah ,tested many servers, multi-threaded, http/ftp/Usenet etc.

Thanks

James

Could you send a printout from:
/int bridge exp comp
/int pppoe-cli exp comp
/int eth exp comp

Turn off connection tracking if not needed.

Sure,

/int bridge exp comp
# jul/11/2012 10:08:46 by RouterOS 5.18
# software id = TC3D-7G87


/interface pppoe-client
add add-default-route=yes allow=chap disabled=no interface="Line 1,Line 2,Line 3" max-mru=1500 max-mtu=1500 mrru=1524 name=MLPPP password=*** use-peer-dns=yes user=***@***.adsl24.co.uk


/int eth exp comp
# jul/11/2012 10:10:05 by RouterOS 5.18
# software id = TC3D-7G87
#
/interface ethernet
set 0 name=Gateway
set 1 mtu=1520 name=LAN
set 2 mtu=1520 name="Line 1"
set 3 mtu=1520 name="Line 2"
set 4 mtu=1520 name="Line 3"

Need NAT so can't turn connection tracking off (unless I just use the RB for the MLPPP and then bridge it to another router to handle the LAN - would that work and what setting is required?).

P.S. Its a RB450G with a 680Mhz CPU and 256MB RAM so I would have thought it could cope better!

Thanks,

James

Could you check the CPU load when running the MLPPP test ?

/interface ethernet
set 0 name=Gateway
set 1 mtu=1520 name=LAN
set 2 mtu=1520 name="Line 1"
set 3 mtu=1520 name="Line 2"
set 4 mtu=1520 name="Line 3"

Need NAT so can't turn connection tracking off (unless I just use the RB for the MLPPP and then bridge it to another router to handle the LAN - would that work and what setting is required?).

P.S. Its a RB450G with a 680Mhz CPU and 256MB RAM so I would have thought it could cope better!

Thanks,

James

Can you make a full export of /int eth exp?
Have you tried other MRRU settings, or consulted the ISP? BTW, what's on the other side?
What is the CPU load while the speed test runs?
If the CPU load is high, it would be interesting to turn off connection tracking and disable all filter rules.
Is it possible for you to have a "testbox" on the ISP side, just to ensure that there are no bottlenecks on their side?

/Paul

Could you check the CPU load when running the MLPPP test ?

Hi, there is a screenshot in my first post that shows the CPU when running a bandwidth test (http://forum.mikrotik.com/viewtopic.php ... 77#p324769)

Can you make a full export of /int eth exp?
Have you tried other MRRU settings, or consulted the ISP? BTW, what's on the other side?
What is the CPU load while the speed test runs?
If the CPU load is high, it would be interesting to turn off connection tracking and disable all filter rules.
Is it possible for you to have a "testbox" on the ISP side, just to ensure that there are no bottlenecks on their side?

/Paul

/interface ethernet
set 0 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1520 mac-address=D4:CA:6D:34:E0:03 master-port=none mtu=1500 name=Gateway speed=100Mbps
set 1 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1520 mac-address=D4:CA:6D:34:E0:04 master-port=none mtu=1520 name=LAN speed=100Mbps
set 2 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1520 mac-address=D4:CA:6D:34:E0:05 master-port=none mtu=1520 name="Line 1" speed=100Mbps
set 3 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1520 mac-address=D4:CA:6D:34:E0:06 master-port=none mtu=1520 name="Line 2" speed=100Mbps
set 4 arp=enabled auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1520 mac-address=D4:CA:6D:34:E0:07 master-port=none mtu=1520 name="Line 3" speed=100Mbps
/interface ethernet switch
set 0 mirror-source=none mirror-target=none name=switch1 switch-all-ports=yes
/interface ethernet switch port
set 0 vlan-header=leave-as-is vlan-mode=fallback
set 1 vlan-header=leave-as-is vlan-mode=fallback
set 2 vlan-header=leave-as-is vlan-mode=fallback
set 3 vlan-header=leave-as-is vlan-mode=fallback
set 4 vlan-header=leave-as-is vlan-mode=fallback
set 5 vlan-header=leave-as-is vlan-mode=fallback

Yes, other MTU/MRU/etc have been tested with no difference.

CPU load is around 90% as you can see in the original post screenshot.

Can't turn off connection tracking without disabling NAT, so it will take the office PC's down - so I'll need to work out a different way of doing that, possibly by putting the RB in a bridge mode and then use a gigabit router for the NAT/LAN? What do you think?

No limits on the ISP side, I've been informed. They've tested it up to around 500 Mbit/s in their labs.

Thanks,

James

Just wanted to see if the switch was involved in the configuration.
I can't see anything wrong in the configuration so, the last thing to test is to configure the RB 450G as a bridge and add an other router for the NAT and firewalling.

/Paul

Just wanted to see if the switch was involved in the configuration.
I can't see anything wrong in the configuration so, the last thing to test is to configure the RB 450G as a bridge and add an other router for the NAT and firewalling.

/Paul

Any quick tutorial on how to do this?

Would turning off connection tracking really make a noticeable difference to the CPU utilisation as I was the only PC on the network at the time of doing the speedtest?

90% CPU load seems really high. (i don't see this value in your first post, where is it ?)

Try with a more powerfull router to get such a high bandwith.

Any quick tutorial on how to do this?

Would turning off connection tracking really make a noticeable difference to the CPU utilisation as I was the only PC on the network at the time of doing the speedtest?

http://wiki.mikrotik.com/wiki/Manual:BC ... _bridging)
This require your ISP to support BCP on the PPPoE server.

Is the lines fiber or VDSL?

Any quick tutorial on how to do this?

Would turning off connection tracking really make a noticeable difference to the CPU utilisation as I was the only PC on the network at the time of doing the speedtest?

http://wiki.mikrotik.com/wiki/Manual:BC ... _bridging)
This require your ISP to support BCP on the PPPoE server.

Is the lines fiber or VDSL?

VDSL2

Can the MT not just do the PPP and then pass all traffic to another router which then handles the NAT setup?

Cheers

90% CPU load seems really high. (i don't see this value in your first post, where is it ?)

Try with a more powerfull router to get such a high bandwith.

Sorry, here you go:
What router is more powerful? A 680MHz router with 256MB RAM is pretty hefty in my experience, most routers are 8 or 16MB RAM with a 300MHz CPU.

VDSL2

Can the MT not just do the PPP and then pass all traffic to another router which then handles the NAT setup?

Then you need to route the traffic through the MT to the NAT router and that you ISP route that NAT-IP through your MT's IP.

So, I bet that the CPU is busy with the MLPPP.
Do you have a RB1100AH for testing, If not try a PC with 4 nics and install ROS with the same configuration.
You can run ROS i386 version in demo mode for 24hours.

A 680 Mhz router with only one core is not powerfull enough to do this.

The RAM quantity is not the problem. The processor speed is.

Do not forget that on a software based router, all packets need to pass through the processor for routing, qos, firewall, and all other tasks like MLPPP framing. In the end this ask a lot of processor ressources.


Perhaps that the new generation RB2011 could do it, or a dual core RB1100AHx2.


If still not enough wait for the 36 cores (1.2 Ghz per core) CCR1036 :=) This is the way to go for near wire speed routing.


And if you need true wire speed without latency, there is no other way than using a FPGA / ASIC based router, but be prepared to spend much more money at the well known high cost manufacturers...

VDSL2

Can the MT not just do the PPP and then pass all traffic to another router which then handles the NAT setup?

Then you need to route the traffic through the MT to the NAT router and that you ISP route that NAT-IP through your MT's IP.

So, I bet that the CPU is busy with the MLPPP.
Do you have a RB1100AH for testing, If not try a PC with 4 nics and install ROS with the same configuration.
You can run ROS i386 version in demo mode for 24hours.

I actually have a spare PC with 4 NIC's in. So do I just download the ROS ISO for the PC / x86 and boot the CD, or is there some special installation requirements with a PC?

Cheers

no special requirements except that your hardware needs to be compatible, and that you'll need to buy a licence after the trial period, (one day if i remember well).

You'll need to install it on HD from the CD if i remember well, this needs a full HD format. you cannot partition your disk.


I don't like a lot the PC solution. A lot of power waste. Using an Intel X86 for routing is like driving with a ferrari in a small city.


More you will not have full support from Mikrotik, because the hardware part is not under their control.

I actually have a spare PC with 4 NIC's in. So do I just download the ROS ISO for the PC / x86 and boot the CD, or is there some special installation requirements with a PC?

Cheers

Read: http://wiki.mikrotik.com/wiki/Manual:CD_Install
You can install on a harddisk, USB flash, CF or whatever device your PC can boot from.
You can also prepare a USB stick from the Netinstall program, but for now it's easier to run off from a CD

/Paul

Thanks very much guys, I shall report back in a day or so and see how it goes.

Is there a list of recommended hardware, NIC cards/chipsets etc?

Thanks

Thanks very much guys, I shall report back in a day or so and see how it goes.

Is there a list of recommended hardware, NIC cards/chipsets etc?

Look in:
http://wiki.mikrotik.com/wiki/Supported_Hardware

It will probably work on any PC, except KVM support.
I recommend Intel NICs and Atheros for WiFi cards

James, out of interest, who is your ISP?

Hi James, out of interest, did you get to the bottom of your issues?

Regards, Dominic.

Hi,

I also have a similar issue. I work for a major ISP in Canada and I'm tasked with config for these for our MLPPP bonding. I keep it as simple as possible but have run into a very odd issue.

I have a client with 2 50 X 10 FTTN VDSL lines. both going through bridged modems. Direct connect to either modem 50Mb no problem. Toss them on the Mikrotik and it gets 25Mb. Almost a perfect 50% loss of speed to the bond.

The router is in full passthrough from a static WAN on the pppoe client to a /30 on port 5 for the LAN. I have NO NATTING what so ever on the device. It simply passes all traffic from the WAN to the LAN direct. It only bonds the lines and handles the pppoe. CPU usage is averaging around 14% on a 750GL running 5.25 firmware.

Identical config for 4 X 25Mb works like a dream. The 50Mb lines kill it. Any ideas? I've been through every thread I can find and tried stripping it down to nothing but I cant find anything that would do this. Even the line stats and profiles are perfect. They are on the same SLAM etc and noise levels are perfect, the guy is almost on top of the CO. The bottle neck is the Mikrotik and there isn't even huge amounts of packet loss to track. I don't get it.

Any ideas where I should look next guys?

James

So I am currently in a fight with one of my network guys over this.

He is saying that there is no way the RB750GL has enough juice to push 2 50Mb lines in MLPPP. I very much disagree based on the config. Its down to nothing. Literally no NATTING or firewalling of any kind, just a static WAN IP and a /30 tied to port 5. So to re-cap all I have on the device is:

Modem 1 eth 1
modem 2 eth 2
LAN gateway eth 5

pppoe - client bonding eth 1 & 2 interface using MLPPP

Modems are bridged bell sagemcom f@st 2864's (garbage firmware but its bridged so this should not matter.)

The exact same config will push 4 25Mbps lines bonded but not 2 50's.

Is this just too much for the device to handle? Do I need to send this guy a 493AH? I hope not. I hate it when the network guys are right.

Please help restore my faith in these awesome devices.

Thanks in advance,
James

Hi,

Still having this issue and not even a blip on the forums?

I suppose its time to start shopping for a new solution for all of these connections then.

James

MLPPP on adsl lines is very dodgy, even great adsl lines...

I tried it on aDSL max lines and on two 8 meg lines got only a little increase in speed. 10 megs or so.

MLPPP was designed i believe for isdn. The least lack of symmetry between the lines can really mess up the speed.

I took the same lines and used a (eBay) Cisco 1841 and set it up for ip per-packet load sharing and lo!! I got 14 megs. Three lines gives 21 or so megs. Look for this and if you mark packets appropriately you can also do this with mikrotik, although the config is a bit more involved than with Cisco.

But when dealing with 80/20 fibre lines you need to remember you need *serious* power in your routers. Cisco kit needs to be modern models, and MT kit will need lots of CPU power.

But the bottom line is MLPPP is not the best way to do WAN bonding

I See u Using an RB450G....

I had the same issue when I was using and RB2011, I upgraded to RB1100X2H and had 20-30% improvement on my bandwidth.

My issue is this........

4 lines at 25 X 7 works no problem. Then if you try 2 50 X 10's it dies and the RB750GL chokes to 50% speed. the processor never goes over 14% usage however. One would think that if it will push 4 X 25 = 100Mbps than 2 X 50 = 100Mbps should not be an issue. Upload is within 6Mbps between the 2 plans, identical modems and identical SLAM sending out the signal.

It is for some reason though. No packet loss, lines are all stable and the noise margins could not be better if I put my own values in there.

Is the RB750GL simply not beefy enough to push this?

James