Hi all,

we have big problems with performance of PPPoE server running 3.x versions. Today we run 3.4 version and still having problem. The problem is that we have packetlose on localhost (127.0.0.1) and all interfaces. It seems like it is a performance problem or problem of interface queues.

We run PPPoE server with quad core cpu xeon 2400, with board chipset intel 3000 server chipset with two gbit onboard ethernet cards. Traffic is up to 40/10mbit download/upload.

Enclosed is screen copy of our pppoe server. You can see packet lose to address 127.0.0.1 and high ping time response. Can someone observe the same problem?

We have this problem on all PPPoE server under heavy load.

I have this same problem here, I changed the machine and it remains to loose about 4% on all interfaces. Even with low CPU usage (20%) I have packet loss on all interfaces (even local).

I already tried to disable conntrack and TCPMSS to reduce processing but it didn´t solve the problem.

In fact I also saw this happening in 2.9.50 version, but not on early versions.

This is a huge problem that I cannot find a solution.

At least, I am not alone.

And what are you doing now? Downgrade back to 2.9.xx?

I have new observation.

When I switched off "Change TCP MSS" on all used profiles, so there were no dynamic iptable's Mangle rules, the problem diminish. It is still here, but not so huge.

I only have one router using version 3.4 (the other ones still use old 2.9 versions).

For me disabling Change TCPMSS doesn´t solve the problem, I made this test today and even with this option disabled I continue to have packetloss (about 2%). Probably with TCPMSS off the packetloss is lower but it still not acceptable.

And I´m not talking about too much connections. With more than 100 pppoe connections it´s already possible to see packetloss on a Pentium D 3.4Ghz (with multicore enabled or not).

We have machines with a lot worst hardware than that using old 2.9 version and more than 500 users with 0% of packeloss..

I think the best way is report it to Mikrotik support, and pray for a solution as soon as possible.

If everybody who has this same problem report it to Mikrotik support they will pay more attention to it.

I did, but still no answer. I wanted to check if there are others that meet the problem.

Do you use queues?
We have the same problem with packet loss and one of the problems is that ROS drops a packet every time a queue is created or deleted.
When many users connecting/disconnecting you get a packet loss.
Maybe that's not the only reason, but thats a reproducable and Mikrotik confirmed this issue.

Yes, I suppose that it can be problem. But in older versions it was correct. Maybe MT should consider this.
But not only packet lose, also high time response to locahost ping is problem. Maybe it is common problem.

I use queues and I agree that it could be related to problem. But i can´t stop to use queues!

If this problem was not solved soon this will make the pppoe server resource unusable for me and everybody I know that use it.

I believe MT is working on this issue.
But there is always workaround - use static queues or queue tree. There are also several benefits to use static queues:
*) few queues, instead of hundreds for each user
*) easy to manage
*) reduced CPU and memory usage.

Probably you can find more.

When I write about this nobody responded

Even some of you write that we (my company) do not know to use mikrotik and other bullshits...

So:)? What now:) Who has the right about packet loss in v3.X ???

The problem is really in MT. From my experience I can tell you that there is not packet loss when you use mikrotik v3.X versions for other purpose, but especially when you use it together with PPPoE, packet loss appear immediatelly.
As number of pppoe tunnels grow, packet loss also grows. As bandwith in this router grows, packet loss also grows.

Then CPU Load go high as 90-100% and than router is not accessible.

I believe MT is working on this issue.
But there is always workaround - use static queues or queue tree. There are also several benefits to use static queues:
Probably you can find more.

Static queues are not suitable for at least one situation: when you have a number of bandwidth plans and you set the speed with "Mikrotik-Rate-Limit" radius attribute.

Maybe you'll recommend, how to set up a speed limit with predefined static queue based on radius reply?
All users have dynamic IPs, so I can't just shape "that group of ips to 128k, other ips to 256k, etc".

I believe MT is working on this issue.
But there is always workaround - use static queues or queue tree. There are also several benefits to use static queues:
*) few queues, instead of hundreds for each user
*) easy to manage
*) reduced CPU and memory usage.

Probably you can find more.

We terminate up to 700-900 users on one PPPoE server, I suppose it is not possible to deal with static queues. It would be possible only if services will be the same for all users or there will be some other limitations.

It seems to be true - I watched log and ping and it seems that every time new PPPoE session is created or closed, packet lose occures.

So, it seems there are two problems:
- packet lose when creating/closing pppoe session
- high ping response to locahost when higher utilization even CPU is about 50%, so it can be problem of some queues like interface queues

Use static queues to solve this problem is impossible. Queues are created dynamically by radius attribute.

Use static queues to solve this problem is impossible. Queues are created dynamically by radius attribute.

It will not help to resolve problem. We already tried.

If using static queues doesnt solve the problem so it´s not related with the creation process of the queue.

I have reply from MT support, they confirmed problem and they are working on solution.

I have reply from MT support, they confirmed problem and they are working on solution.

Let´s cross our fingers and hope they find a solution as soon as possible.

it seems they found. I am testing 3.7 and it works much more better...

it seems they found. I am testing 3.7 and it works much more better...

Good news!

But where did you find this version?

MT sent me to check corrections.

MT sent me to check corrections.

Ok, I think they will release it soon..

How are your tests? It´s not loosing packets?

3.7 beta works fine, as we can see for this short period of time:
Packets: Sent = 4546, Received = 4546, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 6ms, Maximum = 478ms, Average = 8ms

Before this I had 2%-5% loss on 200 packets.

3.7 beta works fine, as we can see for this short period of time:
Packets: Sent = 4546, Received = 4546, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 6ms, Maximum = 478ms, Average = 8ms

Before this I had 2%-5% loss on 200 packets.

Excellent result!!! My packet loss is the same you had with previous versions.

I´m waiting for this version release to solve my problem.

3.7 beta works fine, as we can see for this short period of time:
Packets: Sent = 4546, Received = 4546, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 6ms, Maximum = 478ms, Average = 8ms

Before this I had 2%-5% loss on 200 packets.

This is great news!
Why do you have so big latency?

What queue type do you use? Is this over wireless or ?

It's a DSL shaped to 128kbit and latency is pretty good (except "maximum", but maybe it was a lag on a line).

Pity, but after a day of uptime it began loosing packets and a minimal ping is 60ms :-/

Yes, I can confirm this problem too. I have ping to localhost over 100ms. After restart it is ok. I've already informed MT, they are working on it.

And a new beta of 3.7 just arrived. Testing again...

Hope that "7" will be lucky number I have same problem, but i rolled back to 2.9.51. I can't afford this to happen in my net.

Cheers
Michal

Hope that "7" will be lucky number I have same problem, but i rolled back to 2.9.51. I can't afford this to happen in my net.

Cheers
Michal

I have this same problem since version 2.9.50, so rollback didn´t solve my problem. But with older versions (2.9.4x) works fine.

Don't wory, 3.7 seems to be really very good.
Running 24 hours, no problems yet.

I have this same problem since version 2.9.50, so rollback didn´t solve my problem. But with older versions (2.9.4x) works fine.

You see, this is what worries me for me packet losses stopped as soon as i rolled back to 2.9.51, OMG maybe i have different problem

Don't wory, 3.7 seems to be really very good.
Running 24 hours, no problems yet.

Is there any way i could get as well that beta to test it?

cheers
Michal

Don't wory, 3.7 seems to be really very good.
Running 24 hours, no problems yet.

Glad to hear

Don't wory, 3.7 seems to be really very good.
Running 24 hours, no problems yet.

any update ??

Still the same, working fine..., no problems as before.

I recently upgraded a 2.9.26 PPPoE Server box to 2.9.51. Nothing has changed except the version.
What I see are users falling off one by one slowly with message logged 'peer not responding'. They log back in fine next time they retry after 30 secs (SOHO router PPPoE clients).
Over a 24hr period, maybe 50 users per PPPoE server willl fall off for no apparent reason.

The ability of 2.9.51 to 'hold' PPPoE sessions is not solid.

Can anyone confirm the last known good version of 2.9.x ?

Also, accoring to this thread, v3.7 should fix this problem. Does this mean they will also release 2.9.52 to fix the 2.9 branch ?

We use PPPoE servers for thousands of PPPoE clients and we never had such problem. Check your connectino timeout settigs may be. There were another problems on 2.9.5x.

When will this version 3.7 be available for download ?

We are having this packet loss problem in two of our pppoe concentrators (versions 3.1 and 3.6) and need to solve it really soon... We are having a lot of users compliants...

Thanks in advance.

yeah we could really get use of that, even of that beta version you say is working correctly, so please if someone can please share.

cheers
Michal

We still have problems with 3.7 beta, it works good only a short period of time.

We do not have any problems with 3.7 build time Mar/27/2008.
We run it 4 days on 4 PPPoE server with 500-800 active connections each.
We have only little performance problems that are common for older versions 2.9.x

Then please share this beta somewhere or give us link to it or whatever

Hi EveryOne!

We also have this problem with our PPPOE server (600 users). Every day once we have to reboot the server (packetlose, performance problem)...

Where can we download the 3.7 beta??? Or when will be 3.7 ???

Thanks in advice!

Josh

We tested the 3.7 package, from our point of view it is suitable to be released. But now it depends only on MT.

thanks!

So Normis???

We do not have any problems with 3.7 build time Mar/27/2008.
We run it 4 days on 4 PPPoE server with 500-800 active connections each.
We have only little performance problems that are common for older versions 2.9.x

Just to know have kind of little performance problem you have?

1. when there are more users (700-900), we can see that sometimes ping to localhost is higher, up to 2-5ms, but not every ping, only 10-30%
2. we have problem that users are not able to get higher speeds than 15-20Mbit for download and 5-9Mbit for upload. But this is common problem with older 2.9.x versions and it is only one unpleasant feature of PPPoE compared to DHCP

1. when there are more users (700-900), we can see that sometimes ping to localhost is higher, up to 2-5ms, but not every ping, only 10-30%
2. we have problem that users are not able to get higher speeds than 15-20Mbit for download and 5-9Mbit for upload. But this is common problem with older 2.9.x versions and it is only one unpleasant feature of PPPoE compared to DHCP

Thanks for reply. At least for my case these are minor issues.

Hello

Tell me guys is it so hard to put this build somewhere? Can't you see that there some desperate guys here, that really need to try this out? Please put it somewhere and share this can't be that hard.

Cheers
Michal

At least one more problem was found with test build.

Problem will be fixed in next build and at this point it looks like it will be
the last - so, please, wait for the release of 3.7

At least one more problem was found with test build.

Problem will be fixed in next build and at this point it looks like it will be
the last - so, please, wait for the release of 3.7

Yeah, tell that to my users

I think i have a little more patience in me, hopefully my users too

Cheers
Michal

3.7 was released today but changelog has nothing about PPPoE fixes!!!!!!!!!!!!!!!!

pppoe fixes are included but i dont know why not described in changelog :>

pppoe fixes are included but i dont know why not described in changelog :>

Are pppoe problems really fixed?

Yes pppoe problem is fixed take a look in this topic.

http://forum.mikrotik.com/viewtopic.php?f=2&t=23013

Thanks

Today I upgraded one of our Access Concentrator to 3.7
I only load it with 10Mbps and 200 users, its work great, packet loss and latency issues vanished
I will try tomorrow with 40Mbps and 600-700 users
Good work Mikrotik team

Currently 50-58mbit/s, 580 users (325 queues) - packet loss i still here
To RouterOS itself:
600 packets transmitted, 594 packets received, 1% packet loss

The absolutely same story via connected PPPoE:
Packets: Sent = 600, Received = 594, Lost = 6 (1% loss),

It's better than 2-5% loss in older versions, but our freebsd/pppoed server with 500 users online did not lose even 1 packet

Currently 50-58mbit/s, 580 users (325 queues) - packet loss i still here
To RouterOS itself:
600 packets transmitted, 594 packets received, 1% packet loss

The absolutely same story via connected PPPoE:
Packets: Sent = 600, Received = 594, Lost = 6 (1% loss),

It's better than 2-5% loss in older versions, but our freebsd/pppoed server with 500 users online did not lose even 1 packet

Do you have experiences with freebsd/pppoe? Is it good solution? Why are you trying MT PPPoE?

that's not a topic for routeros forum
you can write me a mail to

Here are my results with version 3.7.

150 pppoe users, 8Mbits - 1000 packets - lost average 3 packets (it less than 1% but packet loss is still there).

I´m not happy with this result, old Mk versions never loose a single packet with much more users and traffic.

We also have packet loss 1% at least, but sometimes I've seen a 3% loss on a 600 packets

I think we should report to Mk support that we are not happy with this solution, or else they will believe it´s fine.

I cannot accept a new version that works worst than an old version. And loose packets in LAN enviroment make this solution useless.

I think we should report to Mk support that we are not happy with this solution, or else they will believe it´s fine.

I cannot accept a new version that works worst than an old version. And loose packets in LAN enviroment make this solution useless.

We didn't face any problem in 3.7 PPPoE
0 packet loss , with nearly 350 user
Packet loss only from 1 to 3 % for users with near there maxim speed limit and I think that is normal
We disabled compression and encryption, but still use Change TCP MSS
Server is 3.2 MHz, 512 Ram, multi-cpu=no
And we will try with more users ….

is the cpu sage little bit lower then the previous releases?

regards
Ros

is the cpu sage little bit lower then the previous releases?

regards
Ros

No I think it’s the same.

It seems they fixed a problem with traffic shape queue creation but it also lose packets when an authentication request is denied (so this is not related with queues). If you have users that try to authenticante entire day you will lose packets.

I have an old machine (Sempron AMD processor) with Mk 2.9.44 with about 300 users, more than 15Mbits and never lost a packet, same Mk configuration and user profile. With new versions even with a Pentium D 3.4Ghz 512M RAM (Intel Gigabit PCI-E interf.) with multi-core on or off I have packetloss.

At some point near Mk version 2.9.50 it starts to lost packets on user authentication request (denied or not).

And remember I´m talking about lose packets even at local interface (inside my local network).

Yes, we have exactly same situation.

Any experience on multi-cpu=yes about 3.7?
is it stable the smp verision about pppoe server?

regards
Ros

Any experience on multi-cpu=yes about 3.7?
is it stable the smp verision about pppoe server?

regards
Ros

For us since 3.4 it was already stable with multi-cpu=yes, but it´s not helping to enable the second core since I continue to have dropped packets.

Yes, we have exactly same situation.

Have you reported to Mk support you still have problem with it? We are trying to report to them but they insist it was solved (they really solved part of the problem). I know more people which continue to have this problem so I think everybody should report. Look like when you have a few more authenticantion request (even if all denied) it lose packets.

I sent an e-mail with a supout.rif, screenshot, and ping log of the problem.

I've found that doing:

ping 127.0.0.1 size=1500 interval=100ms

makes it really easy to see the problem. We run both PPPoE as well as DHCP. As Mikrotik indicated, it really does seem to be related to Simple Queues, because any time a PPPoE user disconnects/reconnects or a DHCP lease deassigns/assigned, both of which cause a Simple Queue to be dropped and added, we drop about 3 pings.

I haven't done enough testing to determine if it is the dropping of the queue or the building of the queue that causes the packet loss. Maybe it's both.

Everyone else who is seeing this, please send an e-mail to support@mikrotik.com so we can hopefully get this fixed.

Dave

Yes, we have exactly same situation.

me too

So you have a big QoS system and some little packet loss?
Bus as far as I know it is possible to limit traffic ONLY by dropping packets!!!

regarding the constant packet loss - did you really determined it?
if you say you have 3% packet loss thats mean you should have the same number with 1pps 10pps 100pps and 1000pps.

I just checked same of my routers (not only mikrotiks) with your unique method (ping 127.0.0.1) and almost in all I was able to see some pings missing.

So bottom line is - do you have real packet loss problem???? I know only one way how to determine it is - online gamers (Quake, Unreal tournament, Counter-Strike) - those guys will be first who will find out even the smallest disturbance in the network...

So you have a big QoS system and some little packet loss?
Bus as far as I know it is possible to limit traffic ONLY by dropping packets!!!

regarding the constant packet loss - did you really determined it?
if you say you have 3% packet loss thats mean you should have the same number with 1pps 10pps 100pps and 1000pps.

I just checked same of my routers (not only mikrotiks) with your unique method (ping 127.0.0.1) and almost in all I was able to see some pings missing.

So bottom line is - do you have real packet loss problem???? I know only one way how to determine it is - online gamers (Quake, Unreal tournament, Counter-Strike) - those guys will be first who will find out even the smallest disturbance in the network...

Packet loss occurs on interfaces that doesn´t have any QoS or queues applied. We are not talking about packetloss for a user thru Mk route. It happens on 100Mbits or Gigabit interface with no QoS applied. It must not happen on a regular router, never! I use Mk as PPPoE server for years and it never happens before. Something is clearly wrong here.

In my case this happen with only 200 users online, which is not much.

And if problem is only related to queues why I lose packets even when I authentication request is denied?

As best I can tell, the problem occurs where all packets on all interfaces are dropped for this period of time, regardless if they are packets to be shaped by the simple queue being added/removed. All packets on all interfaces drop for that 250 ms window (roughly) while the simple queue is being added/dropped. Yes, this is a very bad thing.

Dave

we are running 3.7 on our pppoe servers and are seeing packt loss and another strange issue:

when we run bandwidth test to any host directly connected to the switch about private network we are able to send only 10mbps of udp traffic at 1500.
If I run bandwidth test through a pc connected on the same switch we are able to send at full speed.

The pppoe servers are not configured with any qos system that can cut the bandwidth.
My idead is that on this scenario that could y a packt flow problem.

Regards
Ros

Are you using simple queues? If so you should really be running a later version of RouterOS since a performance issue regarding simple queues was corrected in version 3.8.

i think it was corrected into 3.7
I don't see any change loc about simple queue into the 3.8

regards
Ros

Sorry you are correct:

What's new in 3.7:

*) improved simple queue list updating;

but I still suggest you start using version 3.9 because very few people, including probably the folk at Mikrotik, will help you with prior versions. The problem is Mikrotik 3.x is new and changes very quickly so it's mostly better to use the latest version. Then again I also realize there are risks associated with upgrading routers.

We are running 3.9 on a PPPoE server serving about ~800 users. We are seeing all sorts of weird things. Some times proxy arp just does not seem to work right. Sometimes users just cannot go anywhere. And an old bug we thought we got rid of when we moved too 3.x of queues not always being enforced on PPPoE users. A user with a 256k profile will be seen running at 4.5m or so.

Tempted to downgrade too 3.3 or 3.4 release. Hopefully 3.10 will be out soon and solve all our problems?

Matt

We are seeing all sorts of weird things.

Ouch. Just for interest sake did you encounter any problems with 2.9.51? Is there a specific reason why you upgraded to 3.x?

I upgraded 2 x PC routers to 3.9 and both crashed...I didn't experiment much thereafter because they were production routers so I quickly downgraded. Since then no problems.

Personally I would like to implement 3.x routers because I need faster / larger SATA hard drives and more RAM for my proxy. But I can wait for stability.

Ouch. Just for interest sake did you encounter any problems with 2.9.51? Is there a specific reason why you upgraded to 3.x?

The main reason for the move to v3 was to get the fix for queues not always being inforced and moving to v3 seemed to fix it. We are short on bandwidth right now waiting for our new circuits to be installed and not inforcing queues was a real issue. Everything actually seemed to work fine with v3 until something like 3.5 or somewhere in there. Not at all sure when, its hard to keep track. I heard starting with 3.5 Mikrotik has a new kernel. Not sure if thats true.

Matt

As far as I know the Kernel change was between 2.9.x and 3.x.

Have you actually even tried 2.9.x and did you have a queue issue with that version?

Any stable version for PPPOE? We use 3.10 but we have queue problem...
2.9.51? Anybody use 2.9.51 to PPPOE? We downgrade now to 3.3 but now we have the packet loss problem before 3.7...

has someone that is running latest releas sent supout to MT about the not working queue?

I had this bug on some of the latest 2.9 releases. Now with 3.7 I dont see this behavior but i am still getting some packet loss.
My experience is that it is more pronunced on Broadcom tigon3 gbit ethernet then Intel gbit ethernet cards.

In fact i have two pppoe server with the same exact configuration and i see this problem only on the server equiped with broadcom tigon3 driver, on the other one with intel i don't lose the packets.


regards
Ros

Anybody use 2.9.51 to PPPOE?

I use 2.9.x and 2.9.51 on various servers with PPPoE client on high sites. The largest site has about 100 PPPoE clients. I have never had any problems with queues.

100 is a very small number.

You have to test them with 800-1000 clients.

Regards

for so many clients, I hope you are using RADIUS authentication ...

sure!

we have around 1200 on the intel gbit interfaces router and they are fine.
around 400 on the broadcom tigon and we are experiencing some little packet loss.......

same identical configuration to both machines. Supout sent two day ago to support.

Regards
Ros

We have some packetloss with version 3.7 and Intel Gigabit NIC. I don´t know if with Broadcom is worst but we have this problem with Intel (PCI or PCI-E).

strange that the same hardware (dell poweredge 860) configured as core router so without any dynamic simple queue but with a queue tree is working great without any packet loss even pinging the pppoe server with broadcom ethernet nics.

The vice versa pinging from pppoe server to the core router I lose packets..........very strange

Ros.

for so many clients, I hope you are using RADIUS authentication ...

Wouldn't an up to date PPPoE radius how too in the wiki be nice. Needs to cover IP pools, duplicate rejection, profiles and etc.

Matt

We have some packetloss with version 3.7 and Intel Gigabit NIC. I don´t know if with Broadcom is worst but we have this problem with Intel (PCI or PCI-E).

Is it a multi-cpu box or an old fashion single core?

Regards
Ros

We have some packetloss with version 3.7 and Intel Gigabit NIC. I don´t know if with Broadcom is worst but we have this problem with Intel (PCI or PCI-E).

Is it a multi-cpu box or an old fashion single core?

Regards
Ros

It´s a P4 3.4Ghz dual-core with intel server mobo.

uldis......

i think the actual packet loss is about an snm issue!!!!

this report from josefranco seems confirming my idea.

Regards
Ros

uldis......

i think the actual packet loss is about an snm issue!!!!

this report from josefranco seems confirming my idea.

Regards
Ros

Sorry about my bad english but what´s a snm?

Just for information I had this same problem with a P4 3.0 single CPU on a totally different hardware. It starts to happen for me on 2.9.4x version.

it means multi core cpu.

Regards
Ros

it means multi core cpu.

Regards
Ros

In this case I don´t think it´s related to multi-core cpu. I had this problem with single core CPU too.