Community discussions

MikroTik App
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Web proxy cache

Mon Apr 20, 2009 2:46 pm

I've noticed one of my routers isn't as fast as it used to be and am investigating to find out what the problem is.

The router is a 433AH with a 4GB microSD card. The unit is running transprent web proxy, with unlimited cache on the SD card. The router is also running User Manager.

If I set the value of the web cache to "none" the end user's internet experience is good. If I switch the web cache on, the end user's internet experience is pretty awful.

My whole network runs on ROS 3.22

Any ideas why?

More importanly, any ideas why just this router? I have several that have the exact same configuration. The others are going quite nicely, but this one's lagging.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Wed Apr 22, 2009 9:50 pm

Please see this http://wiki.mikrotik.com/wiki/Supported ... mory_cards + http://wiki.mikrotik.com/wiki/Image:Pro ... torage.gif
maybe its a microSD card problem?

Please post some screenshots of the proxy etc.
Last edited by NetworkPro on Thu Apr 23, 2009 3:02 pm, edited 3 times in total.
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Wed Apr 22, 2009 10:16 pm

Please see this http://wiki.mikrotik.com/wiki/Kingston_ ... _on_RB450G
maybe its a microSD card problem?

Please post some screenshots of the proxy etc.
Basically the same as the link.

But mine gets better, I'm getting lots and LOTS of these in my syslog - after which the machine reboots itself.

What do I do?

2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/1/1d/1dfbd559d4ea666b2d22b84f2c1b9f95') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/1/1f/1ffa6b2703a2f08ef3da41c546add7ab') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/2/24/244978fe22f4a306569785dda424f632') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/2/27/27bf9f2e64ce9b41623942f28e52e480') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/2/2c/2c19b7e6e3b732e84269b5aade902a5b') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/2/2c/2c7a63e49dbf62467c97d0be736b984f') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/3/34/34e655098147688302f5a383376fb892') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/3/36/365549b97c9d79bab6538da296a8790f') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/3/38/38582c5e28d11be4e63f74e75459f8f7') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/3/3c/3ca99b179e559876e186ccf82e9d6c30') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/3/3c/3c83d11bde0a23f92253173b5b6a45a1') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/3/3f/3fa5474c661f1c54a7a30c3f3e8d9fde') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/3/3f/3f7f37fa53ab0b10ae1dd15bb5a4add9') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/4/41/4185d180d61132fb6d4fe4e1c3cd6ebb') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/4/46/465d2bd9e3d1fcb2a3d5f63ecf6e46d6') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/4/4a/4a12ffa4e543ffbd8de077a6811861e2') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/4/4b/4b2695b2e08af9708d5c14cc1241fecb') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/4/4b/4b1c097814211c3b6cf06cdb2f033141') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/5/50/50b969ab03c1546a860d88ddeb733d4b') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/5/51/51d7cd6deecee5e58798d09db5c39c51') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/5/53/537367da2f5afc628e9ee617244acbd1') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/5/56/568da7c2ac7b0f37e7c943a18242ca1d') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/5/5b/5b55ef5ace9dc714e349ac67c694baa5') failed: Stale NFS file handle
2009-04-22 21:09:28 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/5/5c/5c4b852e66e2bbb0505c9d769475950d') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/b/b4/b451e1a8d3333f324ffeb1bc2d85702b') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/d/d0/d0d11b3a41336c21684fb339bc29cbea') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/d/d3/d35dc41f865b82632020e0f299b12a04') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/d/dd/ddd47a0ce4ae35e7c8d09a13d0c4bade') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/d/dd/dd8469834628fe662a000368da461f8f') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/e/e0/e04e969f7bd5f57f05033ed1314df66c') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/e/e1/e1f0da2c0977090763c8448bb6dfc1d0') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/e/e1/e1604f2596df88aafbf3c57213957953') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/e/e3/e39f724e1737f8bb5890c28012c9e4b6') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/e/e4/e4a9b12df02064c74379a3001401b921') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/e/ec/ec273a41bdf19dc51f23e8076e4afd62') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/e/ed/ed9bae65909c7c29d707ee736a1f2028') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/e/ef/efa3fee4d966e737c79029c717a24e3d') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/e/ef/efbc3f8af8646350f7a72affc3a2f93d') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/f/f0/f06995feb9803dd37b809df29b38d864') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/f/f7/f7c3210cb9fffdcb59acbcc15f0730b0') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/f/f9/f94b7d27bec524b4d28275d0c1c03661') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/f/fa/fa0c419d66166e2ca7b65a47558ddb8e') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/f/fb/fb3cefe086508f72717e4a1e0d8d062c') failed: Stale NFS file handle
2009-04-22 21:09:29 Local7.Debug 192.168.75.254 web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/f/fd/fd43513616d0d358cfa4bfff880f2849') failed: Stale NFS file handle
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 2:28 pm

Try "Clear Cache" from the proxy, or Format from "Stores" :) If it does not work after that - replace the microSD.
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 3:13 pm

Try "Clear Cache" from the proxy, or Format from "Stores" :) If it does not work after that - replace the microSD.
Both disc write options fail due to the drive being "busy". We're waiting for the rain to stop so we can go and remove the microSD. I don't understand why it worked so well until I upgraded to 3.22
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 4:11 pm

Maybe the microSD is at the end of its life? How long have it served your caching needs? Has it been used for something else before that?
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 4:17 pm

Maybe the microSD is at the end of its life? How long have it served your caching needs? Has it been used for something else before that?
Commissioned on 27 Jan 2009. Brand new, out of the box.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 4:32 pm

It will not hurt to test with an earlier version. Upload the old packages and it should downgrade OK.
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 4:40 pm

It will not hurt to test with an earlier version. Upload the old packages and it should downgrade OK.
But not over 146 RB's :(

We have a policy of running the same version on all our boards and our client boards.

The card's on my desk. I'll pop it into a phone later and see if it works properly there.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 4:45 pm

It will not hurt to test with an earlier version. Upload the old packages and it should downgrade OK.
But not over 146 RB's :(

We have a policy of running the same version on all our boards and our client boards.

The card's on my desk. I'll pop it into a phone later and see if it works properly there.
Why not downgrade only the problematic RB?

Did you put another in the RB, does it work now?

You can test it with a mircoSD to SD adapter on a PC as well, since there you have more control to format it and speed test it, see bad sectors etc.
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 5:02 pm

It will not hurt to test with an earlier version. Upload the old packages and it should downgrade OK.
But not over 146 RB's :(

We have a policy of running the same version on all our boards and our client boards.

The card's on my desk. I'll pop it into a phone later and see if it works properly there.
Why not downgrade only the problematic RB?
It goes against our company policy to run different versions of operating systems in production enviroment. I just run the place. I don't decide policies.
Did you put another in the RB, does it work now?
I left it out. With so many web pages being dynamic these days, I think the days of web proxy cache may well be behind us.
You can test it with a mircoSD to SD adapter on a PC as well, since there you have more control to format it and speed test it, see bad sectors etc.
My widescreen Dell laptop with the inbuild SD reader has a blown screen and my multi card reader is out with one of the techs. I'll give it a shot when the reader's back.

The card appears to work fine in my phone though. I initialised it and it's happy.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 5:15 pm

There are times in life when policies must me worked-around in my opinion.

Anyway,

Web Proxy Cache is still very very very usable and quality of the end user experience is much higher due to Web Proxy Cache!

Can't you see "Hits sent to clients" how big that number gets on some proxies? It means it is working well ;)
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 5:38 pm

Can't you see "Hits sent to clients" how big that number gets on some proxies? It means it is working well ;)
I have one ROS machine on my network that's based on Intel hardware (P4 1GB RAM) with a 250GB SATA proxy with 6 radios (that go on to other APs) and that also provides IP to about 40 networks via UTP / fibre. It's connected to the outside world via 4 x 2Mb leased lines to our traffic provider - so it's a big machine.

Anyway, it runs web proxy and obviously there's a lot of traffic going through the proxy. There were 194 "hits sent to clients" on the proxy since the router was last rebooted some time in February 2009.

Web proxy is also processor intensive. We'll probably disable them throughout the organisation in the next month or so.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Thu Apr 23, 2009 5:41 pm

Something is wrong. Hits sent to clients should be A LOT. Like 2 gigabytes of data for example. And in my experience, what I saw, is tha MT 3.20+ Web Proxy uses very little resources.
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Mon Apr 27, 2009 3:31 pm

I managed to get the microSD card into it's adapter and into my SD card reader.

It's FUBAR and is going back to the suppliers on Tuesday.

It appears that these solid state drives only have a finite life. Each sector can only be written and overwritten a certain number of times before it's useless. That is why the space on memory sticks decreases. So I'm thinking maybe one of the critical sectors on the card was written to one time too many by RouterOS, so that's the end of the card.

Either way, I've been doing a bit of further research on web proxy cache at the various places where we break out to the internet. Some places have very high hit ratios, some have very low hit ratios. I'm putting it down to demographics. Older people have different browsing patters to young people. Oldies get more cache hits while the facebook generation get less cache hits.

Yes, I know it doesn't make sense, but this is what I see when I log into routers and compare them with each other.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Mon Apr 27, 2009 4:27 pm

Sure. Just run the cache near the border gw so it catches http requests by all users. So if one users visits site and causes a bunch of objects get cached, the next user visiting that site could get them from cache. I would even go so far as to request my upstream ISP to run a high performance big drive cache proxy.
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Mon Apr 27, 2009 6:19 pm

Sure. Just run the cache near the border gw so it catches http requests by all users. So if one users visits site and causes a bunch of objects get cached, the next user visiting that site could get them from cache. I would even go so far as to request my upstream ISP to run a high performance big drive cache proxy.
I have six border gateways in two different countries.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Mon Apr 27, 2009 6:33 pm

The Web Proxy Cache must be in a place where the end user has high speed to it and at the same time - the more users go through a cache proxy - the better. So 6 proxies in your case maybe ?
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Mon Apr 27, 2009 6:43 pm

The Web Proxy Cache must be in a place where the end user has high speed to it and at the same time - the more users go through a cache proxy - the better. So 6 proxies in your case maybe ?
We are actually running more than 6 proxies, but as I said previously it's something we are looking at scaling down on.

For example, if we have an access point using a P4 with 1GB of RAM, ROS Level 5 and 4 x 90 degree 2.4GHz sectoral, 4 x 90 degree 5GHz sectoral and 5GHz NStream (this is a typical medium site with 10 radios) back to one of the our gateways, we'll have at least 60GB of SATA proxy. We've only ever once used solid state memory and that was with the RB433AH on a microsite and that failed.

We have a very large MikroTik network.
 
User avatar
agsochi
newbie
Posts: 26
Joined: Wed Apr 22, 2009 8:41 pm

Re: Web proxy cache

Tue Apr 28, 2009 3:17 pm

I have same problem with my RB450G and two microSD cards (one SDHC 4gb, and another SD 2gb)
http://forum.mikrotik.com/viewtopic.php?f=3&t=31325
nobody answers me!
 
User avatar
janisk
MikroTik Support
MikroTik Support
Posts: 6263
Joined: Tue Feb 14, 2006 9:46 am
Location: Riga, Latvia

Re: Web proxy cache

Tue Apr 28, 2009 3:30 pm

also, parent proxy feature can be used with caching enabled on the proxy, as result you will get more hits on your main proxies, and utilize your network.
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Tue Apr 28, 2009 7:48 pm

also, parent proxy feature can be used with caching enabled on the proxy, as result you will get more hits on your main proxies, and utilize your network.
We're doing that :D
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Sun May 03, 2009 4:25 am

I just had the same problem :( It happened like this:

Cache was fine on 2GB microSD, almost full; opened many many tabs at once with firefox+fasterfox add-on+dns prefetch add-on (XP32+EvID4226Patch223d); RB450G crashed - rebooted by watchdog; after restart cache was corrupt and saw errors in log; disabled proxy, cleared cache; complained to support about it explaining all; enabled proxy on cleared microSD; tried opening many tabs without fasterfox add-on but to my amazement - RB450G CPU Usage was very very high and went to 100% leading to major loss in end-user experience/performance!

Conclusions: MT web-proxy will work bad with many concurrent connections and may even crash and reboot the router.

MikroTik Latvia, any guide for us about this ? Any patches and fixes? Optimizations? Workarounds? "best practice" advices? Thanks!.

P.S. by the way, I have set the max connections for the proxy both to 5000 the maximum. Maybe there is a safer lower value for these? But I couldnrt find it out by experiment, maybe someone could suggest these for RB450G ?
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Thu May 21, 2009 7:33 pm

OK New test today: a brand new RB450G, updated 3.24 pre-relase downlaoded today at noon. Web-proxy cache turned up to 5000 connections. Using RAM as cache storage.

Opnening many many tabs with Firefox (mods like my above post) makes not more than 50% CPU usage of the board. So I will use the RAM for cache from now on instead of the cheap microSDs' ... and maybe wait for much faster microSDs to come out.

Anyone else tested something like this ?
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Thu May 21, 2009 7:40 pm

.
Anyone else tested something like this ?
Here's a little test that I'm running on a RB433AUH - also in RAM.
You do not have the required permissions to view the files attached to this post.
 
User avatar
agsochi
newbie
Posts: 26
Joined: Wed Apr 22, 2009 8:41 pm

Re: Web proxy cache

Sat May 23, 2009 10:16 am

... and maybe wait for much faster microSDs to come out. Anyone else tested something like this ?
I tested 3 microSD - one slow, another class4, and class6 microSD (it's very fast - view screenshot).
8Gb class6 I using in RB540G but still errors appear (like in two others):
...
May/19/2009 09:01:54 web-proxy,error stat('/mnt/disk2/sd/c/c3/c37c60903e6bf39da28fdd73dc191bc6') failed: Stale NFS file handle
May/19/2009 09:01:54 web-proxy,error stat('/mnt/disk2/sd/c/c3/c31a9ce06efdc6519c7e97f23db80d88') failed: Stale NFS file handle
...

I think that reason is not SD.
You do not have the required permissions to view the files attached to this post.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Sat May 23, 2009 11:01 am

Maybe we have to set a maximum cache limit, which is the amount of space on the microSD but slightly less. For example for 2GB microSD 1.6G or 1.8G which will be 1600000 ?
 
User avatar
agsochi
newbie
Posts: 26
Joined: Wed Apr 22, 2009 8:41 pm

Re: Web proxy cache

Sat May 23, 2009 11:27 am

Maybe we have to set a maximum cache limit, which is the amount of space on the microSD but slightly less. For example for 2GB microSD 1.6G or 1.8G which will be 1600000 ?
I set 500mb on 2gb card - but still got errors.
 
InoX
Forum Guru
Forum Guru
Posts: 1966
Joined: Tue Jan 09, 2007 6:44 pm

Re: Web proxy cache

Sat May 23, 2009 11:49 am

those errors appears on reboot and is normal
 
User avatar
agsochi
newbie
Posts: 26
Joined: Wed Apr 22, 2009 8:41 pm

Re: Web proxy cache

Sat May 23, 2009 12:02 pm

those errors appears on reboot and is normal
May be some errors after reboot its normal, but what is the reason of rebooting? - watch dog (watchdog timer)!
When I use web-cache in RAM - 10 days - ok.
When I turn cache on disk - since 9:00 to 10:00 - several reboots. Is it normal?
 
InoX
Forum Guru
Forum Guru
Posts: 1966
Joined: Tue Jan 09, 2007 6:44 pm

Re: Web proxy cache

Sat May 23, 2009 6:37 pm

those errors appears on reboot and is normal
May be some errors after reboot its normal, but what is the reason of rebooting? - watch dog (watchdog timer)!
When I use web-cache in RAM - 10 days - ok.
When I turn cache on disk - since 9:00 to 10:00 - several reboots. Is it normal?
i was talking about system reboot or power failure
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Wed May 27, 2009 7:08 pm

I turned off web proxy on three routers. It's not stable. Platform is mipsbe. Support are quiet about it. That is not good.
 
InoX
Forum Guru
Forum Guru
Posts: 1966
Joined: Tue Jan 09, 2007 6:44 pm

Re: Web proxy cache

Wed May 27, 2009 11:26 pm

proxy is stable on x86...after 30days of settings...
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Thu May 28, 2009 12:02 am

:) I noticed its stable on x86 but on mipsbe - it goes to 100% CPU sometimes, even with no requests to it. A memory leak of some kind? I don't know. Then, watchdog may reboot it or it may not reboot it leaving for me to reboot manually and what do I see - an autosupout file.
 
InoX
Forum Guru
Forum Guru
Posts: 1966
Joined: Tue Jan 09, 2007 6:44 pm

Re: Web proxy cache

Thu May 28, 2009 3:44 am

RB CPU's are not design for cache, even a 3Ghz CPU hits 100% if you have 100 users and 40GB of cache. Dual core will save you.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Thu May 28, 2009 4:00 am

Bullshit. CPU stays 100 percent forever after activity. There is a bug - thats problem. I tested it numerous times on CPUs like 600MHz. It can be used with 1-2-3-4-5 clients behind it!
 
User avatar
sergejs
MikroTik Support
MikroTik Support
Posts: 6697
Joined: Thu Mar 31, 2005 3:33 pm
Location: Riga, Latvia
Contact:

Re: Web proxy cache

Fri May 29, 2009 12:15 pm

2009-04-22 21:09:28	Local7.Debug	192.168.75.254	web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/1/1d/1dfbd559d4ea666b2d22b84f2c1b9f95') failed: Stale NFS file handle
2009-04-22 21:09:28	Local7.Debug	192.168.75.254	web-proxy,error CC_HP-1: stat('/mnt/disk2/microSD-proxy/1/1f/1ffa6b2703a2f08ef3da41c546add7ab') failed: Stale NFS file handle
For the errors like this, we have made few very important improvements in version 3.24. Non-system disk handling is improved significantly. Right now there should no be such errors in 3.24.

Note, when upgrading to 3.24 from older version, some disk checks are necessary, which can take few moments (when you may think that cache/disk is not working properly), after it non-system disk should work perfectly.
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Fri May 29, 2009 12:28 pm

Cool. I will try to enable Web Proxy on the 2GB microSD again :) If it fails - the clients do not have an alternative ISP he he. But what if they had and what if they were pissed.. ? Will find out...

p.s. by the way I bet 10 euro that there will be problems :P
Last edited by NetworkPro on Fri May 29, 2009 12:30 pm, edited 1 time in total.
 
User avatar
eben
Member
Member
Topic Author
Posts: 479
Joined: Mon Feb 16, 2009 8:37 pm
Location: Somerset West, South Africa
Contact:

Re: Web proxy cache

Fri May 29, 2009 12:29 pm

I've set up Watchdog on a couple of my machines running web proxy to email me the supout file if there's a crash.

Now we wait :)
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Fri May 29, 2009 12:33 pm

Hey wait a second! This fix was not in the changelog! Changes are important to my work! Please list them all!!!!
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26918
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia
Contact:

Re: Web proxy cache

Fri May 29, 2009 12:35 pm

we list what we can, usually there are 30% more changes than listed, some are minor, and don't fix specific bugs, but change underlying mechanisms that probably also improve other functionality.
 
YUJOBIKA
Member Candidate
Member Candidate
Posts: 122
Joined: Sat Sep 15, 2007 5:55 pm

Re: Web proxy cache

Fri Jun 05, 2009 8:16 am

In v3.24 such problem "failed: Stale NFS file handle" is no more exist in my log.
But....everyday...still... my Mikrotik reboot by its self if I use internal proxy. This self reboot also happen in v3.23.

And yes, I did "check disk" after upgrade to v3.24. The process took about 15 minutes.
But I did not "clean disk" and "format disk"....should I do it too ?

My system in USB Flash Disk 1GB
My cache in HD SATA 250GB
/system resource> pr 
                   uptime: 7h7m30s
                  version: "3.24"
              free-memory: 1298076kB
             total-memory: 1945864kB
                      cpu: "Intel(R)"
                cpu-count: 2
            cpu-frequency: 2992MHz
                 cpu-load: 13
           free-hdd-space: 907884kB
          total-hdd-space: 987124kB
  write-sect-since-reboot: 768856
         write-sect-total: 768856
        architecture-name: "x86"
               board-name: "x86"
 /ip proxy> pr 
                 enabled: yes
             src-address: 0.0.0.0
                    port: 31283
            parent-proxy: 0.0.0.0
       parent-proxy-port: 0
     cache-administrator: ""
          max-cache-size: 50000000KiB
           cache-on-disk: yes
  max-client-connections: 5000
  max-server-connections: 5000
          max-fresh-time: 3d
   serialize-connections: yes
       always-from-cache: yes
          cache-hit-dscp: 10
             cache-drive: primary-master
/ip proxy> monit
                 status: running
                 uptime: 7h13m1s
               requests: 471967
                   hits: 134090
             cache-used: 18218579KiB
         total-ram-used: 33994KiB
  received-from-servers: 6020748KiB
        sent-to-clients: 6473022KiB
   hits-sent-to-clients: 241775KiB

Anyone have problem like me ? How to solve it ?


Regards
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Fri Jun 05, 2009 9:14 am

9 days without a problem when the clients do not download from proxy at full speed put are limited with the QoS for the upstream link/proxy traffic not exlcuded from QoS max-limit.

So MikroTik, maybe you should really stress it hard when testing. No limits, a lot of conenctions, etc..
 
YUJOBIKA
Member Candidate
Member Candidate
Posts: 122
Joined: Sat Sep 15, 2007 5:55 pm

Re: Web proxy cache

Fri Jun 05, 2009 9:48 am

Yes, I always limit "Proxy-Hit" to clients. 512 Kbps for each client. Because I don't want my wireless link overload.

I am using PCQ with rate 512 Kbps. But still...self reboot happen :(
 
User avatar
NetworkPro
Forum Guru
Forum Guru
Posts: 1376
Joined: Mon Jan 05, 2009 6:23 pm
Location: bit.ly/the-qos
Contact:

Re: Web proxy cache

Fri Jun 05, 2009 10:06 am

Is it grounded properly? I have this hunch problem could be static eletricity, improper grounding etc. etc.
 
YUJOBIKA
Member Candidate
Member Candidate
Posts: 122
Joined: Sat Sep 15, 2007 5:55 pm

Re: Web proxy cache

Fri Jun 05, 2009 10:21 am

Grounding is xtremely good for more than 2 years.
Because I had a bad experience with ligthning.

I am using:
Grounding < 0.5 Ω
Cable BCC-50 with depth 40 Meters.


Regards
 
User avatar
mramos
Member Candidate
Member Candidate
Posts: 231
Joined: Sun Nov 23, 2008 1:05 am
Location: S. B do Campo - SP - Brazil

Re: Web proxy cache

Thu Jun 11, 2009 9:39 pm

Yes, I always limit "Proxy-Hit" to clients. 512 Kbps for each client. Because I don't want my wireless link overload.

I am using PCQ with rate 512 Kbps. But still...self reboot happen :(
Donno if aplies 'cause I'm running a vy small system but:

I use PCQ for internet traffic and unlimited BW from web-proxy to clients. My concept is = better a 6Mbps peak from web-proxy to clients for a fraction of a second than make them wait for a page to be fully loaded. The client "feeling" this way is a better surfing experience as if there is an higher bw internet connection than it really has.

And proxy will be doing two jobs anyway: reducing internet requests and delivering contents at higher speeds to clients. Good for me and for them.

CPU loads are more or less the same as imposed by PCQ queues routines.
I am using PCQ with rate 512 Kbps. But still...self reboot happen
Well, yesterday I faced a problem like this at one of my cards that is under burn in at the bench (WRAP).

As soon as any big file that was at the web-proxy storage was requested cpu load jumps to 100% and router reboot.

At this specific HW I was using primary-disk (Sandisk CF) as proxy storage. At the others (same HWs) I use 2GB pen-drives as web-proxy storages (high speed ones, e.g. those ok for ready-boost on Vista).

So I installed one pen-drive at this HW. But the sucessive reboots still going on. 100% CPU load at the proxy hits.

Downgraded to 3.23 and the problem still the same.

I have a single SR2 card on it. Removed the card, clean up mini-pci contacts, nothing happened.

Then I removed the wall-transformer power supply and started using a good power supply on it. Bingo! No more reboots, CPU loads when proxy-hit occur dropped from 100% to 10 ... 12 % (single client). Seems that this 16V @ 2A rated wall transformer in fact has too much ripple (it's not a switching ps but a linear unregulated one).

The PS that solved the problem is a regulated 2A @ 12Vdc (linear).

The behaviour was the same or using the on-board PS jack or POE with 20m CAT5 cables using this good PS or 12V from an ATX PC power supply.

Regards;
 
InoX
Forum Guru
Forum Guru
Posts: 1966
Joined: Tue Jan 09, 2007 6:44 pm

Re: Web proxy cache

Fri Jun 12, 2009 3:45 am

on my 80GB hdd I can only use 60GB or 62914560KiB or the proxy will reset the sistem.

Who is online

Users browsing this forum: certinet, naazh, onepixet and 68 guests