Configuring Speed from proxy to client?

thermant · Tue Nov 10, 2009 9:16 pm

Hi all,

I've setup transparent proxying on my router, like so:

 1   ;;; Proxy for Local LAN
     chain=dstnat action=redirect to-ports=8080 protocol=tcp src-address=192.168.1.0/24 dst-address=!192.168.1.0/24 in-interface=LOCAL dst-port=80 

 2   chain=dstnat action=redirect to-ports=8080 protocol=tcp src-address=192.168.1.0/24 dst-address=!192.168.1.0/24 in-interface=LOCAL dst-port=3128 

 3   chain=dstnat action=redirect to-ports=8080 protocol=tcp src-address=192.168.1.0/24 dst-address=!192.168.1.0/24 in-interface=LOCAL dst-port=8080

I've also setup a simple queue for each client in my network, limiting them to speeds of 256k/512k, like so:

16    name="A01" target-addresses=192.168.1.101/32 dst-address=0.0.0.0/0 interface=all parent=none direction=both priority=8 queue=default-small/default-small limit-at=0/0 max-limit=256k/512k burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s total-queue=default-small

Now here's my question:
Let's say I want to get a file from download.com for 10MB, and this file then is then stored (or "cached", is that the word?) in the mikrotik proxy. If then another client want to get the same file, it would get the file from the proxy (well, at least I think it would), but at the same limited speed (512k).

Is there ANY way to bypass this queue rule for data that's already stored in the proxy? So, maybe use 10M/100M for cached data, and use limited speed for stuff that's not cached... Is this possible at all?

I'm thinking of using mangle on traffic going to 192.168.1.1 (that's the router's address) port 80, 3128, and 8080, and then creating a separate simple queue rule for each client for 10M/10M. But then wouldn't that render the previous rules useless, since ALL traffic going to those ports (whether it's for cached or uncached data) would be using the new rule?

Pretty confused newbie here.
Help please.

fewi · Wed Nov 11, 2009 4:26 am

You are correct. You cannot speed up just content that was already cached.

skillful · Wed Nov 11, 2009 6:06 pm

You are correct. You cannot speed up just content that was already cached.

Of course you can!

1. Assign a Cache hit DSCP (TOS) value in webproxy settings. The default in ROS is 4.

/ip proxy
set cache-hit-dscp=4

2. Set up a mangle rule on the output chain to mark packets with DSCP=4. Place this rule ahead of other mangle rules.

/ip firewall mangle add action=mark-packet chain=output comment="CACHE HIT" disabled=no dscp=4 new-packet-mark=cache-packets place-before=0 passthrough=no

3. Make a simple queue for cache hit packets. Place this rule ahead of other queues.

/queue simple
add burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s comment="" direction=both disabled=no dst-address=0.0.0.0/0 interface=all limit-at=0/0 max-limit=10M/100M name=cached packet-marks=cache-packets parent=none priority=8 queue=default/default target-addresses=0.0.0.0/0 total-queue=default place-before=0

Thats all!

fewi · Wed Nov 11, 2009 7:24 pm

Wow. Very nice!

thermant · Wed Nov 11, 2009 9:29 pm

Awesome. I'll try it right away and let you know how it goes.

Thanks.

thermant · Fri Nov 13, 2009 9:28 am

@skillful,

Tried everything you suggested (dscp, mangle, and the new queue), then I download 2 files that I know is in the cache-content. Can't seem to bypass the existing queue... It's still in effect.

Any other ideas?

-Z-

fewi · Fri Nov 13, 2009 7:53 pm

Thinking about it, if I remember correctly dynamic simple queues created by logging into a Hotspot always get inserted at the top so your rule wouldn't hit since it moves down the list every time someone logs in. Maybe try deleting the simple queue hitting on the packet mark and use this one instead:

/queue tree
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=100M name=from-proxy packet-mark=cache-packets parent=global-out priority=8 queue=default

To grant 100Mbps to downstream from the proxy. Queue trees attached to global-out should be processed before simple queues.

thermant · Mon Nov 16, 2009 9:28 am

Hmm, still doesn't work...

See, this is the simple queue that I use for testing:

14    name="Netbook" target-addresses=192.168.2.204/32 dst-address=0.0.0.0/0 
      interface=all parent=none direction=both priority=8 
      queue=default-small/default-small limit-at=0/0 max-limit=384k/1M 
      burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s 
      total-queue=default-small

Deactivated the simple queue proposed by skillful, changed the max-limit to 64k/64k, activated the queue tree rule, and download a 4MB file that is in cache-content. I got between 4kbps to 7kbps, which is within the 64k limit, and never faster than that.

Am I doing something wrong with my simple queue?

skillful · Tue Nov 17, 2009 10:05 pm

Please post the output of the following:

/ip fire man pr without-paging

/que sim pr without-paging

/ip proxy pr

thermant · Sat Nov 21, 2009 10:19 am

Your mangle rule is currently disabled, since it doesn't work when I tried it...

/ip fir man pr without-paging 
Flags: X - disabled, I - invalid, D - dynamic 
 0 X ;;; CACHE HIT
     chain=output action=mark-packet new-packet-mark=cache-packets 
     passthrough=no dscp=4

You want -all- of the simple queues? I have like a hundred of em... I'll just pasted the first 5 or something, yeah?

/que sim pr without-paging
Flags: X - disabled, I - invalid, D - dynamic 
 0 X  name="CACHED DATA" target-addresses=0.0.0.0/0 dst-address=0.0.0.0/0 
      interface=all parent=none packet-marks=cache-packets direction=both 
      priority=8 queue=default/default limit-at=0/0 max-limit=100M/100M 
      burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s 
      total-queue=default 
 6    name="Billing" target-addresses=192.168.1.100/32 dst-address=0.0.0.0/0 
      interface=all parent=none direction=both priority=8 
      queue=default-small/default-small limit-at=0/0 max-limit=512k/1M 
      burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s 
      total-queue=default-small 
10    name="Ted" target-addresses=192.168.2.254/32 dst-address=0.0.0.0/0 
      interface=all parent=none direction=both priority=8 
      queue=default-small/default-small limit-at=0/0 max-limit=1M/1M 
      burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s 
      total-queue=default-small 
16    name="Christine" target-addresses=192.168.2.206/32 dst-address=0.0.0.0/0 
      interface=all parent=none direction=both priority=8 
      queue=default-small/default-small limit-at=0/0 max-limit=24k/96k 
      burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s 
      total-queue=default-small 
17    name="A01" target-addresses=192.168.1.101/32 dst-address=0.0.0.0/0 
      interface=all parent=none direction=both priority=8 
      queue=default-small/default-small limit-at=0/0 max-limit=256k/512k 
      burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s 
      total-queue=default-small 
18    name="A02" target-addresses=192.168.1.102/32 dst-address=0.0.0.0/0 
      interface=all parent=none direction=both priority=8 
      queue=default-small/default-small limit-at=0/0 max-limit=256k/512k 
      burst-limit=0/0 burst-threshold=0/0 burst-time=0s/0s 
      total-queue=default-small

The first one (rule 0) is yours, no? After I tried it and found that it doesn't work, I disabled it, as also fewi's queue tree rule... And finally...

/ip proxy print            
                 enabled: yes
             src-address: 0.0.0.0
                    port: 8080
            parent-proxy: 0.0.0.0
       parent-proxy-port: 0
     cache-administrator: "theSuperAdmin"
          max-cache-size: unlimited
           cache-on-disk: yes
  max-client-connections: 600
  max-server-connections: 600
          max-fresh-time: 8w4d
   serialize-connections: no
       always-from-cache: no
          cache-hit-dscp: 4
             cache-drive: primary-master

Oh, I have one more question... When I turned off/rebooted the router, are ALL cached data deleted? I thought it wouldn't be deleted, since it's stored on the hard drive. But whenever I checked the proxy status, the HD usage is always below 1G, while my clients downloaded quite a bunch of items... I expect it to be somewhere around 2-4 G... Any ideas?

vzouh · Thu Dec 17, 2009 1:59 pm

why don't you try to use external proxy instead of internal proxy from Mikrotik ?
when i use ROS v.2xx it's internal proxy work great... but after using v.3.xx i prefer to use squid on external proxy... work better for me especialy if you want to mark "X-cache: HIT"

thermant · Sat Dec 26, 2009 2:08 pm

Speaking of squid...

I -am- planning to use squid as an external proxy, but am currently confused about the positioning of the proxy. You see, I have not set up my clients to use any proxy, and set up mikrotik as a transparent proxy that listens on port 80, 3128, and 8080. I also use the mikrotik proxy as a web filter, blocking/allowing only certain sites that I set.

Here's where it gets confusing for me:
1. Let's say my local net is 192.168.1.0/24. I installed squid on a windows machine (I'm still learning to use FreeBSD, will migrate squid to that once I'm more familiar with it) with the IP 192.168.1.252. Should squid act just like another client (just one ethernet cable plugged in to a hub), or as a bridge (2 ethernet cards & 2 cables)? If it's a bridge, should I put it between the router and the client, or between the ISP and the router? I'm more inclined to go with the 1 ethernet card answer, but I'm not quite sure.

2. Performance-wise, would it be better just to use squid as the only proxy (which means I have to input the squid machine's IP and port to ALL my clients *gulp*)? Or would it be better to use it as a parent proxy with the current mikrotik proxy as the child proxy?

3. Any other caveats that I must know with regards to implementing an external proxy?

Thanks guys.

-Z-

vzouh · Sun Dec 27, 2009 9:41 pm

have you try to enable "always from cache" on web-proxy ?

and about squid you can put it as a bridge or same level with Mikrotik or your local...
for your multiple port, you can simply dst-nat them going to squid ip:port and make your squid transparent or you can use web-proxy to forward your proxy acces to squid as parent proxy, but this is not recomended coz you can't cache https on MT cmiiw

and also using squidNT is not recomended

vzouh · Sun Dec 27, 2009 9:53 pm

i use this to mangle proxy cache and then make simple queue to give higher speed for proxy cache

add chain=postrouting content="X-Cache: HIT" action=mark-connection new-connection-mark=proxy-con passthrough=yes comment="HIT proxy" disabled=no 
add chain=postrouting content="X-Cache: HIT" action=mark-packet new-packet-mark=proxy-pkt passthrough=no comment="" disabled=no

and simple queue

/ queue simple 
add name="Proxy-Hits" target-addresses=10.0.1.0/24 dst-address=0.0.0.0/0 interface=all parent=none packet-marks=proxy-pkt direction=both priority=2 \
    queue=default-small/default-small limit-at=0/0 max-limit=0/0 total-queue=default-small disabled=no

hope it will work to yours

Configuring Speed from proxy to client?

Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Re: Configuring Speed from proxy to client?

Who is online