2 WANs, 2 LANs, chaos ensues. :)

thermant · Mon Jan 18, 2010 3:01 am

Hi guys, I'm baackk!
Heh, that's kinda lame.

Anyhoo, right now I have 2 WANs (one of them has just beed added a few days ago), and 2 LANs on my router. Let's call em PUBLIC1 (domestic + international), PUBLIC2 (domestic only), PRIVATE (192.168.2.0/24), and LOCAL (192.168.1.0/24), shall we?

Here's how I setup the mangle part: (I've never used any mangle before, and this is my first attempt of writing my own mangle rules after reading some articles, tutorials, and the "QoS best practices" PDF. So bear with me if I made some stupid, yet obvious, mistakes)

 0   chain=prerouting action=accept src-address=192.168.2.0/24 

 1   chain=prerouting action=accept src-address=192.168.1.0/24 

 2   ;;; PRIVATE >> ALL Domestic + INTL traffic
     chain=prerouting action=mark-routing new-routing-mark=INTL_route passthrough=no in-interface=PRIVATE 

 4   ;;; LOCAL >> All HTTP Traffic
     chain=prerouting action=mark-routing new-routing-mark=INTL_route passthrough=no protocol=tcp in-interface=LOCAL dst-port=80 

 5   chain=prerouting action=mark-routing new-routing-mark=INTL_route passthrough=no protocol=tcp in-interface=LOCAL dst-port=8080 

 6   chain=prerouting action=mark-routing new-routing-mark=INTL_route passthrough=no protocol=tcp in-interface=LOCAL dst-port=3128 

 7   ;;; LOCAL >> All INTL Traffic 
     chain=prerouting action=mark-routing new-routing-mark=INTL_route passthrough=no dst-address-list=!domestic in-interface=LOCAL 

 8 X ;;; LOCAL >> All Domestic traffic 
     chain=prerouting action=mark-routing new-routing-mark=domestic_route passthrough=no dst-address-list=domestic in-interface=LOCAL 

13   ;;; new conn-mark = proxy_con >> new pkt-mark = proxy_pkt
     chain=output action=mark-connection new-connection-mark=proxy_con passthrough=yes content=X-Cache: HIT 

14   chain=output action=mark-packet new-packet-mark=proxy_pkt passthrough=no connection-mark=proxy_con 

15   ;;; !proxy_con >> new conn-mark = basic_client_con. adr-list=client_basic
     chain=forward action=mark-connection new-connection-mark=basic_client_con passthrough=yes src-address-list=client_basic in-interface=LOCAL connection-mark=!proxy_con 

16   ;;; basic_client_con >> new pkt-mark = basic_client_pkt
     chain=forward action=mark-packet new-packet-mark=basic_client_pkt passthrough=no connection-mark=basic_client_con 

17   ;;; !proxy_con >> new conn-mark = vip_client_con. adr-list=client_vip
     chain=forward action=mark-connection new-connection-mark=vip_client_con passthrough=yes src-address-list=client_vip in-interface=LOCAL connection-mark=!proxy_con 

18   ;;; vip_client_con >> new pkt-mark = vip_client_pkt
     chain=forward action=mark-packet new-packet-mark=vip_client_pkt passthrough=no connection-mark=vip_client_con 

23   ;;; !proxy_con >> new conn-mark = all_con (OUTPUT, tcp only) >> ftp_pkt & winbox_pkt
     chain=output action=mark-connection new-connection-mark=other_con passthrough=yes protocol=tcp connection-mark=!proxy_con 

24   chain=output action=mark-packet new-packet-mark=winbox_pkt passthrough=no protocol=tcp src-port=8291 connection-mark=other_con 

25   chain=output action=mark-packet new-packet-mark=ftp_pkt passthrough=no connection-mark=other_con connection-type=ftp

Rule 8 is currently disabled, due to the way I setup my gateways. I pointed everything with the routing mark INTL_route, to PUBLIC1, and those that do not have that routing mark goes to PUBLIC2. I might use it later, or I might not. Here's my ip/route printout:

 #      DST-ADDRESS        PREF-SRC        GATEWAY-STATE GATEWAY                                            DISTANCE INTERFACE                                  
 0 A S  ;;; Domestic Gateway
        0.0.0.0/0                          reachable     xxx.yyy.zzz.000                                1  PUBLIC2                                    
 1   S  ;;; Backup Gateway (Failover)
        0.0.0.0/0                          reachable     aaa.bbb.ccc.ddd                                   111      PUBLIC                                     
 2 A S  ;;; INTL Gateway
        0.0.0.0/0                          reachable     aaa.bbb.ccc.ddd                                     1        PUBLIC                                     
 3 ADC  xxx.yyy.zzz.000/28  xxx.yyy.zzz.000                                                                   0        PUBLIC                                     
 4 ADC  aaa.bbb.ccc.ddd/28    aaa.bbb.ccc.ddd                                                                     0        PUBLIC2                                    
 5 ADC  192.168.1.0/24     192.168.1.1                                                                      0        LOCAL                                      
 6 ADC  192.168.2.0/24     192.168.2.1                                                                      0        PRIVATE                                    
 7 ADC  192.168.168.0/24   192.168.168.1                                                                    0        PUBLIC

Anyway, here's the weird part:
You see mangle rule 0 & 1 there? If I enable them, I can connect from PRIVATE to LOCAL and vice versa, but all traffic, domestic and international, from PRIVATE will be forwarded to PUBLIC2. Which means there are NO international traffic for PRIVATE. I find this to be VERY ODD, since according to the mangle rule #1, all PRIVATE traffic should be marked with INTL_route, and therefore should go to PUBLIC1. I do not know about domestic & international traffic from LOCAL, I haven't had a chance to test it yet.

If I disable rule 0 & 1, I can't connect to the other LAN, but my outgoing traffic is forwarded just fine. Everything from PRIVATE is forwarded to PUBLIC1, like I wanted. Again, I haven't had the chance to go to the LOCAL site yet, so I can't test how things are there, but I have a hunch it's going to be the same.

Am I doing anything wrong? Do you guys need to see any other settings? Filter, NAT? I didn't change any of them, so it shouldn't matter... Well, there's this one line that's added by the ISP guys:

 0   ;;; NAT--Domestic
     chain=srcnat action=src-nat to-addresses=xxx.yyy.zzz.000 out-interface=PUBLIC2  
 1   ;;; Main Route
     chain=srcnat action=masquerade

I tried enabling/disabling rule #0 there, but it doesn't seem to do anything at all. Should/shouldn't it be there?

Well, that's all that was changed during this whole adding new WAN process. Hope you guys can provide some insights.

Oh, one more thing... Is the prerouting chain in mangle ordered in any way? I mean, they all have passthrough=no.

Thanks all.

-Z-

EDIT:
Oh, I just remembered. If mangle rule 0 & 1 is disabled, when I traced a connection to one of the LOCAL PCs from PRIVATE, the traffic was forwarded to PUBLIC1's gateway, and got lost there. So instead of PRIVATE--router--LOCAL, it went something like this: PRIVATE--router--GATEWAY PUBLIC1--... (eternal RTOs)... Hope this helps somewhat.

thermant · Mon Jan 18, 2010 10:46 am

OK, you guys, an update:

First, lemme ask this to see if I'm understanding this whole mangle thing correctly:
Mangle rules are processed in chains and from top to bottom, exactly like the firewall filter rules. And the only ways RoS will stop this top-to-bottom processing are:
1. when it finds a matching rule that contains "passthrough=no"
2. when it gets to the bottom of a chain and nothing matches
So as long as it doesn't find a matching rule that contains "passthrough=no", then it will continue going until the end of the chain.
QUESTION 1: Am I right so far?

Now, I switched the way my outgoing traffic is processed, and I enabled mangle 8. IF my understanding is correct, rule 8 will only be fulfilled by traffic that has passed through rule 2-7, which means it is:
1. NOT coming from PRIVATE
2. NOT coming from LOCAL and going to port 80, 8080, or 3128 (non-http)
3. matches the 'domestic' address list (non-international traffic)
Essentially, "leftover" traffic, since it doesn't match rule 2-7.
QUESTION 2: Am I right so far?
QUESTION 3: Am I missing any packets? I really don't think so, but...

After that, I switched the rules in IP/Route and made it so that all traffic with the domestic_route routing mark goes to the domestic gateway, and everything else (meaning routing mark INTL_route) to the other gateway. All is still working so far. I then disabled mangle rule 0 and 1, and turns out I CAN connect to the other LAN. Yay!

However, I find it to be quite odd. Is it because of mangle rule 2 (pass EVERYTHING from PRIVATE to PUBLIC)? I mean, the route to 192.168.1.0/24 is right there in the IP route table. If I were the router, and I was asked to pass a book to John, I would take the book with one hand (PRIVATE), pass it to the other hand (LOCAL), and pass it to John. I wouldn't take the book and put it in a slot (PUBLIC) that would transfer the book to who knows where...
So, QUESTION 4: Is the IP route table is processed from top to bottom as well?
QUESTION 5: IF it is processed from top to bottom, almost all routes in the table is dynamically assigned, so it would seem that I have no full control of the ordering... How would I tell the router that these packets MUST be passed on to this interface, no matter what? And how would I decide that this rule should be rule 0, this one should be rule 1, etc?

Only thing remaining is NAT rule 0 that the ISP guys added. I enabled/disabled it, and still don't see any changes. Only thing that happens when I disable it is the bandwidth usage went down quite a lot for 1-2 seconds, and then went right back up. Reorganizing packet routes, maybe?
QUESTION 6: Should I leave it there, or delete it? Anyone care to guess what the rule does?

QUESTION 7/BICKERING/COMPLAINING/ETC:

<phew> This whole thing is giving me quite a headache. But no pain, no gain, eh?
Anyway, once I got this whole mangle thing down, I will move on to traffic shaping & prioritizing using queue trees. I already got the basic packet markings there (mangle 15-18), and I just need to expand it to mark http, voip, etc, etc. I never used queue trees before, only simple queues (tons of em

). So I'm gonna be needing more of your guys' help soon.

Cheers!

-Z-

2 WANs, 2 LANs, chaos ensues. :)

2 WANs, 2 LANs, chaos ensues. :)

Re: 2 WANs, 2 LANs, chaos ensues. :)

Who is online