Load Balancing + Port Forwarding

spookman · Mon Feb 28, 2011 10:06 am

Come Guys help me out here

WAN IP: (ISP xDSL modems, we can't make the RG750 pppoe as the modems/routers are locked down)
ether1 : 10.0.0.100
ether2 : 10.0.1.100
LAN IP
ether1 : 10.10.0.100

1. We also have an internal email and webserver on 10.10.0.250 which needs to be accessible over WAN1 and WAN2, on ports (21,25,80,110,443)
2. Out going mail goes to 2 different ISP's so basically if trying to deliver mail on ISP1(IP:x.x.x.x) traffic must go through WAN1 and for ISP2(IP:x.x.x.x) WAN2
3. Failover, if 1 link should fail traffic should failover to the other as we can't control the ISP modems the gateway-check can't help, we need to actually check against the ISP's gateway 2 hops away.

All inbound traffic must be FIRE-WALLED so traffic is block except for the ports in 2. are forwarded to the internal servers.

All help would be appreciated, even better if descriptions are added to why it's done that way.

/ip address
add address=10.10.0.1/24 broadcast=10.10.0.255 comment=LAN disabled=no interface=ether5 network=10.10.0.0
add address=10.0.0.100/24 broadcast=10.0.0.255 comment=LinkSys disabled=no interface=ether1 network=10.0.0.0
add address=10.0.1.100/24 broadcast=10.0.1.255 comment=NetGear disabled=no interface=ether2 network=10.0.1.0

/ip route
add check-gateway=ping comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.0.0.1 routing-mark=GW1 scope=30 target-scope=10
add comment="" disabled=no distance=10 dst-address=0.0.0.0/0 gateway=10.0.1.1 routing-mark=GW1 scope=30 target-scope=10
add check-gateway=ping comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.0.1.1 routing-mark=GW2 scope=30 target-scope=10
add comment="" disabled=no distance=10 dst-address=0.0.0.0/0 gateway=10.0.0.1 routing-mark=GW2 scope=30 target-scope=10

/ip firewall connection tracking
set enabled=yes generic-timeout=10m icmp-timeout=10s tcp-close-timeout=10s tcp-close-wait-timeout=10s tcp-established-timeout=1d tcp-fin-wait-timeout=10s tcp-last-ack-timeout=10s \
    tcp-syn-received-timeout=5s tcp-syn-sent-timeout=5s tcp-syncookie=no tcp-time-wait-timeout=10s udp-stream-timeout=3m udp-timeout=10s

/ip firewall filter
add action=drop chain=forward comment="" connection-state=invalid disabled=no
add action=accept chain=forward comment="" disabled=no dst-address=10.10.0.10 dst-port=3389 protocol=tcp
add action=accept chain=forward comment="" connection-state=established disabled=no
add action=accept chain=forward comment="" connection-state=related disabled=no
add action=drop chain=forward comment="" disabled=no in-interface=ether1
add action=drop chain=forward comment="" disabled=no in-interface=ether2
add action=accept chain=forward comment="" disabled=no

/ip firewall mangle
add action=mark-connection chain=prerouting comment="CM for GW1" disabled=no in-interface=ether5 new-connection-mark=GW1 passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting comment="CM for GW2" disabled=no in-interface=ether5 new-connection-mark=GW2 passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-connection chain=output comment="CM for GW1 - output" connection-mark=no-mark disabled=no new-connection-mark=GW1 passthrough=yes per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=output comment="CM for GW2 - output" connection-mark=no-mark disabled=no new-connection-mark=GW2 passthrough=yes per-connection-classifier=both-addresses:2/1
add action=mark-connection chain=input comment="CM input GW1" connection-mark=no-mark disabled=no in-interface=ether1 new-connection-mark=GW1 passthrough=yes
add action=mark-connection chain=input comment="CM input GW2" connection-mark=no-mark disabled=no in-interface=ether2 new-connection-mark=GW2 passthrough=yes
add action=mark-routing chain=prerouting comment="RM for GW1" connection-mark=GW1 disabled=no in-interface=ether5 new-routing-mark=GW1 passthrough=yes
add action=mark-routing chain=prerouting comment="RM for GW2" connection-mark=GW2 disabled=no in-interface=ether5 new-routing-mark=GW2 passthrough=yes

/ip firewall nat
add action=masquerade chain=srcnat comment="Masq for GW1" disabled=no out-interface=ether1
add action=masquerade chain=srcnat comment="Masq for GW2" disabled=no out-interface=ether2
add action=dst-nat chain=dstnat comment="" disabled=no dst-address=10.0.1.100 dst-port=443 protocol=tcp to-addresses=10.10.0.10 to-ports=3389

/ip firewall service-port
set ftp disabled=no ports=21
set tftp disabled=no ports=69
set irc disabled=no ports=6667
set h323 disabled=no
set sip disabled=no ports=5060,5061
set pptp disabled=no

spookman · Fri Mar 04, 2011 1:21 pm

Come guys give us a hand here.

It would now seem when I try connecting to the ether2 on 443 the dst-nat is happening but it looks like it's returning on the wrong GW, I can't be sure though

pedja · Fri Mar 04, 2011 1:35 pm

It was already asked, some suggestions are provided but none of them works.

Basicaly, you should mark incomoing connection to remmeber incoming interface, and then route repsonces using that mark to send them to the sam interface. Looks like simple but I did not manage to make it work, and as far as I could see, I am not alone.

This is something that should work by default without specific settings.

spookman · Fri Mar 04, 2011 2:26 pm

Strange, because by all the post I have read it seems that there are people out there doing this but no one can tell us how,

The routers then become useless if they can do nothing more comlpex than load balancing when you want to host servers behind them/

mrz · Fri Mar 04, 2011 2:34 pm

You have to mark connections coming from ether1 and ether2 in forward chain and ensure that servers respond to the same gateway from which connection came in.

spookman · Fri Mar 04, 2011 2:48 pm

Ok can you give me an example ?

pedja · Sun Mar 06, 2011 3:42 am

Strange, because by all the post I have read it seems that there are people out there doing this but no one can tell us how,

The routers then become useless if they can do nothing more comlpex than load balancing when you want to host servers behind them/

Even more strange is thatit worked fine until some MT version. i have router that runs three WAN links and I had it workign fine. after some upgrade ov Mikrotik, it stopped. I cannot even ping other interfaces except one which is default gateway.

fewi · Sun Mar 06, 2011 4:41 am

Alright, here an example based on the wiki manual from http://wiki.mikrotik.com/wiki/Manual:PCC

Refer to the full scenario and configuration there, this post only discusses changes to the configuration shown in the manual. All sections that are changed are posted here in their entirety, i.e. the mangle section here are the ONLY rules that should exist in the wiki's scenario extended to what is outlined below.

Now, let's add a web server at 19.168.0.100 that is supposed to be reachable on ports tcp/80 and tcp/443 via both WAN circuits. To give an example for your original requirement of mail being sent out via specific circuits let's also make up a pretend policy that every time anyone on the LAN connects to 1.1.1.1/32 this must happen via wlan1, and every time anyone on the LAN connects to 2.2.2.2/32 this must happen via wlan2.

The port forwarding happens via the usual NAT rules, but you need two rules, one for each WAN interface. You must also make sure to mark the connection at the time it is being established so you can route traffic back out that interface. Also, you need to make sure that you don't overwrite the marks applied at that time later on for return packets when PCC becomes active. This is easily achieved by only applying PCC rules to previously unmarked connections, this is checked by adding a new qualifier: connection-mark=no-mark.

The policy routing based on LAN traffic to specific destination IPs (1.1.1.1 and 2.2.2.2) is simple: add connection marks based on the IPs before the PCC section. Since we've already established that the PCC rules will be adjusted to only apply to previously unmarked packets, PCC wont remark.

Here the changed mangle section in its entirety:

/ip firewall mangle
# standard stuff for router traffic. Doesn't apply to our case since we will later firewall filter traffic that would be covered by this, but included as to be complete
add chain=input in-interface=wlan1 action=mark-connection new-connection-mark=wlan1_conn
add chain=input in-interface=wlan2 action=mark-connection new-connection-mark=wlan2_conn
add chain=output connection-mark=wlan1_conn action=mark-routing new-routing-mark=to_wlan1
add chain=output connection-mark=wlan2_conn action=mark-routing new-routing-mark=to_wlan2
# don't mark traffic going to directly connected WAN networks
add chain=prerouting dst-address=10.111.0.0/24  action=accept in-interface=Local 
add chain=prerouting dst-address=10.112.0.0/24  action=accept in-interface=Local
# mark web server connections established from WAN to LAN coming in wlan1 accordingly. If you have static IPs, you can also refer to them here as dst-address.
add chain=prerouting connection-state=new in-interface=wlan1 protocol=tcp dst-port=80,443 action=mark-connection new-connection-mark=wlan1_conn
# mark web server connections established from WAN to LAN coming in wlan2 accordingly
add chain=prerouting connection-state=new in-interface=wlan2 protocol=tcp dst-port=80,443  action=mark-connection new-connection-mark=wlan2_conn
# force traffic to 1.1.1.1/32 out wlan1
add chain=prerouting dst-address=1.1.1.1/32 in-interface=Local action=mark-connection new-connection-mark=wlan1_conn passthrough=yes
# force traffic to 2.2.2.2/32 out wlan2
add chain=prerouting dst-address=2.2.2.2/32 in-interface=Local action=mark-connection new-connection-mark=wlan2_conn passthrough=yes
# apply PCC, but only to connections that aren't marked yet
add chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=Local per-connection-classifier=both-addresses:2/0 \
    action=mark-connection new-connection-mark=wlan1_conn passthrough=yes
add chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=Local per-connection-classifier=both-addresses:2/1 \
    action=mark-connection new-connection-mark=wlan2_conn passthrough=yes
# mark packets from LAN to WAN with routing marks according to their connection marks
add chain=prerouting connection-mark=wlan1_conn in-interface=Local action=mark-routing new-routing-mark=to_wlan1
add chain=prerouting connection-mark=wlan2_conn in-interface=Local action=mark-routing new-routing-mark=to_wlan2

And here the changed NAT section in its entirety, adding the port forwarding for the web server at 192.168.0.100. Forwarding by interface only - that makes it easy to use this for dynamically addresses interfaces. Replace with IPs if you wish.

/ip firewall nat 
# forward tcp/80 and tcp/443 to 192.168.0.100 on both WAN interfaces
add chain=dstnat in-interface=wlan1 protocol=tcp dst-port=80,443 action=dst-nat to-addresses=192.168.0.100
add chain=dstnat in-interface=wlan2 protocol=tcp dst-port=80,443 action=dst-nat to-addresses=192.168.0.100
# source NAT all traffic out to the Internet
add chain=srcnat out-interface=wlan1 action=masquerade
add chain=srcnat out-interface=wlan2 action=masquerade

And firewall filter rules. This is a very, very simple rule set that assumes that there is no router access from the WAN, that hosts at 192.168.10, .11, and .12 have full administrative access to the router, and that other LAN hosts can talk to the router for DNS and NTP. DHCP can't be firewalled, so that will be a service offered to clients. All traffic initiated from LAN to WAN is permitted, traffic from WAN to LAN can only be initiated to the web server at 192.168.0.100.

/ip firewall address-list
# create address list of hosts allowed to administrate the router
add list=admin_hosts address=192.168.0.10
add list=admin_hosts address=192.168.0.11
add list=admin_hosts address=192.168.0.12
/ip firewall filter
# statefully firewall traffic to the router
add chain=input connection-state=established action=accept
add chain=input connection-state=related action=accept
add chain=input connection-state=invalid action=drop
# allow services for all LAN clients: DNS and NTP
add chain=input in-interface=Local protocol=udp dst-port=53,123 action=accept
add chain=input in-interface=Local protocol=tcp dst-port=53 action=accept
# allow all router access from admin hosts
add chain=input in-interface=Local src-address-list=admin_hosts action=accept
# default deny
add chain=input action=drop
# statefully firewall traffic through the router
add chain=forward connection-state=established action=accept
add chain=forward connection-state=related action=accept
add chain=forward connection-state=invalid action=drop
# allow all traffic from LAN to WAN
add chain=forward in-interface=Local action=accept
# allow tcp/80 and tcp/443 to web server
add chain=forward dst-address=192.168.0.100 protocol=tcp dst-port=80,443
# default deny
add chain=forward action=drop

This was all written together in a text editor and not tested in a lab, so it may contain errors. It should, however, get you started.

Please post any corrections back here, or whether it worked for you. Once it is determined to be correct I'll copy it over to a wiki article since this question comes up a lot. I have no use for PCC so I'm unlikely to try it out myself.

forne · Sun Mar 06, 2011 5:55 am

This is the real config used in my home router RB750G (5.0rc10) connected to two ISP over ethernet + vpn. There are 5 interfaces:

lan         - ethernet connection to local (home) network
local-isp1  - ethernet connection to ISP1, ip address is assigned via dhcp
local-isp2  - ethernet connection to ISP2, ip address is assigned via dhcp
inet1-isp1  - l2tp connection to ISP1 (10/10 Mbps)
inet2-isp2  - pppoe connection to ISP2 (10/10 Mbps)

So there are 1 internal and 4 external interfaces. We expect that there are some static routes obtained from dhcp. We want to send an outgoing packet to the interface local-isp1 or local-isp2 if there is a specific static route for that packet in the main routing table. Otherwise we send the packet to the interface inet1-isp1 or inet2-isp2 using PCC load balancing and failover. Actually we do so called russian pppoe/l2tp. Srcnat is working on all 4 external interfaces. Incoming connections are allowed for some services (torrent client). All outgoing packets belonging to incoming connections are sent to the external interface from which the connection was initiated.

We use queue trees to prioritize all incoming and outgoing traffic via inet1-isp1 and inet2-isp2 interfaces. The only reliable way to distinguish p2p traffic from other traffic is to bind all p2p applications to separate ip address. Note that we limit all queue trees to 9 Mbps despite we have 10 Mbps internet connections. This is required for correct work of incoming traffic prioritization.

/ip address
add address=192.168.0.1/24 disabled=no interface=lan network=192.168.0.0

/ip dhcp-client
add add-default-route=yes default-route-distance=11 disabled=no interface=\
    local-isp1
add add-default-route=yes default-route-distance=12 disabled=no interface=\
    local-isp2

/ip dns
set allow-remote-requests=yes cache-max-ttl=1w cache-size=2048KiB \
    max-udp-packet-size=512 servers=8.8.8.8,8.8.4.4
add address=10.10.10.10 disabled=no name=retracker.local ttl=1d

/ppp profile
add change-tcp-mss=no name=inet1 only-one=default remote-address=\
    127.127.127.101 use-compression=default use-encryption=default use-mpls=\
    default use-vj-compression=yes
add change-tcp-mss=no name=inet2 only-one=default remote-address=\
    127.127.127.102 use-compression=default use-encryption=default use-mpls=\
    default use-vj-compression=yes

/interface l2tp-client
add add-default-route=no allow=chap,mschap1,mschap2 connect-to=10.10.0.70 \
    dial-on-demand=no disabled=no max-mru=1460 max-mtu=1460 mrru=2048 name=\
    inet1-isp1 password=XXXXXXXX profile=inet1 user=XXXXXXXX

/interface pppoe-client
add ac-name="" add-default-route=no allow=chap,mschap1,mschap2 \
    dial-on-demand=no disabled=no interface=local-isp2 max-mru=1492 \
    max-mtu=1492 mrru=2048 name=inet2-isp2 password=XXXXXXXX profile=\
    inet2 service-name="" use-peer-dns=no user=XXXXXXXX

/ip firewall filter
add action=accept chain=input comment="Allow established connections" \
    connection-state=established disabled=no
add action=accept chain=input comment=\
    "Allow all traffic from the local network" disabled=no in-interface=lan
add action=accept chain=input comment="Allow ICMP packets" disabled=no \
    protocol=icmp
add action=drop chain=input comment="Deny all other traffic" disabled=no
add action=accept chain=forward comment=\
    "Allow all outgoing traffic received from the local network" disabled=no \
    in-interface=lan
add action=jump chain=forward comment=\
    "Process incoming traffic going to the local network" disabled=no \
    jump-target=forward-in out-interface=lan
add action=drop chain=forward comment="Deny all other traffic" disabled=no
add action=accept chain=forward-in comment="Allow established connections" \
    connection-state=established disabled=no
add action=accept chain=forward-in comment="Allow related connections" \
    connection-state=related disabled=no
add action=accept chain=forward-in comment=\
    "Allow connections to torrent client" disabled=no dst-address=\
    192.168.0.13 dst-port=12345 protocol=udp
add action=accept chain=forward-in comment=\
    "Allow connections to torrent client" disabled=no dst-address=\
    192.168.0.13 dst-port=12345 protocol=tcp
add action=drop chain=forward-in comment="Deny all other traffic" disabled=no

/ip firewall mangle
add action=jump chain=prerouting comment="Choose the outgoing interface for th\
    e packets received from the local network" disabled=no in-interface=lan \
    jump-target=choose-out-iface
add action=mark-connection chain=prerouting comment="Bind the whole connection\
    \_to the interface inet1 if at least one packet\
    \nbelonging to the connection is received from the interface inet1" \
    disabled=no in-interface=inet1-isp1 new-connection-mark=inet1 \
    passthrough=no
add action=mark-connection chain=prerouting comment="Bind the whole connection\
    \_to the interface inet2 if at least one packet\
    \nbelonging to the connection is received from the interface inet2" \
    disabled=no in-interface=inet2-isp2 new-connection-mark=inet2 \
    passthrough=no
add action=jump chain=output comment="Choose the outgoing interface for the pa\
    ckets originated from the router itself" disabled=no jump-target=\
    choose-out-iface
add action=mark-routing chain=choose-out-iface comment="Send the packets belon\
    ging to the connection bound to the interface inet1\
    \nto the interface inet1" connection-mark=inet1 disabled=no \
    new-routing-mark=inet1 passthrough=no
add action=mark-routing chain=choose-out-iface comment="Send the packets belon\
    ging to the connection bound to the interface inet2\
    \nto the interface inet2" connection-mark=inet2 disabled=no \
    new-routing-mark=inet2 passthrough=no
add action=mark-routing chain=choose-out-iface comment="Send some of the packe\
    ts belonging to unbound connections to the interface inet2\
    \naccording to PCC. All the remaining packets will be sent to the interfac\
    e inet1." disabled=no new-routing-mark=inet2 passthrough=no \
    per-connection-classifier=both-addresses:2/1
add action=jump chain=forward comment="Adjust tcp mss on vpn connections" \
    disabled=no jump-target=tcp-mss-adjust protocol=tcp tcp-flags=syn
add action=jump chain=forward comment=\
    "Mark the packets going to the interface inet1 for QoS" disabled=no \
    jump-target=qos-inet-out out-interface=inet1-isp1
add action=jump chain=forward comment=\
    "Mark the packets received from the interface inet1 for QoS" disabled=no \
    in-interface=inet1-isp1 jump-target=qos-inet1-in
add action=jump chain=forward comment=\
    "Mark the packets going to the interface inet2 for QoS" disabled=no \
    jump-target=qos-inet-out out-interface=inet2-isp2
add action=jump chain=forward comment=\
    "Mark the packets received from the interface inet2 for QoS" disabled=no \
    in-interface=inet2-isp2 jump-target=qos-inet2-in
add action=change-mss chain=tcp-mss-adjust comment=\
    "Adjust tcp mss on the interface inet1" disabled=no new-mss=1420 \
    out-interface=inet1-isp1 protocol=tcp tcp-flags=syn tcp-mss=1421-65535
add action=change-mss chain=tcp-mss-adjust comment=\
    "Adjust tcp mss on the interface inet1" disabled=no in-interface=\
    inet1-isp1 new-mss=1420 protocol=tcp tcp-flags=syn tcp-mss=1421-65535
add action=return chain=tcp-mss-adjust comment=\
    "Return to the chain \"forward\"" disabled=no
add action=mark-packet chain=qos-inet-out comment="Mark p2p traffic" \
    disabled=no new-packet-mark=inet-out-p2p passthrough=no src-address=\
    192.168.0.13
add action=mark-packet chain=qos-inet-out comment="Mark server traffic" \
    disabled=no new-packet-mark=inet-out-server passthrough=no src-address=\
    192.168.0.5
add action=mark-packet chain=qos-inet1-in comment="Mark p2p traffic" \
    disabled=no dst-address=192.168.0.13 new-packet-mark=inet1-in-p2p \
    passthrough=no
add action=mark-packet chain=qos-inet1-in comment="Mark server traffic" \
    disabled=no dst-address=192.168.0.5 new-packet-mark=inet1-in-server \
    passthrough=no
add action=mark-packet chain=qos-inet1-in comment="Mark other traffic" \
    disabled=no new-packet-mark=inet1-in-other passthrough=no
add action=mark-packet chain=qos-inet2-in comment="Mark p2p traffic" \
    disabled=no dst-address=192.168.0.13 new-packet-mark=inet2-in-p2p \
    passthrough=no
add action=mark-packet chain=qos-inet2-in comment="Mark server traffic" \
    disabled=no dst-address=192.168.0.5 new-packet-mark=inet2-in-server \
    passthrough=no
add action=mark-packet chain=qos-inet2-in comment="Mark other traffic" \
    disabled=no new-packet-mark=inet2-in-other passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat comment="NAT on all external interfaces" \
    disabled=no out-interface=!lan
add action=dst-nat chain=dstnat comment="Port mapping for torrent client" \
    disabled=no dst-port=12345 in-interface=!lan protocol=udp to-addresses=\
    192.168.0.13
add action=dst-nat chain=dstnat comment="Port mapping for torrent client" \
    disabled=no dst-port=12345 in-interface=!lan protocol=tcp to-addresses=\
    192.168.0.13

/routing filter
add action=discard chain=dynamic-in comment=\
    "Discard all dynamic default routes" disabled=no invert-match=no prefix=\
    0.0.0.0/0

/ip route rule
add action=lookup disabled=no table=main
add action=lookup-only-in-table disabled=no routing-mark=inet2 table=inet2
add action=lookup-only-in-table disabled=no table=inet1

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=127.127.127.1 \
    routing-mark=inet1 scope=30 target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=127.127.127.2 \
    routing-mark=inet1 scope=30 target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=127.127.127.2 \
    routing-mark=inet2 scope=30 target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=127.127.127.1 \
    routing-mark=inet2 scope=30 target-scope=10
add comment="Remote host to monitor inet1" disabled=no distance=1 \
    dst-address=81.19.66.61/32 gateway=127.127.127.101 scope=10 target-scope=\
    10
add comment="Remote host to monitor inet2" disabled=no distance=1 \
    dst-address=81.19.67.89/32 gateway=127.127.127.102 scope=10 target-scope=\
    10
add comment="Remote host to monitor inet1" disabled=no distance=1 \
    dst-address=213.180.193.1/32 gateway=127.127.127.101 scope=10 \
    target-scope=10
add comment="Remote host to monitor inet2" disabled=no distance=1 \
    dst-address=213.180.199.34/32 gateway=127.127.127.102 scope=10 \
    target-scope=10
add comment="Remote host to monitor inet1" disabled=no distance=1 \
    dst-address=216.239.32.10/32 gateway=127.127.127.101 scope=10 \
    target-scope=10
add comment="Remote host to monitor inet2" disabled=no distance=1 \
    dst-address=216.239.34.10/32 gateway=127.127.127.102 scope=10 \
    target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.1/32 \
    gateway=213.180.193.1 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.1/32 \
    gateway=81.19.66.61 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.1/32 \
    gateway=216.239.32.10 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.2/32 \
    gateway=81.19.67.89 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.2/32 \
    gateway=213.180.199.34 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.2/32 \
    gateway=216.239.34.10 scope=10 target-scope=10

/queue type
add kind=pfifo name=vpn pfifo-limit=500

/queue tree
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet1-out parent=inet1-isp1 priority=8
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet2-out parent=inet2-isp2 priority=8
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet1-in packet-mark=\
    inet1-in-server,inet1-in-p2p,inet1-in-other parent=lan priority=8
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet2-in packet-mark=\
    inet2-in-server,inet2-in-p2p,inet2-in-other parent=lan priority=8
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=9M \
    max-limit=9M name=inet1-out-server packet-mark=inet-out-server parent=\
    inet1-out priority=1 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet1-out-p2p packet-mark=inet-out-p2p parent=inet1-out \
    priority=8 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet1-out-other packet-mark=no-mark parent=inet1-out \
    priority=4 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=9M \
    max-limit=9M name=inet2-out-server packet-mark=inet-out-server parent=\
    inet2-out priority=1 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet2-out-p2p packet-mark=inet-out-p2p parent=inet2-out \
    priority=8 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet2-out-other packet-mark=no-mark parent=inet2-out \
    priority=4 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=9M \
    max-limit=9M name=inet1-in-server packet-mark=inet1-in-server parent=\
    inet1-in priority=1 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet1-in-p2p packet-mark=inet1-in-p2p parent=inet1-in \
    priority=8 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet1-in-other packet-mark=inet1-in-other parent=\
    inet1-in priority=4 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=9M \
    max-limit=9M name=inet2-in-server packet-mark=inet2-in-server parent=\
    inet2-in priority=1 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet2-in-p2p packet-mark=inet2-in-p2p parent=inet2-in \
    priority=8 queue=vpn
add burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 \
    max-limit=9M name=inet2-in-other packet-mark=inet2-in-other parent=\
    inet2-in priority=4 queue=vpn

spookman · Sun Mar 06, 2011 9:56 am

Awesome thanks guys,

I assume then that the routes we create we just add the relivant routing marks ?

spookman · Sun Mar 06, 2011 11:05 am

Question with my setup at teh top of the post,

why does this not work

/ping 196.25.1.1 routing-table=GW1 interface=ether1

but if I omitt the interface it does
/ping 196.25.1.1 routing-table=GW1

mrz · Sun Mar 06, 2011 11:21 am

interface is only for ipv6

spookman · Sun Mar 06, 2011 12:03 pm

Cool then why when I do the same from within winbox and I specify an interface it works....?

Ok here is my problem, I am trying to write a script that will fail over when a link goes down, but as I don't control the xDSL modems (ISP does) I can't use the check-gateway, becuase if the xDSL line goes down the gateway will still respond.

So I wrote this, the problem is it will fail over on the intial check and then fail back the next check even though the line is still down as the pings responds down the Backup GW. then on the third check it fails over gain and so on and so forth...

I am trying to ensure when it runs the check that it is forced down a specific line.

0   name="GW1" owner="admin" policy=ftp,reboot,read,write,policy,test,winbox,password,sniff,sensitive last-started=Mar/05/2011 22:37:45 run-count=25
     source=
       :local i 0; {:do {:set i ($i + 1)} while (($i < 3) && ([/ping 196.25.1.1 interval=2 routing-table=GW1 count=3 interface=ether1]<=1))};
       :if ($i>=3) do={
       :log info "WAN1 Down";
       /ip route disable [find comment=GW1];
       /ip route enable [find comment="Backup1"];
       } else { :log info "WAN1 UP";
       /ip route enable [find comment=GW1];
       /ip route disable [find comment="Backup1"];
       }

IP ROUTE
[code] 0 A S  ;;; GW1
        dst-address=0.0.0.0/0 gateway=10.0.0.1 gateway-status=10.0.0.1 reachable ether1 check-gateway=ping distance=1 scope=30 target-scope=10 routing-mark=GW1
1 X S  ;;; Backup1
        dst-address=0.0.0.0/0 gateway=10.0.1.1 gateway-status=10.0.1.1 reachable ether2 distance=10 scope=30 target-scope=10 routing-mark=GW1
2 A S  ;;; GW2
        dst-address=0.0.0.0/0 gateway=10.0.1.1 gateway-status=10.0.1.1 reachable ether2 check-gateway=ping distance=1 scope=30 target-scope=10 routing-mark=GW2
3 X S  ;;; Backup2
        dst-address=0.0.0.0/0 gateway=10.0.0.1 gateway-status=10.0.0.1 reachable ether1 distance=10 scope=30 target-scope=10 routing-mark=GW2
4 ADC  dst-address=10.0.0.0/24 pref-src=10.0.0.100 gateway=ether1 gateway-status=ether1 reachable distance=0 scope=10
5 ADC  dst-address=10.0.1.0/24 pref-src=10.0.1.100 gateway=ether2 gateway-status=ether2 reachable distance=0 scope=10
6 ADC  dst-address=10.10.0.0/24 pref-src=10.10.0.1 gateway=ether5 gateway-status=ether5 reachable distance=0 scope=10

forne · Sun Mar 06, 2011 12:21 pm

Ok here is my problem, I am trying to write a script that will fail over when a link goes down, but as I don't control the xDSL modems (ISP does) I can't use the check-gateway, becuase if the xDSL line goes down the gateway will still respond.

You didn't try to understand the Mikrotik setup that I posted above. It uses check-gateway to ping three arbitrary remote hosts for every uplink in order to check if gateway is alive. There is no need to write a script.

spookman · Sun Mar 06, 2011 12:39 pm

forne,

I have had a look but I can't see how it disables the failed route

in my situation I have

/ip route
add check-gateway=ping comment=TO_WAN1 disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.0.0.1 routing-mark=to_WAN1 scope=30 target-scope=10
add comment=WAN1_BACKUP disabled=no distance=10 dst-address=0.0.0.0/0 gateway=10.0.1.1 routing-mark=to_WAN1 scope=30 target-scope=10
add check-gateway=ping comment=TO_WAN2 disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.0.1.1 routing-mark=to_WAN2 scope=30 target-scope=10
add comment=WAN2_BACKUP disabled=no distance=10 dst-address=0.0.0.0/0 gateway=10.0.0.1 routing-mark=to_WAN2 scope=30 target-scope=10

and if you say adding this

add comment="Remote host to monitor WAN1" disabled=no distance=1 dst-address=196.25.1.1/32 gateway=10.0.0.100 scope=10 target-scope=10
add comment="Remote host to monitor WAN2" disabled=no distance=1 dst-address=196.25.1.1/32 gateway=10.0.1.100 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=10.0.1.1/32 gateway=196.25.1.1 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=10.0.0.1/32 gateway=196.25.1.1 scope=10 target-scope=10

should fail it over. coz it's not,

could you explain ?

forne · Sun Mar 06, 2011 1:03 pm

forne,
I have had a look but I can't see how it disables the failed route
could you explain ?

The setup is done according to the following article: http://wiki.mikrotik.com/wiki/Advanced_ ... _Scripting
It works in ros 5.0rc10 without any workarounds.

forne · Sun Mar 06, 2011 1:33 pm

In short, use the following routing as is. Just replace 127.127.127.101 and 127.127.127.102 with your real gateways and routing marks "inet1" and "inet2" with your own. Other ip addresses should not be changed.

/ip route rule
add action=lookup disabled=no table=main
add action=lookup-only-in-table disabled=no routing-mark=inet2 table=inet2
add action=lookup-only-in-table disabled=no table=inet1

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=127.127.127.1 \
    routing-mark=inet1 scope=30 target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=127.127.127.2 \
    routing-mark=inet1 scope=30 target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=127.127.127.2 \
    routing-mark=inet2 scope=30 target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=127.127.127.1 \
    routing-mark=inet2 scope=30 target-scope=10
add comment="Remote host to monitor inet1" disabled=no distance=1 \
    dst-address=81.19.66.61/32 gateway=127.127.127.101 scope=10 target-scope=\
    10
add comment="Remote host to monitor inet2" disabled=no distance=1 \
    dst-address=81.19.67.89/32 gateway=127.127.127.102 scope=10 target-scope=\
    10
add comment="Remote host to monitor inet1" disabled=no distance=1 \
    dst-address=213.180.193.1/32 gateway=127.127.127.101 scope=10 \
    target-scope=10
add comment="Remote host to monitor inet2" disabled=no distance=1 \
    dst-address=213.180.199.34/32 gateway=127.127.127.102 scope=10 \
    target-scope=10
add comment="Remote host to monitor inet1" disabled=no distance=1 \
    dst-address=216.239.32.10/32 gateway=127.127.127.101 scope=10 \
    target-scope=10
add comment="Remote host to monitor inet2" disabled=no distance=1 \
    dst-address=216.239.34.10/32 gateway=127.127.127.102 scope=10 \
    target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.1/32 \
    gateway=213.180.193.1 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.1/32 \
    gateway=81.19.66.61 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.1/32 \
    gateway=216.239.32.10 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.2/32 \
    gateway=81.19.67.89 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.2/32 \
    gateway=213.180.199.34 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=127.127.127.2/32 \
    gateway=216.239.34.10 scope=10 target-scope=10

spookman · Tue Mar 08, 2011 10:44 am

Thanks fewi & forne, I have managed to get it working and now understand why it works and works well.

Another Question maybe for fewi or anyone that can answer.
If I have multiple addresses on an interface like this (WAN1)

10.0.1.2
10.0.1.3
10.0.1.4

connection-mark & routing-mark & PCC

add chain=input in-interface=WAN1 action=mark-connection new-connection-mark=WAN1_conn
add chain=output connection-mark=WAN1_conn action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=Local per-connection-classifier=both-addresses:3/0 action=mark-connection new-connection-mark=WAN1_conn passthrough=yes
add chain=prerouting connection-mark=WAN1_conn in-interface=Local action=mark-routing new-routing-mark=to_WAN1

will traffic coming in on say 10.0.1.2 leave on 10.0.1.2 ?
and traffic leaving on WAN1 leave on 10.0.1.2 ?

And if I am forwarding port to internal servers

add chain=prerouting connection-state=new in-interface=WAN1 protocol=tcp dst-address=10.0.1.4 dst-port=21,25,80,110,143,443,8080 action=mark-connection new-connection-mark=WAN1_conn

Will this go back out on 10.0.1.4 ?

Or do I have to define a routing mark for each external IP on WAN1 ?

fewi · Tue Mar 08, 2011 4:00 pm

NAT rules are only evaluated on the first packet of a connection, then the same action applies to all packets in that connection. So the IP addresses within a connection are always the same (an must be, or nodes would throw them away as all packets in a connection must have the same source and destination ports and addresses).

spookman · Wed Mar 09, 2011 7:09 am

Awesome thanks fewi.

BraZuky · Thu Aug 09, 2012 8:59 am

I want to port forwarding like this:

ROUTER 1 ____> LOAD BALANCE ___> HOTSPOT SERVER ___>
ROUTER 2

I have to access the hotspot server and I can't put it to work.
Pay attention that the Dynamic Public Addresses are in the ROUTERS NOT IN THE LOAD BALANCE.

I tried to understand and apply what fewi explained but I couldn't. It seems that my situation is quite different from the example he gave us.

I didn't understand what is (1.1.1.1 and 2.2.2.2) in his example. Maybe two gateways ?

Please someone help me.

DUAL WAN LOAD BALANCING RULES:

/ip address
add address=172.16.0.1/24 disabled=no interface=Local network=172.16.0.0
add address=192.168.22.3/24 disabled=no interface=wlan2 network=192.168.22.0
add address=192.168.11.2/24 disabled=no interface=wlan1 network=192.168.11.0

/ip firewall nat
add action=masquerade chain=srcnat disabled=no out-interface=wlan1
add action=masquerade chain=srcnat disabled=no out-interface=wlan2

/ip firewall mangle
add action=mark-connection chain=input disabled=no in-interface=wlan1 new-connection-mark=wlan1_mark passthrough=yes
add action=mark-connection chain=input disabled=no in-interface=wlan2 new-connection-mark=wlan2_mark passthrough=yes

add action=mark-routing chain=output connection-mark=wlan1_mark disabled=no new-routing-mark=to_wlan1 passthrough=yes
add action=mark-routing chain=output connection-mark=wlan2_mark disabled=no new-routing-mark=to_wlan2 passthrough=yes

add action=accept chain=prerouting disabled=no dst-address=192.168.11.0/24 in-interface=Local
add action=accept chain=prerouting disabled=no dst-address=192.168.22.0/24 in-interface=Local

add action=mark-connection chain=prerouting disabled=no dst-address-type=!local in-interface=Local new-connection-mark=wlan1_mark passthrough=yes per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting disabled=no dst-address-type=!local in-interface=Local new-connection-mark=wlan2_mark passthrough=yes per-connection-classifier=both-addresses-and-ports:2/1

add action=mark-routing chain=prerouting connection-mark=wlan1_mark disabled=no in-interface=Local new-routing-mark=to_wlan1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=wlan2_mark disabled=no in-interface=Local new-routing-mark=to_wlan2 passthrough=yes



/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.1.1.1 routing-mark=to_wlan1 scope=30 target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=10.2.2.2 routing-mark=to_wlan1 scope=30 target-scope=10

add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.2.2.2 routing-mark=to_wlan2 scope=30 target-scope=10
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=10.1.1.1 routing-mark=to_wlan2 scope=30 target-scope=10

add disabled=no distance=1 dst-address=8.8.4.4/32 gateway=192.168.22.2 scope=10 target-scope=10
add disabled=no distance=1 dst-address=8.8.8.8/32 gateway=192.168.11.1 scope=10 target-scope=10

add check-gateway=ping disabled=no distance=1 dst-address=10.1.1.1/32 gateway=8.8.4.4 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=10.1.1.1/32 gateway=72.30.2.43 scope=10 target-scope=10

add check-gateway=ping disabled=no distance=1 dst-address=10.2.2.2/32 gateway=8.8.8.8 scope=10 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=10.2.2.2/32 gateway=199.59.148.82 scope=10 target-scope=10

add disabled=no distance=1 dst-address=72.30.2.43/32 gateway=192.168.11.1 scope=10 target-scope=10
add disabled=no distance=1 dst-address=199.59.148.82/32 gateway=192.168.22.2 scope=10 target-scope=10

Alright, here an example based on the wiki manual from http://wiki.mikrotik.com/wiki/Manual:PCC

Refer to the full scenario and configuration there, this post only discusses changes to the configuration shown in the manual. All sections that are changed are posted here in their entirety, i.e. the mangle section here are the ONLY rules that should exist in the wiki's scenario extended to what is outlined below.

Now, let's add a web server at 19.168.0.100 that is supposed to be reachable on ports tcp/80 and tcp/443 via both WAN circuits. To give an example for your original requirement of mail being sent out via specific circuits let's also make up a pretend policy that every time anyone on the LAN connects to 1.1.1.1/32 this must happen via wlan1, and every time anyone on the LAN connects to 2.2.2.2/32 this must happen via wlan2.

The port forwarding happens via the usual NAT rules, but you need two rules, one for each WAN interface. You must also make sure to mark the connection at the time it is being established so you can route traffic back out that interface. Also, you need to make sure that you don't overwrite the marks applied at that time later on for return packets when PCC becomes active. This is easily achieved by only applying PCC rules to previously unmarked connections, this is checked by adding a new qualifier: connection-mark=no-mark.

The policy routing based on LAN traffic to specific destination IPs (1.1.1.1 and 2.2.2.2) is simple: add connection marks based on the IPs before the PCC section. Since we've already established that the PCC rules will be adjusted to only apply to previously unmarked packets, PCC wont remark.

Here the changed mangle section in its entirety:
Code: Select all
/ip firewall mangle
# standard stuff for router traffic. Doesn't apply to our case since we will later firewall filter traffic that would be covered by this, but included as to be complete
add chain=input in-interface=wlan1 action=mark-connection new-connection-mark=wlan1_conn
add chain=input in-interface=wlan2 action=mark-connection new-connection-mark=wlan2_conn
add chain=output connection-mark=wlan1_conn action=mark-routing new-routing-mark=to_wlan1
add chain=output connection-mark=wlan2_conn action=mark-routing new-routing-mark=to_wlan2
# don't mark traffic going to directly connected WAN networks
add chain=prerouting dst-address=10.111.0.0/24  action=accept in-interface=Local 
add chain=prerouting dst-address=10.112.0.0/24  action=accept in-interface=Local
# mark web server connections established from WAN to LAN coming in wlan1 accordingly. If you have static IPs, you can also refer to them here as dst-address.
add chain=prerouting connection-state=new in-interface=wlan1 protocol=tcp dst-port=80,443 action=mark-connection new-connection-mark=wlan1_conn
# mark web server connections established from WAN to LAN coming in wlan2 accordingly
add chain=prerouting connection-state=new in-interface=wlan2 protocol=tcp dst-port=80,443  action=mark-connection new-connection-mark=wlan2_conn
# force traffic to 1.1.1.1/32 out wlan1
add chain=prerouting dst-address=1.1.1.1/32 in-interface=Local action=mark-connection new-connection-mark=wlan1_conn passthrough=yes
# force traffic to 2.2.2.2/32 out wlan2
add chain=prerouting dst-address=2.2.2.2/32 in-interface=Local action=mark-connection new-connection-mark=wlan2_conn passthrough=yes
# apply PCC, but only to connections that aren't marked yet
add chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=Local per-connection-classifier=both-addresses:2/0 \
    action=mark-connection new-connection-mark=wlan1_conn passthrough=yes
add chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=Local per-connection-classifier=both-addresses:2/1 \
    action=mark-connection new-connection-mark=wlan2_conn passthrough=yes
# mark packets from LAN to WAN with routing marks according to their connection marks
add chain=prerouting connection-mark=wlan1_conn in-interface=Local action=mark-routing new-routing-mark=to_wlan1
add chain=prerouting connection-mark=wlan2_conn in-interface=Local action=mark-routing new-routing-mark=to_wlan2
And here the changed NAT section in its entirety, adding the port forwarding for the web server at 192.168.0.100. Forwarding by interface only - that makes it easy to use this for dynamically addresses interfaces. Replace with IPs if you wish.
Code: Select all
/ip firewall nat 
# forward tcp/80 and tcp/443 to 192.168.0.100 on both WAN interfaces
add chain=dstnat in-interface=wlan1 protocol=tcp dst-port=80,443 action=dst-nat to-addresses=192.168.0.100
add chain=dstnat in-interface=wlan2 protocol=tcp dst-port=80,443 action=dst-nat to-addresses=192.168.0.100
# source NAT all traffic out to the Internet
add chain=srcnat out-interface=wlan1 action=masquerade
add chain=srcnat out-interface=wlan2 action=masquerade
And firewall filter rules. This is a very, very simple rule set that assumes that there is no router access from the WAN, that hosts at 192.168.10, .11, and .12 have full administrative access to the router, and that other LAN hosts can talk to the router for DNS and NTP. DHCP can't be firewalled, so that will be a service offered to clients. All traffic initiated from LAN to WAN is permitted, traffic from WAN to LAN can only be initiated to the web server at 192.168.0.100.
Code: Select all
/ip firewall address-list
# create address list of hosts allowed to administrate the router
add list=admin_hosts address=192.168.0.10
add list=admin_hosts address=192.168.0.11
add list=admin_hosts address=192.168.0.12
/ip firewall filter
# statefully firewall traffic to the router
add chain=input connection-state=established action=accept
add chain=input connection-state=related action=accept
add chain=input connection-state=invalid action=drop
# allow services for all LAN clients: DNS and NTP
add chain=input in-interface=Local protocol=udp dst-port=53,123 action=accept
add chain=input in-interface=Local protocol=tcp dst-port=53 action=accept
# allow all router access from admin hosts
add chain=input in-interface=Local src-address-list=admin_hosts action=accept
# default deny
add chain=input action=drop
# statefully firewall traffic through the router
add chain=forward connection-state=established action=accept
add chain=forward connection-state=related action=accept
add chain=forward connection-state=invalid action=drop
# allow all traffic from LAN to WAN
add chain=forward in-interface=Local action=accept
# allow tcp/80 and tcp/443 to web server
add chain=forward dst-address=192.168.0.100 protocol=tcp dst-port=80,443
# default deny
add chain=forward action=drop
This was all written together in a text editor and not tested in a lab, so it may contain errors. It should, however, get you started.

Please post any corrections back here, or whether it worked for you. Once it is determined to be correct I'll copy it over to a wiki article since this question comes up a lot. I have no use for PCC so I'm unlikely to try it out myself.

Load Balancing + Port Forwarding

Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding

Re: Load Balancing + Port Forwarding