How can you load-balance between a pppoe gateway and a static gateway on the bridge?

frank333 · Thu Jul 05, 2018 8:37 pm

I would like to make load balacing between the pppoe gateway on the rb3011 and a LTE gateway with static ip that resides on a trunk of the lan connected in wifi . Someone can explain to me if you can do it.

Currently in the second house I take advantage of the two gateways by configuring the routes directly on the laptop; in the first house I can't.

frank333 · Fri Jul 06, 2018 12:42 pm

I did as the manual says, but maybe I am wrong:

#pppoe-out1 on eth1
#ether5 interface connected wifi to house2 (I can ping on the gateway 192.168.8.1)
#bridge all interface rb3011

/ip address
add address=10.0.0.0/24 network=10.0.0.0 broadcast=10.0.0.1 interface=bridge
add address=192.168.8.2/24 network=192.168.8.0 broadcast=192.168.8.255 interface=ether5

/ip firewall mangle
add chain=input in-interface=pppoe-out1 action=mark-connection new-connection-mark=WAN1_conn
add chain=input in-interface=ether5 action=mark-connection new-connection-mark=WAN2_conn

add chain=output connection-mark=WAN1_conn action=mark-routing new-routing-mark=to_WAN1
add chain=output connection-mark=WAN2_conn action=mark-routing new-routing-mark=to_WAN2

add chain=prerouting dst-address=192.168.8.0/24 action=accept in-interface=ether5

add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/0 action=mark-connection new-connection-mark=WAN1_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/1 action=mark-connection new-connection-mark=WAN2_conn passthrough=yes

add chain=prerouting connection-mark=WAN1_conn in-interface=bridge action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn in-interface=bridge action=mark-routing new-routing-mark=to_WAN2

/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.8.1 routing-mark=to_WAN2 check-gateway=ping

/ip firewall nat
add chain=srcnat out-interface=pppoe-out1 action=masquerade
add chain=srcnat out-interface=ether5 action=masquerade

sindy · Fri Jul 06, 2018 2:11 pm

You have provided only configuration from one of the houses plus I don't understand the remark

I take advantage of the two gateways by configuring the routes directly on the laptop

So in general, you have to configure both houses' routers the same way as described here. The critical point is to avoid application of the load distribution rules on connections initiated by packets coming from the other house; these packets must always be sent out via local WAN.

frank333 · Fri Jul 06, 2018 2:19 pm

You have provided only configuration from one of the houses plus I don't understand the remark
I take advantage of the two gateways by configuring the routes directly on the laptop
So in general, you have to configure both houses' routers the same way as described here. The critical point is to avoid application of the load distribution rules on connections initiated by packets coming from the other house; these packets must always be sent out via local WAN.

I do not have routers on the second home is all a single lan , in the second home I added a LTE gateway that provides only a static ip.
In the second house, I can use both gateways with my laptop (win7)because I added the routes and gateways manually.
From the laptop of my first home I can't ping to the gatewayLTE.

sindy · Fri Jul 06, 2018 2:37 pm

Sorry, I've missed that. So actually the only interface of the router in house 1 which hosts the 192.168.8.0/24 subnet is ether5, and when you connect a laptop to that router, the laptop gets its IP address from some other subnet associated to bridge, right? And when the laptop is connected in house1, those routes on it which it uses for load balancing when connected in house2 (I still don't know how that should be working unless it is a linux laptop) are replaced by a default route to Mikrotik's IP address associated to bridge1?

If the above is correct, what exactly does not work?

frank333 · Fri Jul 06, 2018 2:54 pm

Sorry, I've missed that. So actually the only interface of the router in house 1 which hosts the 192.168.8.0/24 subnet is ether5, and when you connect a laptop to that router, the laptop gets its IP address from some other subnet associated to bridge, right? And when the laptop is connected in house1, those routes on it which it uses for load balancing when connected in house2 (I still don't know how that should be working unless it is a linux laptop) are replaced by a default route to Mikrotik's IP address associated to bridge1?

If the above is correct, what exactly does not work?

There are two computers; one laptop in house 1 and another laptop in house 2.

From the laptop in the house 1 the ping test on the gateway LTE 192.168.8.1 does not work even if I assigned static ip.
If I do the ping test directly from the RB3011 on the LTE gateway it works instead.

The problem is that I would like to combine the two gateways to achieve load balancing and failover without using static routes on laptops.

sindy · Fri Jul 06, 2018 3:19 pm

There are two computers; one laptop in house 1 and another laptop in house 2.

OK, I wasn't sure so I wanted to avoid doubt.

From the laptop in the house 1 the ping test on the gateway LTE 192.168.8.1 does not work even if I assigned static ip.
If I do the ping test directly from the RB3011 on the LTE gateway it works instead.

That's not a big surprise as from the perspective of the Mikrotik, packets from the laptop to the LTE gateway are forwarded ones, so the way your mangle rules are currently set, the PCC handling affects these packets too. I haven't found a way how to check how the both-addresses-and-ports hash treats icmp packets - as there is no port in icmp, unless the hash includes the icmp id instead of one of the ports in the calculation, you may end up marking all packets for 192.168.8.1 coming from the address of your laptop for getting routed via pppoe-out1. You can check whether it is the case by swapping the current mapping of the remainder value to connection-mark in the two rules (i.e. by making packets with reminder 0 be connection-marked with WAN2_conn and packets with reminder 1 with WAN1_conn).

If you want to be able to access the LTE gateway and/or other devices in the 192.168.8.0/24 subnet from a laptop connected to the Mikrotik, you have to exclude this subnet from the PCC handling. You can do that by adding /ip route rule add action=lookup-only-in-table table=main dst-address=192.168.8.0/24 or by adding dst-address=!192.168.8.0/24 to the action=mark-routing or action=mark-connection rules' conditions.

The problem is that I would like to combine the two gateways to achieve load balancing and failover without using static routes on laptops.

It's not a problem, it's the goal and it was clear, so I wanted to know what are the actual problems which prevent achieving of this goal

So far it seems that it actually works as-is but you've chosen a method to test it which gave you misleading results because you were actually testing something else.

frank333 · Fri Jul 06, 2018 3:32 pm

thanks sindy ,
now I try to make the changes that you have recommended .
I'll post the results in a while (if it doesn't stop everything...) don't let me down, you're my lifeline

frank333 · Fri Jul 06, 2018 5:44 pm

I tried the changes but can't get them to work at this point I'd be happy with a simple failover. Now my LAN is configured as originally without any PCC rules has a pppoe gateway, and a dhcp server managed by the RB3011 . In the second house there is always the gateway with only the static ip and the access to the whole LAN works normally.

I tried doing so:
/ip route add gateway=10.0.0.1 check-gateway=ping
/ip route add gateway=192.168.8.1 distance=2

but there is something in the firewall that prevents you from reaching the LTE gateway.

sindy · Fri Jul 06, 2018 6:01 pm

In the second house there is always the gateway with only the static ip and the access to the whole LAN works normally.

Where exactly in the second house? On the laptop which lives there? I suppose that if its gateway is the 192.168.8.1, it uses the LTE box, and if the gateway is 192.168.8.2, it uses the PPPoE via the Mikrotik

I tried doing so:
/ip route add gateway=10.0.0.1 check-gateway=ping
/ip route add gateway=192.168.8.1 distance=2

This way, the route with distance=2 is only used if the other one (with default value of distance which is 1) is down; it can happen that the gateway of the higher priority route is pingable but you cannot get anywhere further through it so it is not a good approach. Plus, in your case, 10.0.0.1 is in the LAN subnet, so it cannot be a gateway; a gateway must be either the pppoe-out1 itself as an interface or, better if possible, the remote address indicated by the PPPoE server because in such case you can use the recursive next-hop search to track the link state all the way to the Internet.

but there is something in the firewall that prevents you from reaching the LTE gateway.

This is what happens if you don't post the complete configuration

You've only posted part of the firewall configuration but it wasn't clear from the posting, and in some other similar topics someone has stated he had no firewall at all and I have mixed the two together, concluding that firewall cannot be what prevents the load distribution in your case.

So try again with the complete output of /export hide-sensitive, obfuscating eventual public addresses if they appear there (hide-sensitive only removes passwords and alike)

frank333 · Fri Jul 06, 2018 6:54 pm

In the laptop (win7) of house 2 I configured the gateways statically: 192.168.8.1 (LTE) with metric 1 and the LAN gateway 10.0.0.1(RB3011) with metric 2 to connect to the rest of the LAN.

for failover I try to use the PPPOE gateway IP provided by the isp

I'll attach the router configuration

sindy · Fri Jul 06, 2018 11:34 pm

I couldn't get how you can configure two gateways in different subnets unless the two subnets are accessible via the same interface or there are two interfaces on the laptop, and the config export reveals you've really attached two subnets to the same L2 space

Doing so is not exactly "wrong" but it may complicate not only understanding but also making the firewall rules selective enough (as e.g. packets matching in-interface=bridge can come from any of the subnets).
I've originally built an impression that the 192.168.8.0/24 was attached to a dedicated interface (ether5) and that the devices in house2 had addresses from that network.

But to the topics, when you look at the rules in chain=forward of the /ip firewall filter, the last two explain why it doesn't work:

add action=accept chain=forward comment="accetta pacchetti in uscita sull'interfaccia pppoe -separazione vlan" out-interface=pppoe-out1
add action=drop chain=forward

All the rules before those two accept only packets belonging to already established connections and those coming in/out via ipsec tunnels, so any new connections can only establish via pppoe-out1.

So a rule accepting also packets with out-interface=bridge inserted before the action=drop one should be enough to make it possible to use the LTE as another uplink, so the result would be

/ip firewall filter
...
add action=accept chain=forward comment="accetta pacchetti in uscita sull'interfaccia pppoe -separazione vlan" out-interface=pppoe-out1
add action=accept chain=forward comment="accetta pacchetti che usano LTE come gateway" out-interface=bridge
add action=drop chain=forward

But another problem remains - it may not be possible to tell the LTE box that to deliver packets to 10.0.0.0/24, it has to use 192.168.8.2 as a gateway. So you've used an action=masquerade rule saying that whatever is sent out via ether5 should be src-nat'ed to the IP address associated to ether5 which is in the LAN subnet of the LTE box so no route to 10.0.0.0/24 is necessary on that box. However, no IP address at all is associated to ether5; it is a member interface of a bridge interface named bridge, and that interface has two distinct IP addresses attached to it so the action=masquerade rule could easily choose a wrong one if only modified to use out-interface=bridge.

It is not possible to have an IP configuration attached to a physical interface and at the same make that same interface a member port of a bridge.

So we again have to find a workaround for the two subnets sharing the same L2 space. In /ip firewall nat, replace the

add chain=srcnat out-interface=ether5 action=masquerade

by

add chain=srcnat out-interface=bridge dst-address=!10.0.0.0/24 src-address=10.0.0.0/24 action=src-nat to-addresses=192.168.8.2

frank333 · Sat Jul 07, 2018 12:08 am

sindy you are a great wizard!!!! it works wonderfully! I would never have arrived without your help. thanks again!!

frank333 · Sat Jul 07, 2018 12:24 am

failover also works perfectly, now I try to rewrite the CPP rules but I think and hope not to find problems now the most complex stage is over.

sindy · Sat Jul 07, 2018 12:36 am

Please add in-interface=bridge to the rule action=accept chain=forward comment="accetta pacchetti che usano LTE come gateway" out-interface=bridge to improve security. Your setup is so unusual that I haven't realized immediately that this condition should be there. It's not a big deal that it was missing for a while as there's NAT on the pppoe-out1, otherwise it would have been a serious security hole.

frank333 · Sat Jul 07, 2018 12:55 am

Please add in-interface=bridge to the rule action=accept chain=forward comment="accetta pacchetti che usano LTE come gateway" out-interface=bridge to improve security. Your setup is so unusual that I haven't realized immediately that this condition should be there. It's not a big deal that it was missing for a while as there's NAT on the pppoe-out1, otherwise it would have been a serious security hole.

I did so:

Schermata del 2018-07-07 00.08.04.png

frank333 · Sat Jul 07, 2018 10:18 am

This morning I was convinced I could finally get the load balancing instead as soon as I add the rules below the lan freezes.
I thought that having a gateway on the bridge, it was enough to replace in-interface=ether5 with src-address=192.168.8.1 instead the issue is more complex.

/ip firewall mangle
add chain=input in-interface=pppoe-out1 action=mark-connection new-connection-mark=WAN1_conn
add chain=input src-address=192.168.8.1 action=mark-connection new-connection-mark=WAN2_conn

add chain=output connection-mark=WAN1_conn action=mark-routing new-routing-mark=to_WAN1
add chain=output connection-mark=WAN2_conn action=mark-routing new-routing-mark=to_WAN2

add chain=prerouting dst-address=192.168.8.0/24 action=accept src-address=192.168.8.1

add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/0 action=mark-connection new-connection-mark=WAN1_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/1 action=mark-connection new-connection-mark=WAN2_conn passthrough=yes

add chain=prerouting connection-mark=WAN1_conn in-interface=bridge action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn in-interface=bridge action=mark-routing new-routing-mark=to_WAN2




/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.8.1 routing-mark=to_WAN2 check-gateway=ping


/ip firewall nat
add chain=srcnat dst-address=192.168.8.1 action=masquerade

Also, can you, via DHCP server, force the use of the LTE gateway to a certain host?

sindy · Sat Jul 07, 2018 11:46 am

This morning I was convinced I could finally get the load balancing instead as soon as I add the rules below the lan freezes.

The reason is again the fact that you use the unusual network setup with two subnets on the same interface but your firewall rules are copied from a setup which relies on interface name as an unambiguous information about packet source. So what actually happens is that you assign a routing-mark also to packets coming from the LTE gateway and destined to your LAN devices, which makes them use the default route with that routing-mark because the routing-mark wins over the fact that the destination address is in a locally present subnet.

So using @Sob's approach here because your existing configuration is already complex enough, add the following to your configuration:
/ip route rule add dst-address=10.0.0.0/24 action=lookup-only-in-table table=main
This routing rule will override any routing mark assigned and make sure that packets for devices in local LAN will be routed properly, using the dynamic routes for directly connected subnets which are only added to the default routing table (main).

Also your PCC rules rely on the wrong assumption that whatever comes from bridge is coming from the LAN subnet. Your action=mark-connection rules rewrite already assigned connection-marks with new ones; this doesn't matter too much for packets sent from the LAN subnet as the PCC gives the same results for all packets belonging to the same direction of the same connection, but it is a disaster for packets coming from the LTE where the roles of src and dst address and port are swapped so the PCC matches are different. One way to avoid this is to only assign connection-marks to packets belonging to connections which don't have any yet. To do so, it is enough to add connection-mark=no-mark condition to the action=mark-connection rules.

I thought that having a gateway on the bridge, it was enough to replace in-interface=ether5 with src-address=192.168.8.1 instead the issue is more complex.

You are another victim of a common misconception. The packets which come from the internet via the LTE gateway do not have the LTE gateway's IP address as source one. They keep the actual IP address of the remote sender. So to identify such packets, you have to use a different set of conditions in that rule, such as in-interface=bridge src-address=!10.0.0.0/24.
The IP address of a gateway configured for a route is never used in packet's header. It is only used to determine the MAC address of the gateway device so that the packet with the original destination IP address unchanged could be packed into an L2 frame and sent to that MAC address.

So all in all the following modifications are required:

/ip firewall mangle
add chain=input in-interface=pppoe-out1 connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn
add chain=input src-address=192.168.8.1 in-interface=bridge src-address=!10.0.0.0/24 connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_conn

add chain=output connection-mark=WAN1_conn action=mark-routing new-routing-mark=to_WAN1
add chain=output connection-mark=WAN2_conn action=mark-routing new-routing-mark=to_WAN2

add chain=prerouting dst-address=192.168.8.0/24 action=accept src-address=192.168.8.1

add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/0 connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/1 connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_conn passthrough=yes

add chain=prerouting connection-mark=WAN1_conn in-interface=bridge action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn in-interface=bridge action=mark-routing new-routing-mark=to_WAN2

/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.8.1 routing-mark=to_WAN2 check-gateway=ping

/ip firewall nat
add chain=srcnat dst-address=192.168.8.1 action=masquerade

Also, can you, via DHCP server, force the use of the LTE gateway to a certain host?

You can use DHCP to force a specific configuration to a specific host, but it is not possible to force a gateway from some subnet to a device which doesn't have an IP address from that same subnet. So a device which has only an address from 10.0.0.0/24 cannot be given a gateway from 192.168.8.0/24.

What you probably actually want (do you remember the sticker "I wanted a Lamborghini, but I was unable to pronounce it?") is that for a particular device, the PCC rules would be overridden and connections of that device would always be handled using the to-WAN2 table. So do exactly that - create a static DHCP lease for that device's MAC address, causing it to always get the same IP address (say, 10.0.0.10) while the other settings like default gateway, DNS etc. remain the same like for all the other devices, and insert the following rule just before the PCC rules:

add chain=prerouting dst-address-type=!local in-interface=bridge src-address=10.0.0.10 connection-mark=no-mark action=mark-connection new connection-mark=WAN2_conn passthrough=yes

alli · Sat Jul 07, 2018 1:45 pm

you must srcnat traffic going to gateways, it's easy for gateways with master interface, but for gateways inside the bridge you must srcnat to router ip address inside the bridge (usually 192.168.88.1) by checking the connection mark, so the internal gateway respond to router instead of directly to the client.

frank333 · Sat Jul 07, 2018 2:17 pm

sindy,
as soon as I enter the /ip route rules the lan freezes.

/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.8.1 routing-mark=to_WAN2 check-gateway=ping

Schermata del 2018-07-07 13.06.13.png

alli,

you must srcnat traffic going to gateways, it's easy for gateways with master interface.

I do not understand what you mean by,you can explain better?

sindy · Sat Jul 07, 2018 2:38 pm

If no route with a routing-mark matching the packet's one is available, routing of that packet falls back to the default table "main" (which consists of all the routes without any routing-mark, the routing mark and routing table name are two names for the same thing, except that the name "main" can be used as an alias to "no routing-mark associated").

That's why, when you disable the routes with routing-marks, the firewall continues to assign connection marks and routing marks, but the routing marks are not used for routing so the mistakes in connection marking and routing marking do not spoil the process.

Can you post the complete output of /ip route export verbose? and /ip firewall export in the current state (the two routes with routing-mark disabled but the rest in place)? You can filter out the address-list from blocklist.de. Off topic, sending a reject to spam sources is a waste of network bandwidth, mere drop is better.

frank333 · Sat Jul 07, 2018 3:23 pm

route.rsc

firewall.rsc

sindy · Sat Jul 07, 2018 3:46 pm

I don't get it. I've told you that the purpose of the /ip route rule is to protect packets for LAN devices (in subnet 10.0.0.0/24) from being handled by routing-marked routes, and you have instead set it to handle packets towards 192.168.8.0/24 (which mostly do not exist).

If you modify that /ip route rule to deal with dst-address=10.0.0.0/24 and enable the two routes with routing-marks again, does it work or not?

As for NATing the traffic sent out via the two gateways as @alli recommends, don't worry, the relevant /ip firewall nat rules are already there.

frank333 · Sat Jul 07, 2018 3:54 pm

If you modify that /ip route rule to deal with dst-address=10.0.0.0/24 and enable the two routes with routing-marks again, does it work or not?

no does not work.

sindy · Sat Jul 07, 2018 4:07 pm

So two possibilities, either I have missed something else related to your unusual network setup, or there is some issue like in this topics.

So let's start by disabling the special mangle rule for src-address=10.0.0.103, setting both the PCC rules to assign new-connection-mark=WAN1_conn and permitting the routes with routing-marks. After testing whether it works or not, set both PCC rules to assign new-connection-mark=WAN2_conn and try again. Report both results.

frank333 · Sat Jul 07, 2018 4:29 pm

I tried again to insert 10.0.0.0/24 and the LAN seems to work

Schermata del 2018-07-07 15.25.12.png

then I try the additional instructions you gave me even if I didn't understand exactly how to do it; in /ip mangle firewall I have to disable them all leaving the first two active?

frank333 · Sat Jul 07, 2018 4:48 pm

sorry sindy,
I do not use anything ! I tried the /ip route rule above but it does not work .
Now I try with the pcc rules enabling one connection at a time

sindy · Sat Jul 07, 2018 4:50 pm

The idea is to find out more exactly which type of connections (those through PPPoE or those through LTE) causes trouble, as handling of each of the two types is slightly different.

So I wanted you to keep everything in the target configuration, except that you would systematically mark all connections to use PPPoE in the first step, and to use LTE in the second step. The easiest way to do that is to only set the new-connection-mark in both PCC rules to the same value (first to WAN1_conn, then to WAN2_conn). For the moment we can ignore connections initiated by packets coming from outside, as no such connections are permitted anyway, so the mangle rules in chain=input may remain unchanged.

Another important thing I forgot to mention is that fasttracking is incompatible with packet mangling. As the highest priority (lowest distance) default route in the default routing table uses pppoe-out1 (WAN1) as gateway, connections marked with WAN1_conn can be fasttracked; those using the LTE uplink (marked with WAN2_conn) must not be fasttracked. So first of all, add connection-mark=!WAN2_conn to the action=fasttrack-connection rule in /ip firewall filter. Only if that is not sufficient to make everything work, try the steps above (send everything through WAN1 and then everything through WAN2 using modified PCC rules).

The /ip route rule must stay in place all the time - assuming you try from a device on LAN.

frank333 · Sat Jul 07, 2018 5:36 pm

I hope I have fully understood what needs to be done.
I activated the marked conections one at a time:

wan1 mangle.png

wan1 iproules .png

forWAN2:

wan2 iproules.png

wan2 mangle.png

does not work in both cases

sindy · Sat Jul 07, 2018 6:20 pm

I hope I have fully understood what needs to be done.

Unfortunately you haven't

What I want to see is how it behaves when everything is configured the final way except the rules which contain the per-connection-classifier condition and the exceptional rule for src-address=10.0.0.103.
I want you to set the new-connection-mark in all these three rules to WAN1_conn. So all connections will be sent via pppoe-out1 but all the rest of the policy routing mechanism will be in use.

In the next step, you change the new-connection-mark in those three rules to WAN2_conn. So this time all connections will be sent via the LTE uplink.

I expect it to work in one setup (probably when everything is marked to use pppoe-out1) and not to work in the other one.

frank333 · Sat Jul 07, 2018 7:42 pm

I don't know how to do it; excuse me but I don't have much experience

sindy · Sat Jul 07, 2018 7:46 pm

What is so complex about changing the new-connection-mark value to WAN1_conn in all rules which use that field?

frank333 · Sat Jul 07, 2018 8:29 pm

then replacing the WAN2_conn field in all PCC rules; is it correct to do so?

/ip firewall mangle
add chain=input in-interface=pppoe-out1 connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn
add chain=input in-interface=bridge src-address=!10.0.0.0/24 connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_connWAN1_conn

add chain=output connection-mark=WAN1_conn action=mark-routing new-routing-mark=to_WAN1
add chain=output connection-mark=WAN2_connWAN1_conn action=mark-routing new-routing-mark=to_WAN2

add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/0 connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/1 connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_connWAN1_conn passthrough=yes

add chain=prerouting connection-mark=WAN1_conn in-interface=bridge action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_connWAN1_conn in-interface=bridge action=mark-routing new-routing-mark=to_WAN2

/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.8.1 routing-mark=to_WAN2 check-gateway=ping

sindy · Sat Jul 07, 2018 8:33 pm

Yes! You've got it, this is what I had in mind (well, initially I wanted to keep the rules in chain=input untouched, but never mind as they aren't used anyway until you want to access the system from WAN side).

Now, with WAN1_conn everywhere, does it work or not?

Then, next step, with WAN2_conn everywhere, same question

frank333 · Sat Jul 07, 2018 9:04 pm

Schermata del 2018-07-07 19.53.19.png

the two methods with wan1-conn and wan2-conn do not work

sindy · Sat Jul 07, 2018 9:17 pm

The blue text on the picture... did you always disable the two default routes (dst-address=0.0.0.0/0) without routing-mark when enabling those with routing-mark?

Because until now I was expecting the routing-marks to be assigned where they should not have been; if you have always disabled the default routes in table "main", it could be that the routing marks were not assigned when they should have been.

So now please enable the routes with routing-mark but don't disable those without, and try again with both "everything as WAN1_conn" an then "everything as WAN2_conn".

And tell me how exactly you are testing, by attempting to open a web page from a browser on a laptop which has an address from 10.0.0.0/24 or some other way?

frank333 · Sat Jul 07, 2018 9:37 pm

I kept the default routes active and tried WAN1_conn and WAN2_conn , but it still doesn't work.
I have firefox on a computer with ubuntu and I have disabled web page caching, is connected in on the eth2 interface bridge

sindy · Sat Jul 07, 2018 9:45 pm

That's incredible.

OK. Keep all the mark-connection rules set to WAN1_conn. Do /ip firewall mangle reset-counters-all and /ip firewall filter reset-counters-all Then make one attempt to open a web page. And then post the results of /ip firewall mangle print, /ip firewall mangle print stats, /ip firewall filter print, /ip firewall filter print stats.

frank333 · Sat Jul 07, 2018 10:27 pm

ip firewall mangle print stats.txt

ip firewall mangle print.txt

ip firewall filter print stats.txt

ip firewall filter print.txt

we continue tomorrow or another day if you want, it was not my intention to engage you so much!
However, I can still do it,

sindy · Sat Jul 07, 2018 10:51 pm

Well... did the reset-counters-all actually happen before you've printed the stats? There is so much traffic in the counters that it seems unlikely to me, plus when everything is connection-marked to WAN1_conn, there should be no hits on the "mark-routing to to_WAN2" rule and they are there.

But I start thinking whether it's not the same issue which @Vagelis has encountered, as I wrote earlier, where upgrade to 6.42.5 was necessary.

frank333 · Sat Jul 07, 2018 11:04 pm

I reset the counters , just that there is other traffic on the lan (ipad smartphone TV ) that I can not turn off .
Tonight I upgrade to version 6.42.5 then , let's try again tomorrow if you are not bored.

sindy · Sat Jul 07, 2018 11:12 pm

I'm rather disappointed than bored. So OK, enough for today.

frank333 · Sun Jul 08, 2018 10:47 am

Hello sindy and good morning,
I upgraded to 6.42.5, the automatic update worked without any problems.
But there is a novelty now, I can no longer connect from a computer in the LAN to the LTE gateway (192.168.8.1).

In the night I read back all the posts and checked to have written everything correctly, and tried again with WAN1_conn and WAN2_conn but it still doesn't work.

Schermata del 2018-07-08 09.29.17.png

Schermata del 2018-07-08 09.32.33.png

sindy · Sun Jul 08, 2018 10:48 am

Sorry, I have missed in post #33 that you have replaced WAN2_conn by WAN1_conn also in the match conditions of the action=mark-routing rules; that's not correct because in that case, all packets always end up with routing-mark=to_WAN2, no matter which connection-mark they had, which is not what we want.

So please fix only the action=mark-routing rules back so that they translate connection-mark WANx_conn to new-routing-mark=to-WANx properly (i.e. WAN1_conn to to_WAN1 and WAN2_conn to to_WAN2), and then try again with setting all new-connection-mark values set to WAN1_conn first, then with all new-connection-mark values set to WAN2_conn.

I start feeling that you're not sure what exactly the rules do. The point is that the connection-tracker part of the firewall keeps note on packets in both directions which belong to the same communication flow between a client and a server - a connection. The recognized types of connections are a TCP session (where the connection is directly equivalent to the session), a UDP flow (where swapped source and destination addresses and ports identify packets belogning to the opposite direction), or an ICMP echo request/response flow (where the combination of source and destination addresses along with ICMP ID field discriminates one ICMP connection from another).

Now if you use an action=mark-connection rule to assign a connection-mark to one packet belonging to a connection, the connection tracker remembers that and all subsequent packets identified to belong to that connection, regardless their direction, get the same connection-mark automatically. So in other rules, you can match packet against that connection-mark and take specific action (like assignment of routing-mark which is, unlike the connection-mark, only valid for the actual packet to which it has been assigned).

So by assigning a connection-mark once, to a newly initiated connection, you note down for that connection which WAN its packets should use; to actually force them to a different path that the default routing table would choose for them, you must translate the connection-mark to a routing-mark for every single packet of such connection which you route out. For these connections, you must disable fasttracking because fasttracking skips mangle rules, so the routing-mark would be assigned only to the first packet of that connection but not to the subsequent ones.

frank333 · Sun Jul 08, 2018 11:14 am

/ip firewall mangle
add chain=input in-interface=pppoe-out1 connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn comment="PCC for LTE"
add chain=input in-interface=bridge src-address=!10.0.0.0/24 connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_conn

add chain=output connection-mark=WAN1_conn WAN1 action=mark-routing new-routing-mark=to_WAN1
add chain=output connection-mark=WAN2_conn WAN2 action=mark-routing new-routing-mark=to_WAN2

add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/0 connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn passthrough=yes
add chain=prerouting dst-address-type=!local in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/1 connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_conn passthrough=yes

add chain=prerouting connection-mark=WAN1_conn WAN1 in-interface=bridge action=mark-routing new-routing-mark=to_WAN1
add chain=prerouting connection-mark=WAN2_conn WAN2 in-interface=bridge action=mark-routing new-routing-mark=to_WAN2

/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.8.1 routing-mark=to_WAN2 check-gateway=ping

frank333 · Sun Jul 08, 2018 11:33 am

with the above changes I can navigate, but seems to ignore the LTE gateway, also I can no longer ping on 192.168.8.1

sindy · Sun Jul 08, 2018 11:43 am

The rules you've chosen for modification are the correct ones, but the state before changes is correct, example:

chain=prerouting connection-mark=WAN2_conn in-interface=bridge action=mark-routing new-routing-mark=to_WAN2

What you had before and what I was claiming to be wrong was

chain=prerouting action=mark-routing new-routing-mark=to_WAN1 passthrough=yes connection-mark=WAN1_conn in-interface=bridge
chain=prerouting action=mark-routing new-routing-mark=to_WAN2 passthrough=yes connection-mark=WAN1_conn in-interface=bridge

Plus add both /ip route rule:
add action=lookup-only-in-table table=main dst-address=10.0.0.0/24
add action=lookup-only-in-table table=main dst-address=192.168.8.0/24
The second one should make 192.168.8.1 accessible from LAN again.

frank333 · Sun Jul 08, 2018 12:30 pm

navigate on some sites and not on others, the LTE gateway is no longer visible

ip route rule print.txt

ip firewall mangle print.txt

sindy · Sun Jul 08, 2018 1:27 pm

The mistake is still there:

21 chain=prerouting action=mark-routing new-routing-mark=to_WAN1 passthrough=yes connection-mark=WAN1_conn in-interface=bridge
22 chain=prerouting action=mark-routing new-routing-mark=to_WAN2 passthrough=yes connection-mark=WAN1_conn in-interface=bridge

After you fix it (by replacing the red WAN1_conn by WAN2_conn), reset the counters, try again with a single site which did work now, and then provide not only print but also print stats for both /ip firewall mangle and /ip firewall filter.

And give me also /ip firewall nat print, please, as the inaccessibility of the LTE gateway although the correct /ip route rule is in place bothers me.

frank333 · Sun Jul 08, 2018 1:54 pm

ip firewall filter print.txt

firewall filter print stats.txt

ip firewall mangle print.txt

ip firewall mangle print stats.txt

ip firewall nat print.txt

do not navigate

sindy · Sun Jul 08, 2018 2:15 pm

As for the inaccessibility of the LTE gateway, try to add another /ip firewall nat rule:
/ip firewall nat add chain=srcnat action=src-nat to-addresses=192.168.8.2 src-address=10.0.0.0/24 dst-address=!10.0.0.0/24 out-interface=bridge log=no log-prefix=""
I am actually surprised that the LTE's management was accessible from 10.0.0.0/24 before.

As for the load balancing, the last print shows that you've fixed the rules assigning the routing-marks but did not try with all rules assigning connection-marks set to assign WAN1_conn and then all rules assigning connection-marks set to assign WAN2_conn. Are you restoring configuration backups or why is it so complex to do only the necessary steps and not several changes simultaneously? Once again, to test which of the WANs is handled in a wrong way, everything should stay as in the desired final state, except the rules which contain new-connection-mark item. Only these rules should be set to assign new-connection-mark=WAN1_conn for one test, and to assign WAN2_conn for the other test.

Please reset the counters before each test and provide the print and print stats after each test.

frank333 · Sun Jul 08, 2018 2:28 pm

yeahhhh works works ,wonderfu!

You are a great sindy!

I left the old rule that before allowed me access to the LTE gateway

/ip firewall nat print
Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; defconf: masquerade
      chain=srcnat action=masquerade out-interface-list=WAN ipsec-policy=out,none 

 1    chain=srcnat action=masquerade out-interface=pppoe-out1 

 2    chain=srcnat action=src-nat to-addresses=192.168.8.2 src-address=192.168.8.0/24 dst-address=!10.0.0.0/24 out-interface=bridge log=no log-prefix="" 

 3    chain=srcnat action=src-nat to-addresses=192.168.8.2 src-address=10.0.0.0/24 dst-address=!10.0.0.0/24 out-interface=bridge log=no log-prefix=""

sindy · Sun Jul 08, 2018 3:09 pm

Well, what exactly works

?

I would expect only the access to the LTE gateway to start working after adding the last action=src-nat rule, but that rule should not have changed anything about the main issue which is the PCC deployment.

frank333 · Sun Jul 08, 2018 3:27 pm

access to the LTE gateway is working,

Load balancing works, (at least it seems to me...) known traffic on the interface coming from the LTE gateway and the pppoe gateway.

Using icanhazip.com I sometimes have ip from WISP1 and some from WISP2 .

If I do a speed test I have values higher than those I have always had; I noticed, however, that latency and jitter have increased

Schermata del 2018-07-08 14.12.57.png

the table of routes that I have is this I hope is right

Schermata del 2018-07-08 14.05.07.png

sindy · Sun Jul 08, 2018 3:38 pm

Can you post your current configuration so that I could understand what made it work completely after all? I cannot believe it was only the additional src-nat rule.

As for the speedtest results, I don't know which tool you use, but if it is speedtest.net, that one sets up four TCP sessions to the same remote server for download and displays their summary bandwidth, and then does the same for upload. It depends on the client side ports used and the PCC hash calculation whether those four sessions are spread evenly between your two uplinks or not, so the bandwidth reported may be the sum of both uplink's bandwidths or less. You would have to use packet sniffing to a file on the 'Tik and then Wireshark to see how the PCC rules have actually distributed the sessions between the uplinks. And next time different client side ports would be used so the result would be different.

frank333 · Sun Jul 08, 2018 3:52 pm

the configuration is the same as post #51 with only the last addition in ip firewall nat

sindy · Sun Jul 08, 2018 4:12 pm

Even in that case, only connections sent by the PCC rules via LTE should have been affected, and those sent via pppoe-out1 should have worked normally.

And I've double checked - I've suggested exactly that rule (id=3 in post #53) already in post #12, and you've put it there correctly as the firewall export in post #23 shows, but later it has somehow metamorphed into the one you've posted in post #51 (id=2 in post #53) - either you've changed it when trying to change something else or the upgrade has modified it, no idea.

So you may first disable and then, if everything is OK, remove the rule id=2 from post #53.

frank333 · Sun Jul 08, 2018 4:39 pm

Summing up all the changes at the end I did this (what do you see wrong?)

/ip firewall filter 

add chain=forward action=fasttrack-connection connection-state=established,related connection-mark=!WAN2_conn log=no log-prefix="" comment="defconf: fasttrack"
add chain=forward action=accept in-interface=bridge out-interface=bridge log=no log-prefix="" comment="accetta pacchetti che usano LTE come gateway"
add chain=forward action=drop

/ip firewall nat 
add chain=srcnat action=src-nat to-addresses=192.168.8.2 src-address=10.0.0.0/24 dst-address=!10.0.0.0/24 out-interface=bridge log=no log-prefix="" 
 
 
 

/ip firewall mangle
add chain=input action=mark-connection new-connection-mark=WAN1_conn connection-mark=no-mark in-interface=pppoe-out1 comment="PCC for LTE"

add chain=input action=mark-connection new-connection-mark=WAN2_conn src-address=!10.0.0.0/24 connection-mark=no-mark in-interface=bridge 

add chain=output action=mark-routing new-routing-mark=to_WAN1 connection-mark=WAN1_conn 

add chain=output action=mark-routing new-routing-mark=to_WAN2 connection-mark=WAN2_conn 

add chain=prerouting action=mark-connection new-connection-mark=WAN1_conn passthrough=yes dst-address-type=!local connection-mark=no-mark in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/0 

add chain=prerouting action=mark-connection new-connection-mark=WAN2_conn passthrough=yes dst-address-type=!local connection-mark=no-mark in-interface=bridge per-connection-classifier=both-addresses-and-ports:2/1 

add chain=prerouting action=mark-routing new-routing-mark=to_WAN1 passthrough=yes connection-mark=WAN1_conn in-interface=bridge 
add chain=prerouting action=mark-routing new-routing-mark=to_WAN2 passthrough=yes connection-mark=WAN2_conn in-interface=bridge log=no log-prefix="" 

/ip route
add dst-address=0.0.0.0/0 gateway=pppoe-out1 routing-mark=to_WAN1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=192.168.8.1 routing-mark=to_WAN2 check-gateway=ping

/ip route rule:
add action=lookup-only-in-table table=main dst-address=10.0.0.0/24
add action=lookup-only-in-table table=main dst-address=192.168.8.0/24

sindy · Sun Jul 08, 2018 4:45 pm

The first src-nat rule in the list above (the one with src-address=192.168.8.0/24) seems unnecessary to me - it should not break anything, it is just unnecessary.

Otherwise everything seems fine to me.

frank333 · Sun Jul 08, 2018 5:18 pm

The first src-nat rule in the list above (the one with src-address=192.168.8.0/24) seems unnecessary to me

I eliminated src-address=192.168.8.0/24 it at least remains as a summary , for those in the future looking for help .

One last thing, turning off the PCC, how could I do to automatically attribute an ip of subnet 192.168.8.0/24 to the hosts of the house2 keeping only the local connections to subnet 10.0.0.0/24 (x printers, nas, server). I have to keep the static routes on the computer?

sindy · Sun Jul 08, 2018 5:47 pm

A DHCP server attached to interface bridge has no possibility to distinguish DHCPDISCOVER requests coming from devices in House 1 from those coming from devices in House 2. You can tell it to use a pool within 10.0.0.0/24 or a pool within 192.168.8.0/24 (or even both if you chain them), and you can define static leases for some MAC addresses, but there is unfortunately no way to treat specially DHCPDISCOVER requests coming in via a particular member interface of a bridge.

So you might e.g. move all devices playing the role of "servers" (or "infrastructure") to 192.168.8.0 and assign them static addresses, and use 10.0.0.0/24 for addresses assigned dynamically to "clients", but there is no way with the current setup to distinguish between "clients" connecting from House 1 and "clients" connecting from House 2.

On the other hand, if you don't mind everything in House 2 to be in 192.168.8.0/24, you may exclude the etherX interface to which the link to House 2 is connected from the bridge and move the 192.168.8.2/24 configuration to that interface. This would require to redo a lot of what we've just reached but it would allow you to clearly discriminate between the two LANs.

frank333 · Sun Jul 08, 2018 6:13 pm

sindy, still immense gratitude, for all you have done for me!tnx!