Hi, I got the setup.

192.168.150.1/24 <- main router
192.168.150.251/24 <- secondary router (secondary gateway)

all devices in 150.0/24 got 150.1 as a gateway, at this gateway is a mangle rule to route some traffic specific traffic (for example all traffic from 192.168.150.60) out through secondary gateway in out L2TP VPN).
ping and ICMP work fine but TCP and udp not. I think the problem is in nat:

If packet for example goes from 192.168.150.60 to 1.1.1.1 it goes like this(s: src address, d: destination address)
(PC) s:150.60 d: 1.1.1.1 -> (MAIN ROUTER) s:150.60 d: 1.1.1.1 -> (SECONDARY ROUTER) s:150.60 d: 1.1.1.1 ->(VPN) s:150.251 d:1.1.1.1 <- so far so good
but if packet is coming back it goes like this:
(VPN) s:1.1.1.1 d:150.251 -> (SECONDARY ROUTER) s:1.1.1.1 d:150.60 -> it is the same subnet so the secondary router sends it straight to PC (PC)s:1.1.1.1 d:150.60 <- wrong because the PC drops that packet as invalid because it does not wait for response from the secondary router but from the main router.

But how solve this problem? I try set src-nat at main router but without success. For some reason, if packet goes through srcnat at a main router at the moment when it came to secondary router the src address is still 192.168.150.60 (PC)

Any advice? thanks

Did you actually confirm that PC drops return packet due to wrong gateway passing return packet?

I don't think that's the problem, gateway's IP address is not recorded anywhere in the packet, gateway's IP address is only used to get next hop's MAC address. I'm not aware of any IP stack implementation that would trip on MAC address.

What I think actually happens is this:

PC sends packet to main router. Main router's connection tracking machinery makes note of a new connection. Main router passes packet to secondary router. Secondary router's connection tracking machinery makes note of a new connection. Secondary router passes packet through its upstream connection. .... Return packet arrives at secondary router. Secondary router's tracking machinery updates connection state from new to established. Secondary router passes packet to PC[*]. PC sends another packet to main router. Main router's connection tracking machinery sees packet which doesn't correspond to connection state (another forward packet with seemingly wrong sequence number and flags), marks that packet as invalid and drops it.
You could verify exact behaviour by doing some wireshark recording of traffic on PC itself, possibly on both routers as well.

[*] Return packet thus bypasses main router and connection tracking machinery on primary router can not update its connection state to established.

Generally there are two solutions, which one is better depends on other considerations that might exist:

1. Move secondary router to different subnet (via dedicated physical connection or VLAN) which will force all traffic passing secondary router to pass primary one as well. No other configuration is needed.
   The drawback is that secondary path will be only available through primary router which becomes single point of failure.
2. keep network layout as is, but on primary router disable connection tracking for connections via secondary router. Which also means you'll have to (largely) duplicate firewall rules on secondary router. Firewall filter rules on primary router will need a rule to accept untracked connections (current default config for SOHO models has it already).

<removed>

rp_filter has nothing to do with multiple gateway hosts, it has to do with multiple network interfaces on a PC (or rather on a router). Which is not the case here, PC communicates with both gateways through single interface.

A pretty good description of what rp_filter on linux host does can be found here.

rp_filter has nothing to do with multiple gateway hosts, it has to do with multiple network interfaces on a PC (or rather on a router). Which is not the case here, PC communicates with both gateways through single interface.

A pretty good description of what rp_filter on linux host does can be found here.

You're right, sorry, I overlooked, also the fact that icmp/ping works should already indicate something.
I've this removed my earlier post.

Hi guys, thanks for the help. I decided to move the secondary router to a separate VLAN. Now it works but I got another question, how mark the packet on the main router to preserve as much resource as possible? Because if I run the speediest over the VPN it killing the main router (100% CPU usage) my mangle rule looks like that:

3 chain=prerouting action=mark-routing new-routing-mark=to_vor
passthrough=yes src-address=192.168.150.60 dst-address-type=!local
log=no log-prefix=""

I try FastTrack in the firewall, no change, also I try to create a mangle to mark-connection where the connection state is new. but it does not work today(unable ping out etc).

I got just old RB2011 as the main router :(

Thanks