So I've looked at the various tutorials on using routing-mark.and as far as I can see, I've done everything right.

I have a VPN connection set up in the Router which uses IPSec.

The default route for 0.0.0.0/0 to use the VPN gateway is created automatically (add-default-route=yes in the l2tp-client interface).

I would like to push data from specific local IP addresses (e.g. Smart TV) through the "raw" internet connection.

I have created an additional route to 0.0.0.0/0 with the ISP's gateway specified. The Preferred Source is my public IP. This looks like the automatically-generated route for the ISP's gateway, but it has a routing mark: "direct"

There is a Masquerade rule for pppoe-out, through which the ISP's gateway is reached.

I have created the following Mangle rules, where src-address is a test box:
When these are enabled, Google (on HTTPS) loads *eventually* but it takes a long time, and sometimes fails with a QUIC protocol error. Other pages (e.g. whatsmyip.org) don't load at all.

When calling out to whatsmyip on port 80, Wireshark shows me the SYN going out, the SYN, ACK coming back, then the GET request going out, but then silence from the server until it sends FIN, ACK shortly afterwards.

Any ideas? Do I need to set up an HTTP server in AWS to see if the request is getting out?

Many thanks

- Hugh

If there is some traffic in both direction maybe try adjusting tcp-mss, you have L2TP/IPsec site-to-site that goes through PPPoE, if i understood your setting correctly. You have some additional overhead because of this protocols, try adjusting TCP-MSS to about 1352, or even lower if that doesnt work.

Thanks for your reply tr00g33k.

The VPN works perfectly, and fast.

The problem I have is with traffic that's NOT going over the VPN - the routing-mark applies to the ISP's gateway.

Try this for test: and then try delete all other mangling rules and only add this one: And then accordingly adjust your routing table:

Thanks for the suggestion.

No chain in this one, do you think it should be prerouting or output?

I thought I'd solved this, but I haven't. Editing...

With prerouting as the chain for your middle command, I got the exact same behaviour as I was seeing previously. (I already had the lookup rule - sorry, forgot to mention it.) I might change back to marking the *connection* and then marking connections for routing.

I noticed that I was resolving addresses (looking at Wireshark's output.) DNS works and Traceroute were working fine. So I decided to do a bit of don't-fragment pinging to see how much I could get through. That quick test came out at 1421 bytes. Which is weird, because it should be 1452 bytes. I changed the MTU of the pppoe-out interface to 1449 bytes (1421+28) and this reset the pppoe connection. Which dropped the VPN connection and must have 'refreshed' the default route via the ISP, and everything sprang back to life - EXCEPT FOR the VPN connection.

So I decided to switch it all around. I've removed the Max MTU and it's sprung back up to 1480, and I can ping up to 1452 bytes on my standard internet connection.

I've removed the routing mark from the ISP's gateway and added one for the VPN's gateway. Basically switching everything around - default is ISP, I'm now setting a routing-mark to go through the VPN.

I get the same behaviour - when my mangle rule is turned on, everything is slow or doesn't work at all. I can ping up to 1422 bytes through the VPN, and the VPN's Max MTU is 1450 (set automatically) so that checks out. The only weirdness is that L2TP headers are usually 32 bytes, not 30 bytes...

I tried 1448 bytes. I tried 1400 bytes. I tried 512 bytes. None of these MTUs resolve the problem.

It is worth mentioning that if I create a static route via the desired gateway, whichever one it is, I get the desired outcome. But I want to be performing more intelligent routing than that, e.g. http over VPN, https over WAN (since https is already encrypted) or as originally described, some "whitelisted" machines that can go over WAN.

Do you maybe have FastPath enabled? If so, disable and try again.
Maybe you could post your whole config so it is easier to see, it there some little thing that needs to be changed.

Thanks tr00g33k

Turning off "Allow FastPath" does indeed appear to have solved the problem.

Do you know why this would be the case?

Once again, many thanks for your help with this.

Because some packets went the slow path and went through mangle rules as they should and applied correct routing decision. Other packets went through fast-path, and only routing decision was made based on the routing table no mangle rule applied to the packets.

Do you understand the explanation?

Yeah, makes sense - thanks! How do packets get routed to fast-path or not?

I suggest that you read: http://mum.mikrotik.com/presentations/U ... 654925.pdf

And the new way of fast-path diagram is that than and there one of the packets go the slow path so that it checks if others packets are ok to go fast-path, quickly explained.

I never have fast-path enabled in any "serious" production networks, because of many problems when fast-path enabled.

Fast path is useful for some small and cheaper Routerboard products that need to decrease cpu and ram usage, if you ask me.

Thanks for posting this HughPH, and tr00g33k for working to resolve it! I literally spent hours troubleshooting this same issue which snuck up on me after resetting my device to its default configuration which added the "(maybe) fast-track" firewall rules.

The behaviour was not consistent. Sometimes traffic would flow fast, then slow. Dumping packets everything looked normal. Twiddling MTUs did nothing.

On the plus side I've learned a lot more about the capabilities of these devices. I just wish the Wiki was better maintained. There should be a big warning on the policy based routing page about these fast-path rules not being compatible.

So in short, disable fast path firewall rule and everything works.

Just adding my two cents to this oldish thread for posterity:

I too had issue with Policy Based Routing (routing marks) and missing packets. I have setup 2 voip phones on my local connection, with an LT2P VPN tunnel to a remote location where the PBX is. I then initiated a call between the two local endpoints (RTP/voicepath should round-trip to the remote location and back again to the other local phone). When I tested audio, only the first syllable every few seconds was heard. I started the Mikrotik packet sniffer and as soon as I clicked START, RTP flowed perfectly! I heard audio, the pcap inspected in Wireshark afterwards was also perfect. Toggling on and off the Packet Sniffer made the problem come and go.

Reading the details of Fast Path, it was clear that when a Packet Sniff was active, Fastpath was being disabled - solving my problem and clearly pointing to a problem with Fastpath. As soon as I disabled Fast Path globally under IP -> Settings, my problem went away.

Moral of the story: Fastpath and policy based routing is busted. I'm running 6.40.5 which is the latest 'current' release as of today. I'm happy I found your post -- it just confirms the issue. Mikrotik support should look into this.

Rob

You should understand dat Fast Path and Fast Track are not directions to release some handbrake so everything is fast and still functional.
They are checked early in the very long decision path taken when forwarding packets, of which route marking is one of the steps.
When "fast" functionality is active, this means a lot of features are not available. So it is not meant to be a way to speed up everything (e.g. by enabling hardware acceleration), but a way to improve speed for some very common usage of the router by cutting around a lot of processing that often is not used.
When you have advanced requirements, you generally cannot use them, or you at least need to configure very carefully (in the case of FastTrack, where you have some control over the use of the feature on a connection basis).