Hi everyone.

I experimented ECMP yesterday, and I think I did hit a bug in RouterOS.

The setup is really simple. I'm talking IPV4 here.

I got 2 ISP , so I added one default 0.0.0.0 route with two gateways. src-NAT at the end, and we are done. Manual add as static routes, no routing protocol involved here.
Pretty simple, RouterOS balances outgoing connections throught those both gateways , I could experiment and demonstrate that easilly.

The problem comes from DNS.
If my RouterOS is my resolver, which it is , it seems like the internel resolver is not able to contact any DNS server.

If I get rid of my RouterOS DNS resolver on my client machine, by directly indicating them an Internet public resolver (like the famous Google one), all is right.
However, if I tell my client machines to use the RouterOS as DNS resolver : nothing resolves at all.

If, I create a special route for the DNS output , to only one of my gateways (bypassing my ECMP), then RouterOS internal DNS resolver is able to resolve.

It seems like RouterOS internal DNS resolver, if used (I do) , cannot resolve anything if the output default route is ECMP based.

Thoughts ?

What DNS servers did you enter in RouterOS?
This kind of config will only work when you put Google or OpenDNS servers and similar there, not
when you have automatic DNS servers as passed by the provider via DHCP or when manually entering provider DNS
server addresses there.
This is because providers normally open their DNS resolvers only for their own clients, and you will be sending
requests for one provider's server to the other.

No provider-locked DNS.
Only public ones as IPs , like 8.8.8.8 etc...

I got 2 ISP , so I added one default 0.0.0.0 route with two gateways. src-NAT at the end, and we are done. Manual add as static routes, no routing protocol involved here.
Pretty simple, RouterOS balances outgoing connections throught those both gateways , I could experiment and demonstrate that easilly.

I'm kinda surprised it works this way without some mangle "magic" and/or routing rules. I guess it might be fasttrack that makes it work this way; if that is the case, it will also explain why it only works for traffic going through the router, and does not work for traffic originated from or destined to the router itself.

Traffic for input , needs mangle tracking , so that the answer takes the same route as the source packet. I agree , but this is not my actual problem.

Why would it need some rules for output traffic ?

Output traffic should just get generated, then be routed like any other traffic (forward traffic f.e) to the internet, while the answers would anyway match the right IP used to get out.
Am I wrong or missing a trick here ?

Actually, I did not mangle, mark-routing, or anything for the ECMP to work. It works like a charm simply with two gateways as default route.
Also, I got some VPN clients, issued from my RouterOS , and those VPN traffic (output chain though) DOES work.

https://wiki.mikrotik.com/wiki/ECMP_loa ... masquerade explains.

Yes it should work, but I am concerned that the source address may not be what you want.
Did you enter a preferred source address (equal to the address you got from the provider) into each static route for 0.0.0.0/0 ?

No I did not.
But the prefer source is filled-in for connected routes.

Aka :

flags dst addr gateway pref.source
AS 0.0.0.0/0 1.2.3.4, 4.5.6.7
DAC 1.2.3.1/24 eth10 1.2.3.4
DAC 4.5.6.1/24 eth9 4.5.6.7

Ok, no idea what is going wrong then. You need to use the sniffer on the internet interfaces.

Output traffic should just get generated, then be routed like any other traffic (forward traffic f.e) to the internet, while the answers would anyway match the right IP used to get out.

Nope... ECMP or not, output chain will be used when router answers a petition towards itself (input): answer source (WAN) used should be the same it entered by; i.e. if petition came by WAN2, answer should go via WAN2.

with ECMP answer could get routed by any of the two WANs (50/50%)

See (Tomas Kirnak's) https://mum.mikrotik.com/presentations/US12/tomas.pdf

mmmm , I still dont get it.

In my case, RouterOS doesn't answer anyone. It doesn't answer an incoming request coming from WANs.
It issues itself a DNS request to a DNS forwarder, using ECMP WANs as output

So for this, it will output a packet to some DNS forwarder. This packet will be ECMP'ed , it will leave by either WAN1 or WAN2 , and the answering packet will hit it back using the same IP it just used as ECMP decision to get out.

So .... ?

I even can ping those DNS resolvers, from RouterOS. But I cant use RouterOS' DNS forwarder, it doesn forward DNS to forwarders (or something blocks in between).

Will read your slides in a moment. Thank you.

That document is not mine, but Tomas Kirnak's.

So for this, it will output a packet to some DNS forwarder. This packet will be ECMP'ed , it will leave by either WAN1 or WAN2 , and the answering packet will hit it back using the same IP it just used as ECMP decision to get out.

You're right on this, I was just pointing out that there are scenarios (if you were to provide any service from the mikrotik router to the Internet, or dst-nat internal servers) where you need to keep track even on the output chain.

Have you torched on the WAN interfaces to see what happens?

You're right on this, I was just pointing out that there are scenarios (if you were to provide any service from the mikrotik router to the Internet, or dst-nat internal servers) where you need to keep track even on the output chain.

Yes I know, I already did that for incoming traffic , to let the answering packets flow back to the right WAN with con-mark and route-mark. That is all right to me, and works.

So for DNS, I torched interfaces, and then I could see interesting things.
What I can see is explicit.

ROS sends (tx) frames for DNS , those are, as expected, ECMPed , one goes through WAN1 , other to WAN2.
However, in the torch, I cannot see any response traffic (rx) from those frames , whereas if I create specific routes for each DNS resolver to contact (and thus bind each of them to a specific WAN, 1 or 2) , then all right : I can see the tx as well as the rx.

I cannot get why the rx don't come back when ECMPed. I suspect a bug somewhere in the stack , when ECMP is activated ; as if I short-circuit it (by forcing DNS output to go through either WAN1 or WAN2) then all is right not only for DNS traffic (from ROS to forwarders), but also for all my internal LANs than get ECMP'ed just the right way.

To force traffic, I do it with specific static routes for each resolver I use.
So DNS_IP_1 goes through WAN1 , DNS_IP_2 routes through WAN2 , etc... That situation works. But its not what I want, I'd like DNS to be ECMPed like any other traffic.

Remember that if I dont use the ROS DNS, aka if my machines I tell them to use a public DNS directly , then they do , such traffic get ECMPed.
Only when the DNS traffic is issued from ROS, with ECMP activated, then I dont see the replies...

See ?

Do you have suitable rules in the INPUT table?
Can you post the config export?

My INPUT is correct because I can make it work by forcing DNS to bypass ECMP. Then all is right in such a case.

I have an Accept Established-related on INPUT.

ECMP is working fine if you have global addresses to route. In case of NAT, better make sure that it explicitly NATs the packets using SRC-NAT depending on what interface they are leaving.

Packets are already masqueraded when they leave using a private src-addr , so they get translated

Another torch here that seems to show a problem.

Requests from ROS (output) select an interface to get out, but don't select the right source IP.

Aka, I can see DNS requests from ROS leaving from WAN1 , but having as source IP the one of WAN2 , and opposite.

Pretty strange.

That is what I tried to address in comment #6

The prefer-source is only set on connected routes, not on any static one.

Then try setting it in your default routes...