Here is a post that outlines something I think caused by this bug:
http://forum.mikrotik.com//viewtopic.ph ... highlight=

I had a different situation but I think related. I run two physical interfaces to the cable provider on the 4 port nic. The MAC addresses for each interface are obviously different. To cut to the chase - Cox Cable is seeing both MAC addresses on their router associated with the wrong interfaces. MT is broadcasting the WRONG MAC for the second interface. Cox then sends me all traffic back to a single interface because MT advertised both ranges on the same cable modem. Since it cable they don't care and will send it back - I assume if you had 2 different providers - like the other posters problem - you'd get broken sessions. I was having problems with firewall rules blocking all traffic because it came in the wrong interface.

Flags: X - disabled, R - running
# NAME MTU MAC-ADDRESS ARP
0 R onboard-inside 1500 00:40:CA:1D:C8:7C enabled
1 R 1-coxBiz 1500 00:20:FC:1E:D1:C0 enabled
2 R 2-sony 1500 00:20:FC:1E:D1:C1 enabled
3 R 3-coxRes 1500 00:20:FC:1E:D1:C2 enabled
4 R 4-hotty 1500 00:20:FC:1E:D1:C3 enabled

7 address=68.15.19.50/27 network=68.15.19.32 broadcast=68.15.19.63
interface=1-coxBiz actual-interface=1-coxBiz

9 D address=68.8.25.137/23 network=68.8.24.0 broadcast=68.8.25.255
interface=3-coxRes actual-interface=3-coxRes

coxBiz and CoxRes are cable modem connections. Each have their own interface directly connected. D1:C0 and D1:C2 One interface is static IP the other is DHCP assigned (business & residential cable modem). The multihoming this way has worked perfectly on 2.8 for a year now. (What's with the 'actual-interface' parameter?)

The snapshot here shows that it's sending MAC addresses out the wrong ports. IP ranges are on totally different physical interfaces and should NEVER cross like this. MT is telling one upstream connection its the IP address of the other connection.

This snapshot of Ethereal shows it perfectly.


Please have a look at this and fix whatever is wrong, please : ) I will not be able to move our production routers onto 2.9 until this happens. I know it's beta and I hope to see this fixed before release.

Supout sent to support already.

Thx,
Sam

PS - If you look at the selected line in the ethereal snapshot you will see the routers timestamp is not being included in the packet sniff either. The routers time is showing okay, it's just only sometimes being written in the pcap with anchient date/times.

Sam

Sam,

Your post (and the related multiple gateway/NAT problems) seams to be what I WILL run into if I don't find a workaround.
First, I have to say I'm in the wrong forum (I use v2.8.26).
I have 2 separate enviroments:
One with Multiple DG, wher both require NAT, and the other we one gateway need NAT and the other is straight thru.

I'm working on this (for case 1):
add src-address=0.0.0/0 out-interface=ethnet1 action=nat to-src-address=10.11.1.1
add src-address=0.0.0/0 out-interface=ethnet2 action=nat to-src-address=10.22.1.1

and disable the second filter for my case 2.

Thoughts? ideas? (I'm not done with testing yet)
thx/RS

Helo all.
Since nobody from MT can agree on this forum this bug or say, when the bug will be fixed, I put info, I get from MT:

> Hello,
>
> It works that way. If your data is sent out through one gateway, and
> then again through another, your chat system doesn't understand how it
> can be that information from one person comes from different places
> and it doesn't know where to send the reply. so it stops. you should
> use policy routing, you will have more control there.
>
> ECMP can cause trouble with chat programs, online games and similar
> applications.
>
> Sincerely,
> Normunds

hmm... I understand this behaviour, however i thought the ECMP cares the session-orientation (keeps the session on the same gateway). if not - where is the functionality of this feature? well, i asked again and I get:

Hello,

there are two solutions, either you use policy routing, or avoid using masquerading. when you use masquerading, the ECMP can have problems like you describe

Sincerely,
Normunds

I'm not routing guru and I really don't understand, when can I use ECMP, if I cannot use masquarading? My whole network - 300 hosts - is masquaraded and I have no idea, what should I do, to use ECMP without Masquarading?
Policy routing works well - but this is another game - and this is not final resolution for me - it takes too much care and it is not so accurate, as ECMP (ECMP really fine balance upload - and this is primary goal while balancing ADSLs). I'm still waiting for working ECMP - I think they will not take care about it now - until really stable and official 2.9 release appear.
From another hand - I heard, that similar ECMP bug is in Linux distributions too... Is it the same module in MT, without modifications?

regards
kj

ECMP gets sticky just because you are dealing with two gateways and all packets have the possibility of using either gateway. Its not necessarily the local network that causes the problem, its the IM gateways - they do not like to see you switching ips every few minutes.

In policy routing i am telling EXACTLY which packets to go where, and it's not obeying the rules. As far as Im concerned there is still a small bug in the rc6 code that causes ARPs to be broadcast on the wrong interface. I sent support the above a few days ago and have heard nothing... i'm sure they are thinking it is a configuration issue or they know its a bug, I just wish they would communicate to tell us.

I am wondering if the cable providers head unit (cisco 7200) is confusing MT - since both gateways (on 2 different subnets) are showing the same MAC on their end, maybe MT is seeing that and ignoring the subnet and just using the first found gateway that can reach it. This is a bug and we need to get it resolved before we can move things over. I am glad to provide more testing and whatever else MT needs to fix this - just communicate what you need.

Sam

Sam,

look at my post above. Meanwhile I've tested it (both cases) and it works.

Try the src-nat (NAT not masquerade)
add src-address=0.0.0/0 out-interface=ethnet1 action=nat to-src-address=10.11.1.1
add src-address=0.0.0/0 out-interface=ethnet2 action=nat to-src-address=10.22.1.1

PS: if you set disable-running-check=no on the ethernet, it even provides some failover (for interface down, not enroute-down).
I'm running this on a wireless reater. it uses its own SDSL backbone(eth/NAT) and if that is down, it routes without nat in a WDS to another AP.

Cannot use src-nat because one of the cable modems has a dhcp address, therefore there is no way to nat it statically. MASQ should work fine if it's routing based on the interface it's leaving, it's always worked this way in 2.8 fine. This is what masq is for.

Try to route certain traffic out one interface and other traffic out the other interface. Mangling shows the route marks getting added however they are not taking their correct route when it comes time to.

The screenshot above shows it perfectly... there is no reason MT should be advertising the wrong IP / ARP combination. Plain and simple.

Sam

if one gateway is a dhcp-modem, then ok, you cannot replace masq by src-nat. At least for my situation (both gateways a static/public IP), I found that these 2x src-nat with out-interface set, replacing masq is a work-round for the IP/ARP mixup (which is clearly a bug, but ignored? by MT for a long time now).
RS

Let me setup NAT instead of MASQ for 10 minutes and see if it still fails. My DHCP address is usually valid at least for a day or so. On top of all this there is still a policy routing problem.

Sam

ECMP gets sticky just because you are dealing with two gateways and all packets have the possibility of using either gateway. Its not necessarily the local network that causes the problem, its the IM gateways - they do not like to see you switching ips every few minutes.

Switching IPs means dropping connection, it is that simple, as any connection works only from one IP to only another one IP, not from two IPs to one IP (well, except some cases of multicast and broadcast, perhaps). BUT ECMP does not (or should not...) switch IPs randomly, or it would not work at all! ECMP decides the path based on IP pair (src-dst) and keeps it this association for undefined period of time (i suppose it is not an association, but rather a hashing function). If something does not work, that mean there is another connection involved from different IP which can not find its way back to the origination host, or something like that. I am not an expert of any kind on game, IM and P2P protocols, so I do not know why it is not working, but one thing for sure - it is a fundamental flaw in these protocols that does not let them work, not a bug in RouterOS - if ECMP works with TCP and UDP, then it works as for routing decision there is no difference between IP protocols.