Hi

i am on current long term 6.48.6 with this switch and i have a weird ARP issue in combination with VLANs.
VLANs are configured on the bridge, all tagged, only L2 (switching), no interfaces apart from management, no routing.

On one of the switchports, i have directly(!) connected a Fortigate 60D firewall to transport some DMZ VLANs from the Firewall to the switch (all tagged). All VLANs are configured exactly the same on the Fortigate and on the MT switch.

Now for most of the VLANs I never see ARP packages on the switch with the MT packet sniffer. Only for always the same two of five VLANs I ever see ARP traffic on the switch, coming from the firewall. Firewall asks who has 192.168.30.20 via broacast bit gets no response. I can see the ARP request on the Fortigate via integrated packet capture going out the correct interface but I can't see ANYTHING on the MT packet sniffer going in (except ARP traffic for the other two VLANs mentioned earlier, which are on the same physical interfaces and always work).

Now my questions.
- Can I 100% trust the MT packet sniffer? If so, that would mean that the packets in question never really leave the Fortigate.
- If I can't trust it 100% - what can I try next? I have basically a Fortigate with a trace that says the packets are leaving the firewall and a MT with a trace which says there are no incoming ARP packets for that VLAN nor IP address.
between them is only a short cable and two of five VLANs work.
Also, if I add a static ARP entry to thr Fortigate, traffic also starts flowing so L2 is actually OK. If I start traffic from the other direction (server within VLAN goes to switch, goes to firewall, instead of firewall goes to switch and wants to reach server), ARP works as well, absolutely flawlessly.

WTF? anyone got an idea?
Thanks!

MT packet sniffer can only see traffic which is not offloaded to hardware. Which is almost none traffic if CRS326 is configured properly, all traffic will be handled by switch chip.

You might see broadcasts on MT sniffer if CRS has VLAN interface created for that VLAN (could be that it also needs IP address defined).

Or you can disable HW offload for port towards fortigate, but that will severely hamper performance (all traffic via that port will be handled by a weak CPU).

I don't know how things work on CRS with HW offloading active, but in principle you should see MAC addresses of ethernet devices either in /interface/ethernet/switch/host/print or in /interface/bridge/host/print ... these are tables which switch is using for frame switching. ARP who has procedure can help to populate these tables as well.

Hi

thanks for your response!

Out of curiosity I now disabled the following things:
- bridge fast-forward
- allow fast path
- hw offload on the affected interfaces (server uplink and firewall uplink where the ARP traffic should come from/go to)
- limit broadcasts on the affected interfaces

Now the issue has gone away. When the ARP entry has expired from the Firewall and I try to access the server from the Internet, the ARP entry comes right back. The firewall asks as usual "who has <ip>" and gets a response from the server via the VLAN without any problem. I can as well see those ARP packets in the MT sniffer trace now (opened with Wireshark), correctly tagged with the correct VLAN...

What I don't know yet is, which of the changed settings above "repaired" it.
BTW in /interfaces/ethernet/switch/host/print I don't see any of the MAC addresses nor IPs within that affected VLAN. Not even during traffic flow through that VLAN.
Also, /ip/arp/print only shows three entries but I guess that's just the ARP entries for the management interface...

edit: Now I can also see the ARP responses in the trace on the Firewall, which weren't there before. I guess I'm now resetting all the changed settings on the MT switch to their original values, retry/verify that the issue has come back and then try the settings one by one until the issue has vanished again. If nobody here comes up with a reason for this, I'll raise a support ticket...

Thanks so far, ideas and suggestions welcome!

Update:
now things are getting spooky. i have set all settings back to their original state, when things weren't working as expected but now everything still works. ARP messages are getting delivered properly and now even in /interface/ethernet/switch/host/print the ARP entries in question are shown (they weren't there at first, not even when traffic was flowing!).

Both switch and firewall haven been rebooted yesterday during troubleshooting, but that didn't change a thing. And now, suddenly ARP is working. Why? How? When will it stop working again?

edit again: even if I manually remove the relevant ARP entries from the firewall and the switch, it still works. ARP immediately rediscovers everything if necessary without any problem. I really have no explanation for this at all.

Can this possibly be a RouterOS 6.48.6 bug in the CRS switch? I'm relatively new to MikroTiks and this is why I'm going easy on pointing with the finger on it but it is just that I have NEVER had issues like that with Fortigates until now...

There have been some (rare) cases where correct setup didn't work as expected. Reboot did not help. Re-doing the same config again made things work. So it seems that sometimes some wrong configuration lingers somewhere and after re-doing the same config it finally gets overwritten.

I'd attribute your problem (and magical fixing) to this kind of problems.

There have been some (rare) cases where correct setup didn't work as expected. Reboot did not help. Re-doing the same config again made things work. So it seems that sometimes some wrong configuration lingers somewhere and after re-doing the same config it finally gets overwritten.

I'd attribute your problem (and magical fixing) to this kind of problems.

major WTF moment. would a config backup/factory reset/config restore fix such a rare occasion as well or is that unknown?

Backup creates binary blob, possinby taking corrupt binary config. Do it's not clear if reset/restore wold clear this kind of problem.

Backup creates binary blob, possinby taking corrupt binary config. Do it's not clear if reset/restore wold clear this kind of problem.

what about (verbose) export? i guess that should work...

If i read correctly, your problem was "fixed" as soon as you applied the config again and before the Reset/Restore procedure...
So there must be something else...

If i read correctly, your problem was "fixed" as soon as you applied the config again and before the Reset/Restore procedure...
So there must be something else...

it was fixed when i disabled

- bridge fast-forward
- allow fast path
- hw offload on the affected interfaces (server uplink and firewall uplink where the ARP traffic should come from/go to)
- limit broadcasts on the affected interfaces

and it kept being "fixed" even after re-enabling all that stuff.

Exactly...
So the reset and restore had nothing to do with fixing the problem or not...
Does everything still work if you lets say reboot both devices ?

Exactly...
So the reset and restore had nothing to do with fixing the problem or not...
Does everything still work if you lets say reboot both devices ?

i did not reset nor restore! that you must have misinterpreted.
i still have to try the reboot (and will do)

Backup creates binary blob, possinby taking corrupt binary config. Do it's not clear if reset/restore wold clear this kind of problem.

what about (verbose) export? i guess that should work...

Verbose export can only be applied to device with no configuration ... at least no visible configuration. So yes, unless a similar gremlin hides somewhere on device without any configuration, importing exported config should do the trick.

Exactly...
So the reset and restore had nothing to do with fixing the problem or not...
Does everything still work if you lets say reboot both devices ?

i did not reset nor restore! that you must have misinterpreted.
i still have to try the reboot (and will do)

ok sorry...
Personally i would reset my device to no-defaults and configure from the beginning... Then i would check if the problem persists or not...

ok thanks, people! if this ever occurs again, I'll know what to try.