[Bug?] InterVLAN Routing not working for directly attached Access Ports

Eldowin · Tue Dec 31, 2019 11:16 am

Edit 2: Not solved yet, but my access port config of my CRS seems to be the issue.
Could someone please check my config for the issue? Thanks a lot!

Edit 1: Added one more device to VLAN 255, which I pinged from VLAN 10 / 20. Same problem.

Hi,

I spent a few hours to set up InterVLAN Routing between 3 VLANs.
This works with one important exception: Traffic of clients, directly attached to an Untagged Access Port of my primary CRS210 router, won't get routed. All of these clients are able to reach the VLAN interfaces of the router, but not the network after the first hop.

This works perfectly fine with tagged ports. All clients behind my SwOS switches are able to reach every network behind my CRS210 router.

One example:
- My pfSense (IP 192.168.255.1) is able to be pinged from all SwOS devices in VLAN 10 / 20
- My pfSense (IP 192.168.255.1) is not pingable from all Access Ports, for which I configured an ingress VLAN directly at my CRS210
- For not working clients: One ping is going through, afterwards everything seems dropped.

VLAN Design:
VLAN 10: 192.168.10.0 /24 for Mgmt
VLAN 20: 192.168.20.0 /24 for Users
VLAN 255: 192.168.255.0 /30 as transit network between CRS210 and my pfSense Firewall --> Internet Access

Connection of my components
RouterOS Switch CRS210
- sfp1: Uplink SwOS Switch (Tagged VLAN 10)
- sfp2: Uplink SwOS Switch (Tagged VLAN 10, 20)
- ether8: Uplink pfSense Firewall (Tagged VLAN 255)
- ether1-7: Access Ports im VLAN 10 (Untagged PVID 10)

CRS210 configuration
Two notes to my config:
- Factory Reset was done before the config was applied, "No Default Configuration" and "Do Not Backup" were checked
- Intentionally no VLAN filtering was used, because HW offloading is needed

Following guides were used to create the config:
https://wiki.mikrotik.com/wiki/Manual:B ... s_switches
https://wiki.mikrotik.com/wiki/Manual:C ... s_examples
viewtopic.php?f=13&t=154701#p764807


This is the full configuration of my CRS210 router right now:
[code]
[admin@MikroTik] > /export hide-sensitive
# jan/01/2020 13:41:10 by RouterOS 6.46.1
#
# model = CRS210-8G-2S+
/interface bridge
add name=bridge protocol-mode=none
/interface vlan
add interface=bridge name=EG vlan-id=20
add interface=bridge name=Internet vlan-id=255
add interface=bridge name=OG vlan-id=10
/interface list
add name=WAN
add name=LAN
/ip pool
add name=dhcp_pool0 ranges=192.168.10.100-192.168.10.199
add name=dhcp_pool1 ranges=192.168.20.100-192.168.20.199
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=OG lease-time=3h name=dhcp1
add address-pool=dhcp_pool1 disabled=no interface=EG lease-time=3h name=dhcp2
/interface bridge port
add bridge=bridge interface=ether1
add bridge=bridge interface=ether2
add bridge=bridge interface=ether3
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5
add bridge=bridge interface=ether6
add bridge=bridge interface=ether7
add bridge=bridge interface=ether8
add bridge=bridge interface=sfp-sfpplus1
add bridge=bridge interface=sfpplus2
/interface ethernet switch egress-vlan-tag
add tagged-ports=switch1-cpu,sfpplus2,sfp-sfpplus1 vlan-id=10
add tagged-ports=switch1-cpu,sfpplus2 vlan-id=20
add tagged-ports=switch1-cpu,ether8 vlan-id=255
/interface ethernet switch ingress-vlan-translation
add customer-vid=0 new-customer-vid=10 ports=ether1,ether2,ether3,ether4,ether5,ether6,ether7
/interface ethernet switch vlan
add ports=switch1-cpu,ether1,ether2,ether3,ether4,ether5,ether6,ether7,sfpplus2,sfp-sfpplus1 vlan-id=10
add ports=switch1-cpu,sfpplus2 vlan-id=20
add ports=switch1-cpu,ether8 vlan-id=255
/interface list member
add interface=ether1 list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfpplus2 list=LAN
/ip address
add address=192.168.10.1/24 interface=OG network=192.168.10.0
add address=192.168.20.1/24 interface=EG network=192.168.20.0
add address=192.168.255.2/30 interface=Internet network=192.168.255.0
/ip dhcp-server network
add address=192.168.10.0/24 gateway=192.168.10.1
add address=192.168.20.0/24 gateway=192.168.20.1
/ip dns
set servers=192.168.255.1
/ip route
add distance=1 gateway=192.168.255.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/lcd
set enabled=no touch-screen=disabled
/system clock
set time-zone-name=Europe/Berlin
/system package update
set channel=long-term
/tool bandwidth-server
set enabled=no

It would be great to get some help from you.
Everyone, have a great and healthy new year!

mkx · Tue Dec 31, 2019 4:04 pm

If you test from a host in e.g. users VLAN, can you ping CRS? Can you ping pfSense? Can you ping border router (if that's not pfSense)?

My first guess would be that pfSense doesn't perform NAT for all of your internal subnets ...

Eldowin · Tue Dec 31, 2019 4:41 pm

Hi,

my pfSense doesn't route the VLAN 10 / 20 traffic. All clients behind my SwOS switches, connected via trunk link to my CRS, can successfully reach the pfSense.
The pfSense enables internet access to all clients.

SwOS clients in VLAN 10 / 20 are able to:
- Reach every RouterOS VLAN Interface in VLAN 10, 20, 255
- Reach the pfSense (IP: 192.168.255.1)
- Reach the internet gateway behind the pfSense

RouterOS clients in VLAN 10 / 20 are able to:
- Reach every RouterOS VLAN Interface in VLAN 10, 20, 255
- Cannot reach the pfSense (IP: 192.168.255.1)
- Cannot reach the internet gateway, the first hop cannot be passed

I guess it's an issue with the access port config on my CRS router. I followed all steps mentioned in the linked guides above, probably something is still missing.
My goal is to stay with HW offloading.

Edit: Do I need to configure the Tagging at "Bridge --> Ports" and "Bridge --> VLANs"?
There are different settings for "Frame Types". I'm afraid to miss HW offloading if I do so, since this isn't mentioned in the guide for CRS2xx.

mkx · Tue Dec 31, 2019 4:58 pm

As your clients are able to ping CRS' interfaces, I guess that L2 / VLAN setup is fine.

Does pfSense know that it has to use 192.168.255.2/30 as gateway to reach 192.168.10.0/24 and 192.168.20.1/24?

You can test this by running commands from CRS:

/ping 192.168.255.1
/ping 192.168.255.1 src-address=192.160.10.1

Eldowin · Tue Dec 31, 2019 5:01 pm

Hi,

this works. I configured my pfSense with two static routes:
- Destination network: 192.168.10.0 /24 Gateway: 192.168.255.2
- Destination network: 192.168.20.0 /24 Gateway: 192.168.255.2

The NAT outbound rules, which were created automatically by pfSense, are working fine. Clients are able to reach the internet.
Firewall rules allow every traffic for IPv4 and IPv6, this works too.

The issue seems to be the access port config at my CRS, sadly.
Edit: Moved a client of my CRS210 to one of my SwOS switches: Everything works immediately.

Could someone please tell me what is wrong with my access port config?

Eldowin · Wed Jan 01, 2020 12:37 pm

After spending more hours (really many now, I'm trying to set this up since a week), I don't know what else to try anymore. When setting an access port to PVID 10, my devices are Layer 2 reachable. The access port configuration on my CRS210 "somewhat works". Just the traffic of directly attached clients on my CRS doesn't get routed.

Like mentioned, there is no problem with clients attached to my SwOS switches, tagged traffic is forwarded correctly at my router.

I checked the behavior with multiple firmware branches, the issue occures with following versions:
- Long term 6.44.6
- Stable 6.46.1
- Testing 6.47beta8

Everything was configured like the Mikrotik guides, linked in my first post, mentioned. Using VLAN filtering via Bridge is no option for me, since I need HW offloading using my CRS210 hardware.

It would be great to get some help regarding the config. Maybe it's a firmware bug in all latest releases.

Best regards and happy new year

mkx · Wed Jan 01, 2020 2:39 pm

You're mentioning SwOS switches in the same sentence as CRS "router". Are those part of same LAN setup? Do devices connected to switches use same CRS as their default router?

I'm thinking about how to verify the access port settings on CRS ... if you could disconnect the rest of network, then use single device connected to access port of a VLAN to ping CRS' corresponding IP address and check which vlan interface changes packet count ... Alternatively you could construct a logging rule to log all ICMP traffic and check logs. However, if your CRS-connected devices receive IP settings via DHCP server and IP address belongs to expected VLAN, then I'd say L2 (VLAN) setup works. DHCP server doesn't seem to be part of your config as it seems, so I wonder ...

BTW, how about posting full actual CRS config? What you posted may be the configuration you entered the unit, but there might be some more remaining from default settings ... so run /export hide-sensitive and post output here (and try to hide as little as possible, ROS version and unit's serial number are not sensitive data in your case).

Eldowin · Wed Jan 01, 2020 2:56 pm

Thanks for your reply.

I'm using three Mikrotik devices in my LAN network, two SwOS switches are part of the same LAN setup. They're connected to the ports sfp1 and sfp2 to my CRS router (mentioned in my first post). The uplink ports are set as trunk ports. The Access ports of my SwOS switches are configured untagged with the needed PVID (10 or 20) assigned. Some of my clients are connected to SwOS switches, some of them directly to my CRS router. Only the traffic of devices connected to my SwOS switches is routed correctly.

The problem is, that traffic of clients connected to my CRS router is not routed to the gateway of my default route.
Traffic for those clients is not passed to the pfSense IP 192.168.255.1.

All clients are able to ping each existing VLAN interface of my CRS router (even 192.168.255.1 which is the Mikrotik IP in the same subnet as the pfSense Firewall), it doesn't matter where they are connected.
All clients are able to ping each other (for example client in VLAN 10 can ping a client in VLAN20), which means InterVLAN routing works somehow.
Clients connected to the CRS router are not able to ping their next hop where the default route is pointing too.
Clients connected to the SwOS switches are able to ping their next hop.

This is the full configuration of my CRS210 router right now:

[admin@MikroTik] > /export hide-sensitive
# jan/01/2020 13:41:10 by RouterOS 6.46.1
#
# model = CRS210-8G-2S+
/interface bridge
add name=bridge protocol-mode=none
/interface vlan
add interface=bridge name=EG vlan-id=20
add interface=bridge name=Internet vlan-id=255
add interface=bridge name=OG vlan-id=10
/interface list
add name=WAN
add name=LAN
/ip pool
add name=dhcp_pool0 ranges=192.168.10.100-192.168.10.199
add name=dhcp_pool1 ranges=192.168.20.100-192.168.20.199
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=OG lease-time=3h name=dhcp1
add address-pool=dhcp_pool1 disabled=no interface=EG lease-time=3h name=dhcp2
/interface bridge port
add bridge=bridge interface=ether1
add bridge=bridge interface=ether2
add bridge=bridge interface=ether3
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5
add bridge=bridge interface=ether6
add bridge=bridge interface=ether7
add bridge=bridge interface=ether8
add bridge=bridge interface=sfp-sfpplus1
add bridge=bridge interface=sfpplus2
/interface ethernet switch egress-vlan-tag
add tagged-ports=switch1-cpu,sfpplus2,sfp-sfpplus1 vlan-id=10
add tagged-ports=switch1-cpu,sfpplus2 vlan-id=20
add tagged-ports=switch1-cpu,ether8 vlan-id=255
/interface ethernet switch ingress-vlan-translation
add customer-vid=0 new-customer-vid=10 ports=ether1,ether2,ether3,ether4,ether5,ether6,ether7
/interface ethernet switch vlan
add ports=switch1-cpu,ether1,ether2,ether3,ether4,ether5,ether6,ether7,sfpplus2,sfp-sfpplus1 vlan-id=10
add ports=switch1-cpu,sfpplus2 vlan-id=20
add ports=switch1-cpu,ether8 vlan-id=255
/interface list member
add interface=ether1 list=LAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
add interface=ether6 list=LAN
add interface=ether7 list=LAN
add interface=ether8 list=LAN
add interface=sfp-sfpplus1 list=LAN
add interface=sfpplus2 list=LAN
/ip address
add address=192.168.10.1/24 interface=OG network=192.168.10.0
add address=192.168.20.1/24 interface=EG network=192.168.20.0
add address=192.168.255.2/30 interface=Internet network=192.168.255.0
/ip dhcp-server network
add address=192.168.10.0/24 gateway=192.168.10.1
add address=192.168.20.0/24 gateway=192.168.20.1
/ip dns
set servers=192.168.255.1
/ip route
add distance=1 gateway=192.168.255.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/lcd
set enabled=no touch-screen=disabled
/system clock
set time-zone-name=Europe/Berlin
/system package update
set channel=long-term
/tool bandwidth-server
set enabled=no

I'll see what I can do about the ICMP logging. Would be great if my config could be checked meanwhile.

Thanks a lot again!

mkx · Wed Jan 01, 2020 5:32 pm

If I compare the description of wanted situation from pist #1 to the actual configuration, I can see a few differences:

ether ports ether6 and ether7 are slightly misconfigured: tgey are listed as tagged members for egress and as access for ingress ... it may appear to be working fine if windows computers are connected to those ports as windows drivers like to strip all VLAN tags on ingress.
ether8 is tagged port ... which makes me wonder how can CRS communicate with pfSense ... unless pfSense is configured to use VLAN 255 as well.
port sfp-sfpplus1 is not tagged member of VLAN 20

I don't know how these things affect the working ...

Eldowin · Wed Jan 01, 2020 6:08 pm

Hi mkx,

I did a few configuration changes after my initial post which I didn't mention, sorry for that:
- ether6&7: Indeed a mistake which happened while testing, Only want them untagged. I will update the posted configuration above.
- ether8: Tagged on purpose, my pfSense is configured to receive 255 tagged frames. This works fine
- sfp-sfpplus1 war removed from VLAN 20, since only VLAN 10 is needed at this SwOS Switch. Just my second SwOS switch needs VLAN 20.

As you suspected, Windows Clients are indeed working with all untagged ports at my CRS. This is the only client type which has no problems with the port configuration.
I connected a Windows notebook via ethernet to port ether1, which is configured as ingress-vlan with PVID 10. Traffic for Windows clients is routed correctly. Other clients, like my Synology NAS and Android TV and Clients connected to my WLAN Access points (AP connected to port ether6, Client traffic also untagged), are not working.

Edit: First post was refreshed with the full config of my switch, also VLAN 20 was removed from my conditions for port sfp-sfpplus1

mkx · Wed Jan 01, 2020 6:33 pm

Other clients, like my Synology NAS and Android TV and Clients connected to my WLAN Access points, are not working.

NAS and Android TV are configured without VLAN awareness I presume.

So my guess is that untagging on egress doesn't work somehow. Can VLAN 10 devices, connected to one of SwOS devices, communicate with VLAN 10 devices mentioned (e.g. NAS)?

I can't see anything wrong in config. Although I don't have any experience with CRS2xx, so I'm possibly overlooking something.
What you can do is the following: save exported config to a PC, reset CRS to factory defaults and re-do the config (use exported config as cheat-sheet or to copy-paste config). Sometimes some setting doesn't get cleared even though it's not seen ... configuration reset clears that.

Eldowin · Wed Jan 01, 2020 6:48 pm

VLAN 10 devices at my SwOS switch are able to communicate with VLAN 10 devices at my CRS210.
Inside the LAN every communication works between all VLANs.

I rebooted the CRS and factory resetted the config multiple times, with the config posted.
When resetting the device, I set the options "No default configuration" and "Do not Backup", to have everything clean when setting up (no auto created bridge, no default 192.168.88.1 default IP)

Eldowin · Thu Jan 02, 2020 7:09 pm

Could the issue be, that the routing port (ether8) was added to my Bridge?
Should I remove ether8 from the bridge to get routing working?
Right now all ports of my switch are inside the same bridge.

Edit:
Nope, not the issue. Removed ether8 from the bridge, configured the VLAN Tagging directly on ether8, same problem.
Added ether8 back to my bridge, config is the same like in the first post now.

[Bug?] InterVLAN Routing not working for directly attached Access Ports

[Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports

Re: [Bug?] InterVLAN Routing not working for directly attached Access Ports