Yet Another ISP VLAN split

BrandonSk · Thu Nov 10, 2022 7:08 pm

Hello,
my ISP is delivering Internet on native VLAN, and then IPTV and VOIP on specific 2 VLANs.
I created a lab environment to fine-tune the configuration before I go to the real network.
Goal is to split the traffic at router (will be RB4011, but for the Lab I am using CRS109) and then connect via separate cables the different VLANs to CRS125 switch, where traffic should be isolated from each other (ie. CRS125 will assume the role of 2 or 3 separate switches).

I have followed this guide: viewtopic.php?t=101586
and have two questions. But first the configuration.
1) I do not care about VOIP, so even that it is configured, let's focus on IPTV and Internet.
2) IPTV boxes expect tagged traffic, so the switch for IPTV should send tagged traffic on access ports.

ISP emulation (this device emulates the incoming connection from ISP on ether2, where there is native + 2 VLANs)

# nov/08/2022 11:27:20 by RouterOS 6.47.9
# software id = 61H1-88Z3
#
# model = RBmAP2nD
# serial number = DE4F0E836163
/interface bridge
add name=bridgeISP vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment=\
    "connect to home LAN to have internet access" name=e1-to-EDI-LAN
set [ find default-name=ether2 ] comment=\
    "emulation of ISP WAN connection to router" name=e2-emulate-ISP-WAN \
    poe-out=off
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface vlan
add interface=bridgeISP name=vlanIPTV3281 vlan-id=3281
add interface=bridgeISP name=vlanVOIP3282 vlan-id=3282
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip pool
add name=dhcp_pool5 ranges=10.20.30.2-10.20.30.254
add name=dhcp_pool6 ranges=10.32.81.2-10.32.81.254
add name=dhcp_pool7 ranges=10.32.82.2-10.32.82.254
/ip dhcp-server
add address-pool=dhcp_pool5 disabled=no interface=bridgeISP name=dhcp1
add address-pool=dhcp_pool6 disabled=no interface=vlanIPTV3281 name=dhcp2
add address-pool=dhcp_pool7 disabled=no interface=vlanVOIP3282 name=dhcp3
/interface bridge port
add bridge=bridgeISP hw=no interface=e2-emulate-ISP-WAN
add bridge=bridgeISP interface=vlanIPTV3281
add bridge=bridgeISP interface=vlanVOIP3282
/interface bridge vlan
add bridge=bridgeISP tagged=vlanIPTV3281,e2-emulate-ISP-WAN,bridgeISP \
    vlan-ids=3281
add bridge=bridgeISP tagged=vlanVOIP3282,e2-emulate-ISP-WAN,bridgeISP \
    vlan-ids=3282
/ip address
add address=10.32.81.1/24 interface=vlanIPTV3281 network=10.32.81.0
add address=10.32.82.1/24 interface=vlanVOIP3282 network=10.32.82.0
add address=10.20.30.1/24 interface=bridgeISP network=10.20.30.0
/ip dhcp-client
add disabled=no interface=e1-to-EDI-LAN
/ip dhcp-server network
add address=10.20.30.0/24 gateway=10.20.30.1
add address=10.32.81.0/24 gateway=10.32.81.1
add address=10.32.82.0/24 gateway=10.32.82.1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=e1-to-EDI-LAN
/ip route
add disabled=yes distance=1 gateway=e1-to-EDI-LAN
/system identity
set name=emulateISP

Router - CRS109

1) No firewall rules at the moment for LAB.
2) On eth8 I run management access network also connected to CRS125. It's not the proper plan management, it's only for the Lab purposes, no need to focus on it.

# nov/09/2022 21:37:41 by RouterOS 6.49.7
# software id = 4N0Y-WMM0
#
# model = CRS109-8G-1S-2HnD
# serial number = 522D04C45082
/interface bridge
add name=bridge-iptv vlan-filtering=yes
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface ethernet
set [ find default-name=ether6 ] name=ether3-LANs
set [ find default-name=ether4 ] name=ether4-IPTV
set [ find default-name=ether5 ] name=ether5-VOIP
/interface vlan
add interface=ether1 name=iptv-vlan vlan-id=3281
add interface=ether1 name=voip-vlan vlan-id=3282
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool1 ranges=192.168.5.2-192.168.5.254
/ip dhcp-server
add address-pool=dhcp_pool1 disabled=no interface=ether8 name=dhcp1
/interface bridge port
add bridge=bridge-iptv interface=iptv-vlan
add bridge=bridge-iptv interface=ether4-IPTV
/interface bridge vlan
add bridge=bridge-iptv tagged=ether1,bridge-iptv,iptv-vlan,ether4-IPTV \
    vlan-ids=3281
/ip address
add address=192.168.5.1/24 interface=ether8 network=192.168.5.0
/ip dhcp-client
add disabled=no interface=ether1
/ip dhcp-server network
add address=192.168.5.0/24 dns-server=192.168.5.1 gateway=192.168.5.1

Note: eth3 will contain my local VLANs which is not configured yet, but the focus from now is on IPTV part.

Switch - CRS125

1) Goal here is to have separate groups of ports (currently separate bridges are used, I haven't been successful with port isolation for some reason) for IPTV and LANs.
2) IPTV group are eth9-eth16, where eth9 is uplink to router (tagged), eth10-eth15 are also tagged, and eth16 is untagged (for testing purposes, so that I can see if I get IP assigned).
3) eth23 & eth24 are for admin access (same as eth8 in router) - no need to take into consideration.

# jan/01/2002 01:04:28 by RouterOS 6.49.1
# software id = 93ZN-U8NZ
#
# model = CRS125-24G-1S-2HnD
# serial number = 6232056FF060
/interface bridge
add name=bridge-admin
add name=bridge-iptv pvid=3281 vlan-filtering=yes
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge-iptv interface=ether9
add bridge=bridge-iptv interface=ether10
add bridge=bridge-iptv interface=ether11
add bridge=bridge-iptv interface=ether12
add bridge=bridge-iptv interface=ether13
add bridge=bridge-iptv interface=ether14
add bridge=bridge-iptv interface=ether15
add bridge=bridge-iptv interface=ether16
add bridge=bridge-admin interface=ether24
add bridge=bridge-admin interface=ether23
/interface bridge vlan
add bridge=bridge-iptv tagged=\
    ether9,bridge-iptv,ether10,ether11,ether12,ether13,ether14,ether15 \
    untagged=ether16 vlan-ids=3281
/ip address
add address=192.168.5.2/24 interface=bridge-admin network=192.168.5.0

With the above configuration the IPTV VLAN works as expected and I get correct IP assigned when connected to ether16.

Question #1:

Is the router configuration the right one (I took it from the post mentioned at the beginning), considering what is written in "Layer2 misconfiguration" topic, specifically the section "VLAN in a bridge with a physical interface"?

/interface vlan
add interface=ether1 name=VLAN99 vlan-id=99
/interface bridge
add name=bridge1
/interface bridge port
add interface=ether2 bridge=bridge1
add interface=VLAN99 bridge=bridge1

To me it seems to be exactly the case.
But then what would be my configuration for ether1 with native VLAN and 2 other VLANs? Something like this?

/interface bridge vlan
add bridge=bridge1 tagged=ether1 untagged=ether3 vlan-ids=1
add bridge=bridge1 tagged=ether1 untagged=ether4 vlan-ids=3281
add bridge=bridge1 tagged=ether1 untagged=ether5 vlan-ids=3282

Question #2:

When I want to use HW offload on CRS125, I have to go for Switch chip VLANs. When I applied the following configuration, I do not get an IP when I connect to ether16.

# jan/01/2002 01:12:01 by RouterOS 6.49.1
# software id = 93ZN-U8NZ
#
# model = CRS125-24G-1S-2HnD
# serial number = 6232056FF060
/interface bridge
add name=bridge-admin
add name=bridge-iptv
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface ethernet switch
set drop-if-invalid-or-src-port-not-member-of-vlan-on-ports=\
    ether9,ether10,ether11,ether12,ether13,ether14,ether15,ether16
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge-iptv interface=ether9
add bridge=bridge-iptv interface=ether10
add bridge=bridge-iptv interface=ether11
add bridge=bridge-iptv interface=ether12
add bridge=bridge-iptv interface=ether13
add bridge=bridge-iptv interface=ether14
add bridge=bridge-iptv interface=ether15
add bridge=bridge-iptv interface=ether16
add bridge=bridge-admin interface=ether24
add bridge=bridge-admin interface=ether23
/interface ethernet switch egress-vlan-tag
add tagged-ports=ether9,ether10,ether11,ether12,ether13,ether14,ether15 \
    vlan-id=3281
/interface ethernet switch ingress-vlan-translation
add new-customer-vid=3281 ports=ether16
/interface ethernet switch vlan
add ports=ether9,ether10,ether11,ether12,ether13,ether14,ether15,ether16 \
    vlan-id=3281
/ip address
add address=192.168.5.2/24 interface=bridge-admin network=192.168.5.0

I have no clue. The only think that comes to my mind is this comment from documentation:

Warning: Multiple hardware offloaded bridge configuration is designed as fast and simple port isolation solution, but it limits part of VLAN functionality supported by CRS switch-chip. For advanced configurations use one bridge within CRS switch chip for all ports, configure VLANs and isolate port groups with port isolation profile configuration.

If this is the reason, can you please provide some guidance how to achieve what I need on CRS125. As I mentioned, my first attempt (I did not save the config) with the port isolation did not work.

Sorry for a long post, but I tried to include everything important.

Thanks.
B.

mkx · Thu Nov 10, 2022 8:05 pm

Regarding config: the VLAN part is mostly wrong on all devices. The rule you broke most often is "never add vlan interface, anchored to bridge, back to bridge as port".

I suggest you to read:

about bridge functions
bridge has multiple personalities and it seems you got confused about what is what (don't feel ashamed, many of us were or still are)
tutorial on how to do VLANs with many example use cases. While your particular use might not be covered, you should get a fairly good idea after studying the article.

The CRS125 seems mostly fine, I guess the problem is that you're configuring two bridges and only one can be HW offloaded. If ROS somehow selects bridge-admin for HW offloading, then you're toast. You should convert the config to single bridge and set the management ports to be access ports to management VLAN (e.g. 666). At the same time set switch-cpu1 "port" as trunk port for management VID and then create and appropriately configure VLAN interface off common bridge.

tdw · Thu Nov 10, 2022 8:55 pm

CRS1xx/2xx devices do not support any hardware offloading when the bridge has vlan-filtering=yes. The bridge itself should be set to no and then configure the switch chip directly under /interface ethernet switch ... - see https://help.mikrotik.com/docs/pages/vi ... =103841835 and https://help.mikrotik.com/docs/pages/vi ... tBasedVLAN

BrandonSk · Fri Nov 11, 2022 2:07 am

CRS1xx/2xx devices do not support any hardware offloading when the bridge has vlan-filtering=yes. The bridge itself should be set to no and then configure the switch chip directly under /interface ethernet switch ... - see https://help.mikrotik.com/docs/pages/vi ... =103841835 and https://help.mikrotik.com/docs/pages/vi ... tBasedVLAN

Thanks tdw. I am aware of that limitation. As per many VLAN discussions on this forum, I wanted to try the "most universal way" = to configure vlans on bridge, first and then once it works, I would try to replicate to appropriate switch chip method. As mkx pointed out, I apparently had some confusion on the bridge part already.

@mkx - Thanks. I must say LOL. I thought I would get a minor correction and I get an "all wrong". While I am very much aware of the pcunite's guide (based on which I run VLANs currently in my network using the switch chip method), I was not aware of the other post explaining the bridge thing (it's also much newer, probably that's why).
And while I must say that I am still not sure I fully understand what sindy tried to explain, it did (hopefully) pushed me in the right direction.

So here is updated configuration:

ISP emulation

# nov/11/2022 00:16:22 by RouterOS 6.47.9
# software id = 61H1-88Z3
#
# model = RBmAP2nD
# serial number = DE4F0E836163
/interface bridge
add name=bridgeISP vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment=\
    "connect to home LAN to have internet access" name=e1-to-EDI-LAN
set [ find default-name=ether2 ] comment=\
    "emulation of ISP WAN connection to router" name=e2-emulate-ISP-WAN \
    poe-out=off
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface vlan
add interface=bridgeISP name=vlanIPTV3281 vlan-id=3281
add interface=bridgeISP name=vlanVOIP3282 vlan-id=3282
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip pool
add name=dhcp_pool5 ranges=10.20.30.2-10.20.30.254
add name=dhcp_pool6 ranges=10.32.81.2-10.32.81.254
add name=dhcp_pool7 ranges=10.32.82.2-10.32.82.254
/ip dhcp-server
add address-pool=dhcp_pool5 disabled=no interface=bridgeISP name=dhcp1
add address-pool=dhcp_pool6 disabled=no interface=vlanIPTV3281 name=dhcp2
add address-pool=dhcp_pool7 disabled=no interface=vlanVOIP3282 name=dhcp3
/interface bridge port
add bridge=bridgeISP hw=no interface=e2-emulate-ISP-WAN
/interface bridge vlan
add bridge=bridgeISP tagged=vlanIPTV3281,e2-emulate-ISP-WAN,bridgeISP \
    vlan-ids=3281
add bridge=bridgeISP tagged=vlanVOIP3282,e2-emulate-ISP-WAN,bridgeISP \
    vlan-ids=3282
/ip address
add address=10.32.81.1/24 interface=vlanIPTV3281 network=10.32.81.0
add address=10.32.82.1/24 interface=vlanVOIP3282 network=10.32.82.0
add address=10.20.30.1/24 interface=bridgeISP network=10.20.30.0
/ip dhcp-client
add disabled=no interface=e1-to-EDI-LAN
/ip dhcp-server network
add address=10.20.30.0/24 gateway=10.20.30.1
add address=10.32.81.0/24 gateway=10.32.81.1
add address=10.32.82.0/24 gateway=10.32.82.1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=e1-to-EDI-LAN
/ip route
add disabled=yes distance=1 gateway=e1-to-EDI-LAN
/system identity
set name=emulateISP

Router

# nov/11/2022 00:20:14 by RouterOS 6.49.7
# software id = 4N0Y-WMM0
#
# model = CRS109-8G-1S-2HnD
# serial number = 522D04C45082
/interface bridge
add name=bridge-router vlan-filtering=yes
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface ethernet
set [ find default-name=ether6 ] name=ether3-LANs
set [ find default-name=ether4 ] name=ether4-IPTV
set [ find default-name=ether5 ] name=ether5-VOIP
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool1 ranges=192.168.5.2-192.168.5.254
/ip dhcp-server
add address-pool=dhcp_pool1 disabled=no interface=ether8 name=dhcp1
/interface bridge port
add bridge=bridge-router interface=ether4-IPTV pvid=3281
add bridge=bridge-router interface=ether1
add bridge=bridge-router interface=ether3-LANs
add bridge=bridge-router interface=ether5-VOIP pvid=3282
/interface bridge vlan
add bridge=bridge-router tagged=ether1,ether4-IPTV vlan-ids=3281
add bridge=bridge-router tagged=ether1,ether5-VOIP vlan-ids=3282
/ip address
add address=192.168.5.1/24 interface=ether8 network=192.168.5.0
/ip dhcp-client
add disabled=no interface=bridge-router
/ip dhcp-server network
add address=192.168.5.0/24 dns-server=192.168.5.1 gateway=192.168.5.1

Switch

# jan/01/2002 08:59:58 by RouterOS 6.49.1
# software id = 93ZN-U8NZ
#
# model = CRS125-24G-1S-2HnD
# serial number = 6232056FF060
/interface bridge
add name=bridge-all vlan-filtering=yes
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge-all interface=ether9 pvid=3281
add bridge=bridge-all interface=ether10 pvid=3281
add bridge=bridge-all interface=ether11 pvid=3281
add bridge=bridge-all interface=ether12 pvid=3281
add bridge=bridge-all interface=ether13 pvid=3281
add bridge=bridge-all interface=ether14 pvid=3281
add bridge=bridge-all interface=ether15 pvid=3281
add bridge=bridge-all interface=ether16 pvid=3281
add bridge=bridge-all interface=ether24
add bridge=bridge-all interface=ether23
add bridge=bridge-all interface=ether1 pvid=3282
add bridge=bridge-all interface=ether2 pvid=3282
add bridge=bridge-all interface=ether3 pvid=3282
add bridge=bridge-all interface=ether4 pvid=3282
add bridge=bridge-all interface=ether5 pvid=3282
add bridge=bridge-all interface=ether6 pvid=3282
add bridge=bridge-all interface=ether7 pvid=3282
add bridge=bridge-all interface=ether8 pvid=3282
/interface bridge vlan
add bridge=bridge-all tagged=\
    ether9,ether10,ether11,ether12,ether13,ether14,ether15 untagged=ether16 \
    vlan-ids=3281
add bridge=bridge-all tagged=ether1,ether2,ether3,ether4,ether5,ether6,ether7 \
    untagged=ether8 vlan-ids=3282
/ip address
add address=192.168.5.2/24 interface=bridge-all network=192.168.5.0

So hopefully I got the bridge vlans now properly implemented.
Now, when I plug PC into ether16, I do get the correct IP assigned from 10.32.81.0 network.
Also ether24 works (the temporary management, not a vlan), but there the DHCP server runs on router, not on ISP emulator.
But when I try to plug PC into ehter8 on the switch, I get no IP address (I would expect one from 10.32.82.0 network).
Just for completeness, switch ether1 is connected to router ether5, and switch ether9 is connected to router ether4.

I cannot see a difference in the configs, so why one VLAN works and the other not?
Could it be related to RSTP? Because I noticed, that ether1 shows alternate. Would that mean that VLAN3282 traffic gets switched to ether9, which is VLAN3281 and therefore dropped? Should I switch to MSTP then?

Thanks.
B.

mkx · Fri Nov 11, 2022 7:09 am

Connecting switch with router using multiple physical links surely confuses RSTP. So you better disable xSTP on interconnection ports (or move to MSTP). More common is to run single trunk link (or use some sort of bonding if you want redundancy or capacity increase).

There's still some errors in VLAN config: you configured some ports with pvid set in /interface bridge port (so you intend them to be access ports for that VLAN) but you set them as tagged members of same VLAN under /interface bridge vlan which creates asymmetry (port expects untagged frames on ingress but doesn't strip tags on egress). This then works only because many windows network drivers strip all VLAN tags on ingress unless explicitly configured with 802.1q settings.

BrandonSk · Mon Nov 14, 2022 12:14 am

There's still some errors in VLAN config: you configured some ports with pvid set in /interface bridge port (so you intend them to be access ports for that VLAN) but you set them as tagged members of same VLAN under /interface bridge vlan which creates asymmetry (port expects untagged frames on ingress but doesn't strip tags on egress). This then works only because many windows network drivers strip all VLAN tags on ingress unless explicitly configured with 802.1q settings.

OK. I will get that fixed. I thought PVID more of a designates port to belong only to one VLAN (i.e. ignoring native and all other vlans) and the tagged/untagged stuff would be defined on the bridge...wrong again

Connecting switch with router using multiple physical links surely confuses RSTP. So you better disable xSTP on interconnection ports (or move to MSTP). More common is to run single trunk link (or use some sort of bonding if you want redundancy or capacity increase).

So very true. I could use a single trunk to have bandwidth. But I want to keep iptv away from everything else. I do not want ISPs VLANs mixed with my local VLANs on the same bridge. We are then back to my original intention. So, please allow me to move away from the actual configuration and let's focus on HW this time and how would you go about what I want to achieve.

Existing setup is as the following: ISP connection coming to a dumb switch, which sends out cables to iptv sockets or other dumb switch and so on. From my perspective it is still ISP's network, despite running on my cables. So the IPTV part never reaches my LAN. One cable from dumb switch runs to eth1 of RB4011, which is my gateway for local network. Router defines several locally used VLANs and is connected to CRS112 which then further distributes local traffic to internet sockets in the walls, to physical servers, APs, etc.

My idea was to get rid of dumb switches and replace them by one of the two switch chips on RB4011:
Incoming ISP connection to RB4011 let's say eth10. I would use RB's switch chip #2 to split ISP VLANs into eth9 (iptv), eth8 (voip), eth7 (internet). For internet, I would connect eth7 to eth1 by cable for wirespeed and DHCP client would be running on eth1, which would become my WAN gateway (I could basically then use what I have now for eth1, including firewall rules etc.). I would use then eth2-eth5 of the RB4011 (switch chip #1) for my local network and local VLANs.

Then on CRS125 I wanted to do port isolation, so I thought that I isolate eth17-eth24 into their own community for iptv and run cable from RB4011 eth9 to one of these 8 ports and remaining 7 would distribute the iptv around the house. I would isolate eth1-eth8 for my LAN and VLANs (and run cable from RB4011 eth2 to one of these and...), and eth9-eth16 for my servers and NAS (again, cable from RB4011 eth3 to one of these...). CRS125 would then simulate 3 physical switches. (Since port isolation does not work with vlans on CRS1xx, then this lead me to dead end...). And RB4011 would be doing all the inter-vlan and internet routing, firewalling etc.

So what would you recommend? Besides CRS125, I have CRS112 and CRS109 available to help, but I would like to keep the number of boxes to minimum. One could say to put CRS125 in front of RB4011 and connect ISP to CRS125, split VLANs and go to RB4011. But then for my local VLANs I would have to go back from RB4011 to CRS125 again.
Or I could use CRS125 as switch and router. But because I am running several IPSEC tunnels to other locations, CRS cannot do hw encryption and would get maybe overloaded by that.

Goal is simple - reduce number of boxes, keep IPTV away from my local LAN and VLANs and use RB4011 as router for the internet as well as local inter-vlan routing. But I feel caught in a circle of hw limitations at the moment.

mkx · Mon Nov 14, 2022 7:25 am

I thought PVID more of a designates port to belong only to one VLAN (i.e. ignoring native and all other vlans) and the tagged/untagged stuff would be defined on the bridge...

PVID setting is ignored only if frame-types property is set to admit-only-vlan-tagged on some particular port.

We are then back to my original intention.

My suggestion: use single (logical) trunk connection between router and switch with all VLANs included. If you have ports to spare both on RB4011 and CRS125, then you can create a bond (which is then your logical link), but that indeed might not distribute traffic evenly. If your WAN is well below 1Gbps and you won't have much of inter-VLAN traffic (both traffic types will hit the router-switch connection), then you can keep things simple and use single ethernet connection.

If you'll do the VLAN setup correctly, then those will be separate as good as if you were using separate (dumb) switches. Even if you'll be using trunk interconnects between switches and router.

BrandonSk · Mon Nov 14, 2022 7:18 pm

Thank you mkx. Although I have a 1 Gbit up & down, I think that the hw should handle that. Now I only need to find time to actually do it and if I run into issues, I will come back to this thread.
Thanks again,
B.

BrandonSk · Sat Nov 26, 2022 9:39 am

Hello,
I thought it might be a good idea to leave the latest configuration here, since I marked the thread as solved.
How we got here:
The original idea was to have incoming connection from ISP (Untagged internet, and tagged IPTV and VOIP) to RB4011 and from there pass the traffic tagged traffic via 2 physical links to CRS125 switch and a 3rd physical connection to the same switch which would be my local VLANs. On the switch I would simulate 3 separate switches by using the port isolation feature. This turned out to be a no-go, because on CRS125 the port isolation and VLANs do not work together well. So next was to have a single "switch"(bridge) on CRS125 and do only VLANs separation, but then when I used 3 separate physical connection, it obviously created loops (STP). Disabling STP would be probably one option, but then it made much more sense, in this particular approach, to combine the 3 separate links into one logical bond (or trunk, as they name it on the switch-chip).

Thanks again to mkx and others for their guidance. I have changed my configuration based on suggestion to use bonds and used appropriately modified configuration from this guide https://wiki.mikrotik.com/wiki/Manual:C ... ith_Trunks

What is below seems, for me, to do everything I need in my testing setup. I have inserted few comments into the export to describe selected sections.
Final note: security is not dealt with in the config. There is only basic firewall example and no L2 security considerations.

ISP (this devices simulates incoming traffic from ISP)

# nov/23/2022 21:55:12 by RouterOS 6.47.9
# software id = 61H1-88Z3
#
# model = RBmAP2nD
# serial number = DE4F0E836163
/interface bridge
add name=bridgeISP vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment=\
    "connect to home LAN to have internet access" name=e1-to-EDI-LAN
set [ find default-name=ether2 ] comment=\
    "emulation of ISP WAN connection to router" name=e2-emulate-ISP-WAN \
    poe-out=off
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface vlan
add interface=bridgeISP name=vlanIPTV3281 vlan-id=3281
add interface=bridgeISP name=vlanVOIP3282 vlan-id=3282
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip pool
add name=dhcp_pool5 ranges=10.20.30.2-10.20.30.254
add name=dhcp_pool6 ranges=10.32.81.2-10.32.81.254
add name=dhcp_pool7 ranges=10.32.82.2-10.32.82.254
/ip dhcp-server
add address-pool=dhcp_pool5 disabled=no interface=bridgeISP name=dhcp1
add address-pool=dhcp_pool6 disabled=no interface=vlanIPTV3281 name=dhcp2
add address-pool=dhcp_pool7 disabled=no interface=vlanVOIP3282 name=dhcp3
/interface bridge port
add bridge=bridgeISP hw=no interface=e2-emulate-ISP-WAN
/interface bridge vlan
add bridge=bridgeISP tagged=vlanIPTV3281,e2-emulate-ISP-WAN,bridgeISP \
    vlan-ids=3281
add bridge=bridgeISP tagged=vlanVOIP3282,e2-emulate-ISP-WAN,bridgeISP \
    vlan-ids=3282
/ip address
add address=10.32.81.1/24 interface=vlanIPTV3281 network=10.32.81.0
add address=10.32.82.1/24 interface=vlanVOIP3282 network=10.32.82.0
add address=10.20.30.1/24 interface=bridgeISP network=10.20.30.0
/ip dhcp-client
add disabled=no interface=e1-to-EDI-LAN
/ip dhcp-server network
add address=10.20.30.0/24 gateway=10.20.30.1
add address=10.32.81.0/24 gateway=10.32.81.1
add address=10.32.82.0/24 gateway=10.32.82.1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=e1-to-EDI-LAN
/ip route
add disabled=yes distance=1 gateway=e1-to-EDI-LAN
/system identity
set name=emulateISP

Router - CRS109 (will be RB4011 in production)

# nov/23/2022 21:53:18 by RouterOS 6.49.7
# software id = 4N0Y-WMM0
#
# model = CRS109-8G-1S-2HnD
# serial number = 522D04C45082

# Disable STP not to mess-up with ISP's network. (Todo: disable BDPUs on ether1)
/interface bridge
add name=bridge-router protocol-mode=none
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik

# IPTV and VOIP VLANs are from ISP and must run under bridge-interface
/interface vlan
add comment="ISP VLAN, must run under bridge interface" interface=\
    bridge-router name=VLAN3281 vlan-id=3281
add comment="ISP VLAN, must run under bridge interface" interface=\
    bridge-router name=VLAN3282 vlan-id=3282

# Create a bond interface to switch
# (Note: CRS109 can do it via switch-chip trunk, but here we simulate RB4011)
/interface bonding
add mode=balance-xor name=trunk2-4_to_switch slaves=ether2,ether3,ether4 \
    transmit-hash-policy=layer-2-and-3

# Create "local VLANs" under the bond interface
/interface vlan
add interface=trunk2-4_to_switch name=VLAN22 vlan-id=22
add interface=trunk2-4_to_switch name=VLAN25 vlan-id=25
add interface=trunk2-4_to_switch name=VLAN1008 vlan-id=1008

# These are used for firewall rules
/interface list
add comment="Incoming from ISP" name=WAN
add comment="Contains all local VLANs" name=LANs
add name=ISP-IPTV_VOIP
add comment="LANs that can be srcnat-ed" name=LAN2INET
add comment="Internet traffic from ISP." name=ISP-INET
add name=SERVERS

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik

# DHCP for local VLANs
/ip pool
add name=dhcp_pool_vlan22 ranges=10.201.22.2-10.201.22.254
add name=dhcp_pool6_vlan25 ranges=10.201.25.2-10.201.25.254
add name=dhcp_pool7 ranges=10.10.08.100-10.10.08.254
/ip dhcp-server
add address-pool=dhcp_pool_vlan22 disabled=no interface=VLAN22 name=\
    dhcp-VLAN22
add address-pool=dhcp_pool6_vlan25 disabled=no interface=VLAN25 name=\
    dhcp-VLAN25
add address-pool=dhcp_pool7 disabled=no interface=VLAN1008 name=\
    dhcp4-VLAN1008

# Bridge connect ISP and Trunk
/interface bridge port
add bridge=bridge-router interface=ether1
add bridge=bridge-router interface=trunk2-4_to_switch

# For firewalling
/interface list member
add interface=ether1 list=WAN
add interface=VLAN3281 list=WAN
add interface=VLAN3282 list=WAN
add comment="Bridge gets IP in the default VLAN from ISP." interface=\
    bridge-router list=WAN
add interface=VLAN22 list=LANs
add interface=VLAN25 list=LANs
add interface=VLAN1008 list=LANs
add interface=trunk2-4_to_switch list=LANs
add interface=VLAN3281 list=ISP-IPTV_VOIP
add interface=VLAN3282 list=ISP-IPTV_VOIP
add comment=Remove interface=VLAN1008 list=LAN2INET
add interface=VLAN25 list=LAN2INET
add interface=VLAN22 list=LAN2INET
add interface=ether1 list=ISP-INET
add interface=bridge-router list=ISP-INET
add interface=VLAN25 list=SERVERS

# Assign local IP addresses
/ip address
add address=192.168.99.1/24 disabled=yes network=192.168.99.0
add address=10.32.81.1/24 comment=REMOVE interface=VLAN3281 network=\
    10.32.81.0
add address=10.32.82.1/24 comment=REMOVE interface=VLAN3282 network=\
    10.32.82.0
add address=10.201.22.1/24 interface=VLAN22 network=10.201.22.0
add address=10.201.25.1/24 interface=VLAN25 network=10.201.25.0
add address=10.10.08.1/24 interface=VLAN1008 network=10.10.08.0

# ISP Internet (untagged/VLAN1) will be connected to bridge, which will be my gateway. 
/ip dhcp-client
add disabled=no interface=bridge-router

# DHCP for local networks (continued)
/ip dhcp-server network
add address=10.10.08.0/24 dns-server=10.10.08.1 gateway=10.10.08.1
add address=10.201.22.0/24 comment=LAN dns-server=10.201.22.1 gateway=\
    10.201.22.1
add address=10.201.25.0/24 comment=Servers dns-server=10.201.25.1 gateway=\
    10.201.25.1

/ip dns
set allow-remote-requests=yes servers=8.8.8.8

# EXAMPLE(!) firewall only. Needs many more rules, but this can be used as example to:
# 1) Isolate IPTV and VOIP traffic completely from input and forwarding.
# 2) Protect from Internet traffic on input and allow only dstnat traffic.
# 3) Masqurade(srcnat) only traffic from local VLANs
# 4) Does not allow routing between local VLANs, except to/from SERVERS.
#
# ...as said, a some more is needed, but this is not the point now.
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LANs
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="No forwarding from ISP interfaces!" \
    connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=accept chain=forward comment=\
    "Allow port forwarding only for Internet traffic from ISP!" \
    connection-nat-state=dstnat connection-state=new in-interface-list=\
    ISP-INET
add action=accept chain=forward comment="Allow LANs to access servers." \
    in-interface-list=LANs out-interface-list=SERVERS
add action=accept chain=forward comment="Allow Servers to access LANs." \
    in-interface-list=SERVERS out-interface-list=LANs
add action=drop chain=forward comment="Drop everything else."
/ip firewall nat
add action=masquerade chain=srcnat comment=\
    "Masquarade internet traffic from LANs, but not VOIP and IPTV." \
    out-interface-list=ISP-INET
#
# Further security must be made at L2 level (e.g. discovery, MAC access, etc.)
# but is not part of this excercise.

Switch - CRS125

# jan/09/2002 00:21:24 by RouterOS 6.49.1
# software id = 93ZN-U8NZ
#
# model = CRS125-24G-1S-2HnD
# serial number = 6232056FF060

# One bridge for all ports
/interface bridge
add name=bridge-switch protocol-mode=none

/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik

# Add management VLAN intercase to the bridge
/interface vlan
add interface=bridge-switch name=VLAN1008 vlan-id=1008

# VLAN filtering on all ports (hw off-loaded via switch-chip.)
/interface ethernet switch
set drop-if-invalid-or-src-port-not-member-of-vlan-on-ports="ether1,ether2,eth\
    er3,ether4,ether5,ether6,ether7,ether8,ether9,ether10,ether11,ether12,ethe\
    r13,ether14,ether15,ether16,ether17,ether18,ether19,ether20,ether21,ether2\
    2,ether23,ether24"

# Define trunks(bonds) ports via switch-chip.
# There are 2 - one to router and one to another switch
/interface ethernet switch trunk
add member-ports=ether2,ether3,ether4 name=trunk2-4_to_router
add comment=Attic_switch member-ports=ether5,ether6 name=trunk5-6_to_switch

/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik

# Ports 2-24 go into bridge. Ether1 is emergency management port for direct connection.
/interface bridge port
add bridge=bridge-switch interface=ether2
add bridge=bridge-switch interface=ether3
add bridge=bridge-switch interface=ether4
add bridge=bridge-switch interface=ether5
add bridge=bridge-switch interface=ether6
add bridge=bridge-switch interface=ether7
add bridge=bridge-switch interface=ether8
add bridge=bridge-switch interface=ether9
add bridge=bridge-switch interface=ether10
add bridge=bridge-switch interface=ether11
add bridge=bridge-switch interface=ether12
add bridge=bridge-switch interface=ether13
add bridge=bridge-switch interface=ether14
add bridge=bridge-switch interface=ether15
add bridge=bridge-switch interface=ether16
add bridge=bridge-switch interface=ether17
add bridge=bridge-switch interface=ether18
add bridge=bridge-switch interface=ether19
add bridge=bridge-switch interface=ether20
add bridge=bridge-switch interface=ether21
add bridge=bridge-switch interface=ether22
add bridge=bridge-switch interface=ether1
add bridge=bridge-switch interface=ether23
add bridge=bridge-switch interface=ether24

# VLAN definition for trunk and access ports.

# Tagged ports
/interface ethernet switch egress-vlan-tag
add comment="VOIP tagged" tagged-ports=trunk2-4_to_router,trunk5-6_to_switch vlan-id=\
    3282
add comment="IPTV tagged" tagged-ports="trunk2-4_to_router,trunk5-6_to_switch,\
    ether17,ether18,ether19,ether20,ether21,ether22" vlan-id=3281
add comment=LAN tagged-ports=trunk2-4_to_router,trunk5-6_to_switch vlan-id=22
add comment=Servers tagged-ports=trunk2-4_to_router,trunk5-6_to_switch \
    vlan-id=25
add comment=Management tagged-ports=trunk2-4_to_router,trunk5-6_to_switch \
    vlan-id=1008

# Access ports (IPTV and VOIP are only for testing purposes. Will be tagged in production.)
/interface ethernet switch ingress-vlan-translation
add comment="IPTV access - temporary - CHANGE TO TAGGED" new-customer-vid=\
    3281 ports=ether24
add comment="VOIP access - temporary - CHANGE TO TAGGED" new-customer-vid=\
    3282 ports=ether23
add comment="Servers - access" new-customer-vid=25 ports=ether7,ether8
add comment="LAN access" new-customer-vid=22 ports=\
    ether9,ether10,ether11,ether12,ether13,ether14,ether15,ether16
add comment=Management new-customer-vid=1008 ports=ether1,switch1-cpu

# Define which ports participate in which VLAN(s)
/interface ethernet switch vlan
add ports=trunk2-4_to_router,trunk5-6_to_switch,ether23 vlan-id=3282
add ports="trunk2-4_to_router,trunk5-6_to_switch,ether17,ether18,ether19,ether\
    20,ether21,ether22,ether24" vlan-id=3281
add ports=trunk2-4_to_router,trunk5-6_to_switch,ether7,ether8 vlan-id=25
add ports="trunk2-4_to_router,trunk5-6_to_switch,ether9,ether10,ether11,ether1\
    2,ether13,ether14,ether15,ether16" vlan-id=22
add ports=trunk2-4_to_router,trunk5-6_to_switch,ether1,switch1-cpu vlan-id=\
    1008

# Define management IP address
/ip address
add address=10.10.08.2/24 interface=bridge-switch network=10.10.08.0

As said, this seems to work (and I pray that forum guru's don't come with another "all wrong"

).
There are still some open questions, for example the bond protocol mode. As the switch-chip in CRS125 does not support 802.3ad (using CPU yes), then I used the balance-xor. Not perfect, because maximum per connection is still 1Gbit, but if the tv traffic and lan traffic get balanced over multiple links, it could be ok. I guess I will only find out in production then, and given my free time to play with this it will be probably in 2023

Have a great day everyone.
B.

mkx · Sat Nov 26, 2022 10:46 am

There are still some open questions, for example the bond protocol mode. As the switch-chip in CRS125 does not support 802.3ad (using CPU yes), then I used the balance-xor. Not perfect, because maximum per connection is still 1Gbit ..

None of bond modes support using multiple bond links for single L4 connection. Which includes 802.3ad.
One notable exception is ballance-rr, which is "proprietary" to linux and hence always in software ... it's not widely used with good reason: it doesn't ensure in-order delivery of packets (out of order delivery messes UDP hard and causes TCP to retransmit packets which in turn causes massive drops in throughput).

Yet Another ISP VLAN split [SOLVED]

Yet Another ISP VLAN split

Re: Yet Another ISP VLAN split

Re: Yet Another ISP VLAN split

Re: Yet Another ISP VLAN split

Re: Yet Another ISP VLAN split

Re: Yet Another ISP VLAN split

Re: Yet Another ISP VLAN split [SOLVED]

Re: Yet Another ISP VLAN split

Re: Yet Another ISP VLAN split

Re: Yet Another ISP VLAN split