Dear friends,

Very long post, but trust me, there are questions at the end
After playing (and posting around this forum) with VLANs on “baby” equipment (CRS1xx series) I finally listened to wise people saying I should get real switches and moved on to the 3xx series, buying CRS326 and CSS326, and also one RB4011. Task remains the same, which I will try to briefly explain before moving on to my setup.

ISP provides internet (native vlan), IPTV (vlan), VOIP (vlan). The tv-boxes, despite broadcast being on iptv vlan, apparently also require native vlan connection, which ISP uses for remote firmware updates etc. Don’t ask me how, let’s just take it as fact.
ISP does not mind if I use “dumb” switches in front of my router, in order to split their signal into multiple cables which lead to multiple TV set-top boxes.
Existing setup is: ISP to dumb-switch1, from there one cable to TV-box, second table to router (behind which sits my internal network), and third cable to dumb-switch2 (and from here 4 cables to wall sockets for iptv).
(by a dumb switch I am refering to a most basic switch without management features, capable of forwarding also vlan frames – …no-name low cost switches for 10 EUR do the job just fine)

Goal: get rid of the dumb switches and use CRS326 and CSS326 to carry both – ISP network and my internal network.
Restrictions: ISP should not be able to see nor be aware of any of my internal devices, except the connection to router WAN port.

On my internal side I use 4 VLANs (22,25,1978 and 1112). Although the 1112 as per current physical setup does not need to be a VLAN, I rather have it future proof just in case I would need it elsewhere.

I have read few posts on this forum. The switchOS documentation is unfortunately little too narrow on information, which leads varying interpretations of what it means.
Therefore, with reference especially to (but not only) this post (viewtopic.php?p=1017926#p1017926), I have few questions below pictures of my setup.

Overview: Notes:

• Colors have two meanings: (1) on CRS326 and CSS326 they show port-isolation groups, i.e. only ports of the same color can forward communication between themselves; (2) color also depicts the VLANs which should be available on these ports (yellow=ISP VLANs, green=internal VLANs, blue=special internal VLAN).
• Under each port box you find information to where the cable from that port goes… (if empty, there is just a port number).
  Inside each port box you have information which VLAN it is carrying (in general, one VLAN number means it is access port – usually also includes text “access” or “acc” or “a##”; multiple VLAN numbers mean it is a trunk port with tagged VLANs; two ports on CSS326 – e9 and e23 – run in hybrid mode with access for 22 and tagged for 1978 – this is for testing purposes).
• CSS326-ether11 runs a trunk to a TP-link smart switch, which is not shown on the picture but it is not relevant (just for info, everything works as expected for devices connected to the TP link)
• Once setup is confirmed, I will run bonded connections (e.g. CRS 5&6 to CSS 1&2; CRS 7&8 to RB4011 2&3). But for now, bonds are not even defined and cables are not plugged-in, so there is just single cable link connecting switches/router as depicted.
• On RB4011 ports are not bridged, everything is routed. VLAN interfaces for vlan 22,25,1978 are defined under ether2 and vlan interface for 1112 is defined under ether5. DHCP servers run on each VLAN interface with appropriate network definitions, ip pools, etc.

CRS326 configuration screens:
System Port-isolation RSTP VLAN VLANs
CSS326 configuration screens:
System Port isolation RSTP VLAN VLANs
RB4011 configuration:
I am not sure if anything beyond what I wrote in Notes above is necessary, but if yes, please let me know and I will post relevant export.


Questions (finally):

1. Can we please clarify relationship between port isolation and vlans? Specifically, it is usually simply siad “using both could cause problems”, but that is too generic. Sub-questions:
   1. Does it hurt if I have both applied, as per my setup?
   2. If I disable (un-tick) Port-isolation on VLAN screen, does the port-isolation definition get ignored? If yes, does it then hurt to have the port isolation defined “just in case”, if it is ignored?
   3. Do I understand correctly, that if I defined VLAN mode “strict” on ALL ports, then I can achieve the same effect as having port isolation? E.g. as per my setup, if on CSS326 ports 4-8 are defined as strict and belong only to ISP VLANs, and the rest of ports is also mode “strict” and belong to (and only to) one or more of my internal VLANs, then ISP will never see any of the green ports and vice-versa?
2. RSTP – given my setup, should I run RSTP on internal (green, blue) ports? I certainly do not want it to run it on yellow ports (ISP can and probably should define my end as edge anyway…). I read a lot about RSTP and VLANs not liking each other, but again, the post are often misleading (e.g. saying one needs MSTP only in large setups where they want different root bridges per VLAN etc.). So in my case – is RSTP enough? Is it desired (probably yes for its potential loop protection)? Will it work?
3. Use of VLAN 1 – as per the forum thread quoted above, it is not recommended. Given my explanation and setup, in my case I think it is required. Can we please elaborate on this? Can I use VLAN 1 as defined? Will it work? What are the possible consequences?
   One different solution that comes to my mind is to use yellow ports in vlan mode=optional and not to define VLAN 1 as such. In such case, is my understanding correct that in such mode the native vlan of ISP will get through the yellow ports, but will never get to green or blue ports, while at the same time, if green and blue are in strict mode, even if a device on these ports would generate traffic with native vlan, it would never reach yellow ports?
4. Default VLAN ID – the use for me is clear in case of access ports and untagged frames, where incoming untagged frames get assigned the ID, and egress tagged with this ID get stripped.
   But what is the implication if:
   1. Port is accepting only tagged frames (probably the setting gets ignored).
   2. Port is accepting “any” frames (probably for untagged it is applied as if it was access port).
   3. How should one set the value in case port is: mode=strict; frame type=any and the port is member of VLANs for example 10,20,30? Somewhere I read that it should be set for a different VLAN id than those three. Why? My understanding is that if I set it to a different ID than the three, effectively the port becomes trunk, allowing only tagged 10, 20 or 30. If I set it to for example 10, port becomes hybrid, allowing tagged 20 and 30, but untagged gets assigned VLAN10. Is this interpretation correct?
5. Finally, Is there something you propose to modify in my setup and why?

These are the questions I would like to focus on first. Once we confirm the setup and I get it running fine, I will add (perhaps unrelated) additional questions, because I am also experiencing some strange behavior on some ports (e.g. look at CSS326 port 10 and port 22 – exactly the same setup, in ether 10 the mesh master does not work, in port 22 it does…). But I do not want to dig into this before we make sure that VLAN setup is correct.

Sorry for very long post, but hopefully contains all required info and will help someone in future as well.

Cheers,
Brandon.

Re: #4
Does your ISP provide the service over VLAN1 or in any other ways forces you to use it?
I might have missed where you explained how you were forced - with a gun pointed at your head - to use VLAN1.

The usual advice is to NOT use VLAN1, unless it is really-really needed because the consequences of using it (depending also on how other devices connected to the network behave) are often unpredictable and - historically - can range from "nothing" to "strange intermittent errors without apparent explanation".

tdw gave an explanation here (that may or may not apply to your case, it is about RouterOS):
viewtopic.php?t=206946#p1071170

Rules of Mikrotik Club (JFYI)
viewtopic.php?t=212419#p1108288

Re: #4
Does your ISP provide the service over VLAN1 or in any other ways forces you to use it?
I might have missed where you explained how you were forced - with a gun pointed at your head - to use VLAN1.

The usual advice is to NOT use VLAN1, unless it is really-really needed because the consequences of using it (depending also on how other devices connected to the network behave) are often unpredictable and - historically - can range from "nothing" to "strange intermittent errors without apparent explanation".

tdw gave an explanation here (that may or may not apply to your case, it is about RouterOS):
viewtopic.php?t=206946#p1071170

Rules of Mikrotik Club (JFYI)
viewtopic.php?t=212419#p1108288

Thank you. I think you refer to #3 not #4, as #4 is (or at least that was my intention) about the setting "Default VLAN ID" on the VLAN tab of the switch OS.
Anyway, I agree and do not argue. In fact, so far I have not been using VLAN 1 at all. The only reason and logic behind it was (I think it's coming from switch os documentation) that VLAN strict mode allows only packets with VLAN ID where that port is a member of that VLAN ID.
So logic was: if I want to replace port-isolation with VLANs only, I have to use strict mode. If I use strict mode, I must define VLAN ID 1 for those ports where ISP's network goes.

I have also proposed alternative solution in my #3 above. So if I am not to use VLAN 1 in this way, that means the yellow ports cannot be in strict mode, but must be in optional (or enabled?) mode. In that case, green and blue remain strict = there should be no native VLAN appearing on these ports, while yellow ports will accept IPTV VLAN as well as native VLAN. However, I am not sure if in this case yellow ports will also see VLAN communication from other VLANs where they do not belong (I hope not).

Would you then propose that this is the way to go to achieve ISP separation from my local VLANs while allowing native VLAN on ISP ports?
This would then answer and close #3 ...in fact, it would perhaps not answer #1, but definitely it would close #1 as well, as I would not have to use port-isolation at all then.

Cheers,
B.

I have also proposed alternative solution in my #3 above. So if I am not to use VLAN 1 in this way, that means the yellow ports cannot be in strict mode, but must be in optional (or enabled?) mode. In that case, green and blue remain strict = there should be no native VLAN appearing on these ports, while yellow ports will accept IPTV VLAN as well as native VLAN. However, I am not sure if in this case yellow ports will also see VLAN communication from other VLANs where they do not belong (I hope not).

Let me answer this myself and also take away some criticism of the switchOS documentation in this case... After going through it one more time it seems that the proper way to achieve the VLANs separation in my case is to use strict mode on internal VLAN ports and mode=enabled (not optional!) on ISP VLAN ports. So I will reconfigure switches this way and #3 and partially #1 is solved.

I still, however, would be grateful for insights on #1 (i.e. does port-isolation conflict with VLANS and what is the relationship to "tick box 'Port isolation' " on VLANs page). I am also interested in RSTP questions #2 and also about #4, although that is becoming more clear now after clarifying the modes...

Cheers,
B.

#3 and #4, my bad,
The point I was trying to make is that the VLAN1 is *somehow* used (at least in RouterOS) as the default VLAN for *something* and there is the risk that using it for *something else* might create conflicts.
Maybe in SwOS it is different, and the (few) people that actually fully understand VLANs on RouterOS (and SwOs) can actually use VLAN1 (not that I am among them).

What I am failing to understand in your post is actually #3, what is the actual reason why you chose VLAN1 (or if you prefer what would change in #3 if you used VLAN42 instead?)

This:

Given my explanation and setup, in my case I think it is required. Can we please elaborate on this?

Can you elaborate on the explanation and why exactly you believe it is required? (it may well be a perfectly valid explanation and requirement)

Oh, I see your question now...

That refers to this part of the introduction at the beginning of my original post:
"ISP provides internet (native vlan), IPTV (vlan), VOIP (vlan). The tv-boxes, despite broadcast being on iptv vlan, apparently also require native vlan connection, which ISP uses for remote firmware updates etc. Don’t ask me how, let’s just take it as fact."

That "fact" I found by coincidence, because once when ISP was reconfiguring their VLANs, my set-top boxes did not get updated, because only iptv.vlan was allowed to them. When troubleshooting with ISP they said it did not update, because they needed also native access.

Cheers,
B.

Yep, this is where I was not understanding.
Native VLAN doesn't necessarily mean VLAN1, do you mean that your ISP is using VLAN1 as "native"?
That would be the typical case of the gun pointed at your head.

Native VLAN doesn't necessarily mean VLAN1, do you mean that your ISP is using VLAN1 as "native"?

Well, I'm not sure how to check that. When I was trying to find out which VLANs they use, I simply ran a torch on input interface to see which VLANs appear there.
Now, since I have a switchOS on their input, when I go to "Hosts" tab, I see that on the ISP.IN interface there are hosts with VLAN corresponding to iptv (I don't want to post exact values here, but let's say its 5), voip (let's say it is 6) and then there are hosts on this interface with vlanID 1.

So this is the only deduction I have to think, that VLAN ID 1 is used as native (because this is also the VLAD ID, which provides internet access and where my router WAN port connects). So I do not know if I can draw a conclusion, that what Mikrotik displays as VLAN ID 1 on incoming traffic from ISP is the native VLAN. But that's how I came to that definition

EDIT: Or let's apply different logic here. If I did not know about iptv and voip VLANs, and wanted to use the connection just for internet, then I would have plugged the cable into the router and that is the VLAN ID 1 (at least what Mikrotik displays) that would be used. In other words, device without any configuration would use this VLAN ID. Set-top boxes communicate on VLAN ID "5", so they must be configured (by ISP) to send and receive tagged traffic on VLAN ID "5"... I do not use their voip service, but I suppose with those devices it is the same.

EDIT2: Would a proper term be, that instead of saying native VLAN or VLAN 1, I should rather say, that tv boxes require also untagged traffic? Maybe this way it makes more sense - from ISP the iptv broadcast comes on VLAN 5 as tagged frames, and ISP performed updates and configuration of the tvboxes via untagged frames (this is what MikroTik shows as VLAN ID 1).
My goal then would be, that this untagged traffic in both switches never gets to my local VLAN ports.

Cheers,
B.

Native VLAN doesn't necessarily mean VLAN1, do you mean that your ISP is using VLAN1 as "native"?

Well, I'm not sure how to check that. When I was trying to find out which VLANs they use, I simply ran a torch on input interface to see which VLANs appear there.

Most often "Native VLAN" means "untagged on wire side of interface" ... so your ISP most likely provides internet as untagged. It's then up to configuration of each individual VLAN-aware device as to which VLAN ID is used internally ... that's pvid property on bridge port.

Most device vendors (Mikrotik included) use VLAN ID 1 in such case. The problem in @OPs particular use case is that all ports will have same PVID setting and if using single bridge "native VLAN" from ISP side will leak into "native VLAN" on LAN side. Which is something not wanted at all. Solution is to use one PVID setting for "ISP native VLAN" and another PVID for "LAN native VLAN" ... and those then won't mix on the bridge as bridge passes all frames VLAN-tagged as soon as vlan-filtering is set to yes.
As mentioned, "native VLANs" are not tagged on wire ... so if there are multiple VLAN-aware devices chained one after another, each could (internally) use different VID as "native VLAN". And hence different VID can be used as "native VLAN" on different ports of same bridge.

Yes, this would make much more sense.
ISP do sometimes have "strange" settings, but in the cases where some services are on tagged VLAN the VLAN number is not 1, it is only the way the Mikrotik "sees" the untagged traffic.
@mkx
So, it still boils down to "do not use VLAN1" (unless you really know where your towel is), right?

So, it still boils down to "do not use VLAN1" (unless you really know where your towel is), right?

That's about it. So when in doubt, it's 42.

EDIT2: Would a proper term be, that instead of saying native VLAN or VLAN 1, I should rather say, that tv boxes require also untagged traffic? Maybe this way it makes more sense - from ISP the iptv broadcast comes on VLAN 5 as tagged frames, and ISP performed updates and configuration of the tvboxes via untagged frames (this is what MikroTik shows as VLAN ID 1).

That's the way my ISP does it: TV boxes use VLAN3999 for IPTV multicasts (and TR069 I guess) and untagged for general internet access (e.g. EPG download, youtube streaming, etc.). When household uses all-ISP-provided devies, their WAN router (can include DSL modem) will have a dedicated WAN port and then a few ports which can be configured as "data", "TV" or "trunk". "data" will provide only internet and will be untagged. "TV" will forward VLAN3999 (and some older TV boxes without fancy apps could do without internet access) and "trunk" will forward everything (I guess also VoIP VLAN, but I don't subscribe to their telephony so I don't know). My ISP uses PPPoE for internet access and their router is permanently in "bridge mode" for PPPoE (I just had to disable PPPoE client in router not to steal my "static IP" PPPoE connectivity). My MT router is then connected to "trunk" port.
On my side, I'm tagging untagged frames with my own VID (in particular pvid=2). WAN port is member of common bridge, just that WAN port has pvid=2 and LAN ports have pvid according to intents (XX for LAN, XY for IPTV internet access, XZ for guest WiFi, XW for certain IoT segment, etc.). Number of ports are tagged members of VID 3999 (WAN port and ports, connecting TV boxes). All in all it works well because TV boxes require tagged VLAN 3999 for multicast streams and untagged for internet ... and my own router provides them with untagged access to VLAN XY and TV boxes are happy with it.
I guess that TV boxes pull their FW upgrades ... via internet (whatever is provided to them), so they don't necessarily have transparent connectivity to ISP's IPTV management segment or whatever. If your ISP does push FW to TV boxes, then constraining them into walled VLAN might be problematic.

And nowhere I'm using VLAN ID 1 ...

Thank you both for taking time to respond. It's now much clearer and I guess a lot of original questions do not need answers anymore.
Although some for general knowledge would still be nice to have

As a follow-up for future readers, here is the revised configuration:
What has changed (besides nicer layout):
* there are no isolated switch groups on any of the two switches, which is later reflected in configuration of each switch. All ports can forward traffic between themselves.

for switches I will not show all screenshots again, I'll just summarise the configs
CRS326
Port isolation - default configuration = all ports can forward traffic to all ports

RSTP - disabled on yellow ports (except e2.RB4011.WAN) and enabled on all green and blue ports - perhaps it could be enabled, but I do not want my switches to interfere with STP running on ISP network (although I think they can handle that from their side).

VLAN

• yellow ports e1-e4: VLAN mode = enabled; VLAN receive = any; Default VLAN ID = 1
• green and blue ports: VLAN mode = strict; VLAN receive = tagged / untagged / any (trunk / access / hybrid); Default VLAN ID = 22

VLANs

• VLAN 5 (iptv): Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = yes; Members = e1, e3, e4
• VLAN 22: Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = yes; Members = e5-e10; e18-e23
• VLAN 25: Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = no; Members = e5-e8; e11-e14; e18-e23
• VLAN 1112: Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = no; Members = e15-e17
• VLAN 1978: Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = no; Members = e5-e8; e24

CSS326
Port isolation - default configuration = all ports can forward traffic to all ports

RSTP - disabled on yellow ports (e4-e8) and enabled on all green ports - perhaps it could be enabled, but I do not want my switches to interfere with STP running on ISP network (although I think they can handle that from their side).

VLAN

• yellow ports e4-e8: VLAN mode = enabled; VLAN receive = any; Default VLAN ID = 1
• green ports: VLAN mode = strict; VLAN receive = tagged / untagged / any (trunk / access / hybrid); Default VLAN ID = 22

VLANs

• VLAN 5 (iptv): Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = yes; Members = e4-e8
• VLAN 22: Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = yes; Members = e1-e2; e3; e9-e23
• VLAN 25: Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = no; Members = e1-e2; e11
• VLAN 1978: Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = no; Members = e1-e2; e24

I have not connected tvboxes yet, but as for local VLANs, seems to work.

Now, one note on Default VLAN ID:
Based on mkx post I understand, that I could run Default VLAN ID = 22 also on yellow ports. In fact, then I would not need a separate trunk connection for iptv, but could use a single trunk port. But because connection to wall sockets and from there to tvboxes are physically separate cables, I opted for this separation also in the configuration.
EDIT: also for memory, Default VLAN ID on hybrid ports defines the VLAN for untagged traffic on that port, so it is perfectly fine to have different default VLAN IDs on hybrid ports.

And from there spins my next question:
Since I have disabled RSTP on yellow ports and I have 2 cables running between the switches (e4-e4 yellow; e5-e1 green) - will that create a loop or not, given my VLAN configuration? (my understanding is, that yellow carries only VLAN 5 and untagged "1"; green carries only VLAN 22, 25, 1978 and untagged "22"... but I am not sure).
Because if it does create a loop, then I have to go mkx way...

Cheers,
B.

UPDATE:
I plugged in iptv cables, but tvboxes were receiving only IPs from ISP related to internet and not iptv.vlan.
Solution was to add another VLAN 3 (working name is ISP Native), and the assignment is as follows:
CRS326
ports e1 and e2 have default VLAN ID = 1
ports e3 and e4 have default VLAN ID = 3
The new VLAN 3 is defined as: Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = yes; Members = e1 - e4

CSS326
ports e4-e8 have default VLAN ID = 3
The new VLAN 3 is defined as: Port isolation = no; Learning = yes; Mirror = no; IGMP Snooping = yes; Members = e4 - e8

So input from ISP and Router WAN port stay on default VLAN ID =1, tv_boxes besides having iptv.vlan 5 (tagged) also need custom VLAN (in my case 3) which is used as default VLAN ID for untagged traffic.

So the above post SOLVES original post.

But I have one final question I have indicated in the first post, which I wanted to deal with once VLANs work...

I will add (perhaps unrelated) additional questions, because I am also experiencing some strange behavior on some ports (e.g. look at CSS326 port 10 and port 22 – exactly the same setup, in ether 10 the mesh master does not work, in port 22 it does…)

Meantime I have found the culprit. VLAN setup was exactly the same, but on System tab non-working ports were marked as not trusted.

Now the official documentation confuses, so can someone please explain it to me:

SwitchOS documentation
Trusted ports
Group of ports, which allows DHCP or PPPoE servers to provide a requested information. When enabled, it allows forwarding DHCP client packets towards the DHCP server through this port. Mainly used to limit unauthorized servers to provide malicious information for users, access ports usually do not configure as trusted. Ports that receive DHCP client packets with already added Option-82 must also be trusted, otherwise these packets are dropped. The setting does not apply to DHCPv6 packets.

From that I deducted that "Access ports should normally not be trusted" (which makes logical sense to me). But since I only get IP from DHCP on these servers only when access ports are trusted ("When enabled, it allows forwarding DHCP client packets towards the DHCP server through this port.") then it seems to be totally contrary statements.
So what should I do? Keep all trusted? Or should I disable the Option-82?

Regarding loop between yellow and green parts: if you're careful not to pass same VLAN (tagged or as native) via multiple ports, then there won't be a loop. RSTP or plain STP would detect a loop (their BPDUs disregard VLAN IDs), MSTP would be fine.

Another remark (it can be called personal preference): I never pass untagged frames between two LAN infrastructure devices ... or worded differently: all my LAN infrastructure interconnects are strictly trunk ports.
Benefit, applied to your setup: you could either use single interconnect between RB4011 and CRS and single interconnect between CRS and CSS. And for that I'd use DAC cables between devices' SFP+ ports (10 Gbps). Alternatively you could use pairs of UTP cables in 802.3ad bond (somehow increases bandwidth and more importantly adds redundancy) between each pair of devices..

(besides nicer layout):

I don't know , those 90° cable bends are ugly and they are not very good for the flow of data.
0's are usually OK, but 1's may get entangled in those sharp corners.
(I know it's an old joke, but I cannot resist)


Happy the whiole stuff is working.

OK. I mark this thread as solved.

Thank you @mkx and @jaclaz for your inputs.
@mkx - SPF connection is no-go for me, as the infrastructure is already buried in the walls. But I plan to run bonds, so I will eventually move to that solution.

As for the trusted ports part, turns out that I needed to turn of the DHCP Option-82 part.
Once turned-off, then the Trusted ports work as I thought - meaning disable trusted port on access ports, keep it on trunks. For ISP part of the network, you can experiment, but I kept it on trusted, as my clients (router WAN and tvboxes) need to get IP from ISP.

So thanks again for helping me out. It's been running for a few hours now without an issue, so I hope it stays that way.
I hope this helps someone else as well.

Cheers,
B.

SPF connection is no-go for me, as the infrastructure is already buried in the walls.

It's always option to take down the walls

You never mentioned how the 3 devices are placed physically, so I (wrongly it seems) assumed they are in same place.