QinQ VLAN's Help needed

CZFan · Sun Jun 10, 2018 8:28 pm

I am treading in waters I have not done before and it is a semi live network, so I need to get my ducks in a row, below is what I need:

Cust 1 ---- C-Vlan 10 -----
\
Cust 2 ---- C-Vlan 20 -------\--- CCR1036 -- S-Vlan 50 ---- Co Loc for ISP's
/
Cust 3 ---- C-Vlan 30 ---- /

Customers coming in on their relevant Vlan's to the CCR1036, then encapsulate the C-Vlan's into one Service Vlan to the co location where the S-Vlan will be stripped again and then be routed accordingly to the relevant ISP based on original Vlan's.

I have read somewhere on the Wiki that only "initial" support for this is available on bridges from 6.43RC14 I think it was (can't find it now). I have some concerns running Release Candidate versions in production.

Is above possible at the moment without running RC version, if so, can you provide some guidance on how I can achieve this?

Thanking you in advance

sindy · Sun Jun 10, 2018 9:34 pm

So you want it like this, right?

Ethernet II, Src: Routerbo_78:5a:37 (64:d1:54:78:4a:37), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
IEEE 802.1ad, ID: 50
    000. .... .... .... = Priority: 0
    ...0 .... .... .... = DEI: 0
    .... 0000 0011 0010 = ID: 50
    Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 10
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = DEI: Ineligible
    .... 0000 0000 1010 = ID: 10
    Type: ARP (0x0806)
Address Resolution Protocol (request)

I haven't tried yet, but from what the wiki says, the "initial support of QinQ" is relevant to using vlan-filtering on frames with 802.1ad tags on a bridge, so unless you need a bridge hosting several s-vlans, you don't need that feature. So if you want to avoid rc in production and you have enough CPU for adding the s-tag using /interface vlan, it is possible to do it the following way:

                 bridge-x
ether1---tag(c10)===|
ether2---tag(c20)===|---s-vlan50===ether4
ether3---tag(c30)===|

(--- ... tagless side, === ... tagged side)
i.e.

/interface vlan
add interface=ether4 name=s-vlan50 use-service-tag=yes vlan-id=50

/interface bridge
add name=bridge-x vlan-filtering=yes

/interface bridge port
add bridge=bridge-x interface=s-vlan50
add bridge=bridge-x interface=ether1 pvid=10
add bridge=bridge-x interface=ether2 pvid=20
add bridge=bridge-x interface=ether3 pvid=30

/interface bridge vlan
add bridge=bridge-x tagged=bridge-x,s-vlan50 untagged=ether1 vlan-ids=10
add bridge=bridge-x tagged=bridge-x,s-vlan50 untagged=ether2 vlan-ids=20
add bridge=bridge-x tagged=bridge-x,s-vlan50 untagged=ether3 vlan-ids=30

I haven't tested it completely, so be careful.

Of course if the CCR already gets c10,c20,c30 tagged in a single trunk, all you need is

      bridge-x
ether1===|---s-vlan50===ether4

so all shrinks down to just

/interface vlan
add interface=ether4 name=s-vlan50 use-service-tag=yes vlan-id=50

/interface bridge
add name=bridge-x vlan-filtering=yes

/interface bridge port
add bridge=bridge-x interface=s-vlan50
add bridge=bridge-x interface=ether1

/interface bridge vlan
add bridge=bridge-x tagged=bridge-x,ether1,s-vlan50 vlan-ids=10,20,30

paulct · Mon Jun 11, 2018 10:04 am

I am treading in waters I have not done before and it is a semi live network, so I need to get my ducks in a row, below is what I need:

Cust 1 ---- C-Vlan 10 -----
\
Cust 2 ---- C-Vlan 20 -------\--- CCR1036 -- S-Vlan 50 ---- Co Loc for ISP's
/
Cust 3 ---- C-Vlan 30 ---- /

Customers coming in on their relevant Vlan's to the CCR1036, then encapsulate the C-Vlan's into one Service Vlan to the co location where the S-Vlan will be stripped again and then be routed accordingly to the relevant ISP based on original Vlan's.

I have read somewhere on the Wiki that only "initial" support for this is available on bridges from 6.43RC14 I think it was (can't find it now). I have some concerns running Release Candidate versions in production.

Is above possible at the moment without running RC version, if so, can you provide some guidance on how I can achieve this?

Thanking you in advance

QinQ is fine for this. However we have only used CCR's for this. We have not tried it in the CRS platforms.
BUT, what happens when you need more than one S-tag?

e.g C-tag 10, 20, 30 to S-tag 100
and
C-tag 11,21,31 to S-tag 101

Then you need Selective q-in-q if it needs to be done on the same uplink interface.

sindy · Mon Jun 11, 2018 10:26 am

This can be obtained using vlan-filtering on the bridge hosting the c-vlans and two s-vlan interfaces:

/interface vlan
add interface=ether4 name=s-vlan-100 use-service-tag=yes vlan-id=100
add interface=ether4 name=s-vlan-101 use-service-tag=yes vlan-id=101

/interface bridge
add name=bridge-x vlan-filtering=yes

/interface bridge port
add bridge=bridge-x interface=s-vlan-100
add bridge=bridge-x interface=s-vlan-101
add bridge=bridge-x interface=ether1

/interface bridge vlan
add bridge=bridge-x tagged=bridge-x,ether1,s-vlan100 vlan-ids=10,20,30
add bridge=bridge-x tagged=bridge-x,ether1,s-vlan101 vlan-ids=11,21,31

paulct · Mon Jun 11, 2018 10:33 am

This can be obtained using vlan-filtering on the bridge hosting the c-vlans and two s-vlan interfaces:

/interface vlan
add interface=ether4 name=s-vlan-100 use-service-tag=yes vlan-id=100
add interface=ether4 name=s-vlan-101 use-service-tag=yes vlan-id=101

/interface bridge
add name=bridge-x vlan-filtering=yes

/interface bridge port
add bridge=bridge-x interface=s-vlan-100
add bridge=bridge-x interface=s-vlan-101
add bridge=bridge-x interface=ether1

/interface bridge vlan
add bridge=bridge-x tagged=bridge-x,ether1,s-vlan100 vlan-ids=10,20,30
add bridge=bridge-x tagged=bridge-x,ether1,s-vlan101 vlan-ids=11,21,31

Strange - this is from Mikrotik support 3 weeks ago.

"At the beginning it is planned to have one s-tag for all c-tags on port.
Selective tagging might be possible later using Switch Chip ACL rules or by another implementation."

sindy · Mon Jun 11, 2018 10:46 am

Strange - this is from Mikrotik support 3 weeks ago.
"At the beginning it is planned to have one s-tag for all c-tags on port.
Selective tagging might be possible later using Switch Chip ACL rules or by another implementation."

Yes, on port. But my suggestion uses two different ports, on one of them only the c-vlans 10,20,30 are permitted by the vlan-filtering rule, and on the other one only the c-vlans 11,21,31.

paulct · Mon Jun 11, 2018 11:00 am

But my suggestion uses two different ports, on one of them only the c-vlans 10,20,30 are permitted by the vlan-filtering rule, and on the other one only the c-vlans 11,21,31.

Ok which is not ideal in all cases, unless ALL your switches can do this.

e.g Switch A (some other brand or such - managed switch but no q-in-q) - one uplink to switch B (mikrotik) - and then to switch C (Mikrotik) - handover point.
So on switch B you wont be able to say c-tag 10-30 goes to s-tag 100 and c-tag 40-100 goes to s-tag 120.

But either way - nice to know Mikrotik supports it and maybe one day can do the above.

sindy · Mon Jun 11, 2018 12:17 pm

which is not ideal in all cases, unless ALL your switches can do this.

e.g Switch A (some other brand or such - managed switch but no q-in-q) - one uplink to switch B (mikrotik) - and then to switch C (Mikrotik) - handover point.
So on switch B you wont be able to say c-tag 10-30 goes to s-tag 100 and c-tag 40-100 goes to s-tag 120.

But either way - nice to know Mikrotik supports it and maybe one day can do the above.

Sorry, I didn't get your point here. How Mikrotik's "more native" support of this "selective tagging" than using the setup I've suggested would change the fact that the other vendor's equipment cannot do it? And, to extend my mental horizon, what would be the application scenario?

The usual application scenario is that the ISP provides L2 WAN service to several unrelated customers and uses s-vlans to isolate from one another the traffic of these customers, which comes as trunks of c-vlans. In such arrangement, there is little use for c-vlan to s-vlan mapping inside the ISP's network, the traffic is normally s-tagged on the border switch, port-based.

Already @CZFan's application scenario is quite far from typical, as he

uses c-vlans to isolate his customers from one another instead of s-vlans (but that's exactly what saves him from the need to use rc which would be needed to support vlan-filtering of s-vlans), and
hands over the result already s-tagged to his carrier ISP while the usual approach is that the carrier ISP s-tags the traffic on their own equipment.

Your scenario is even more unusual, as you take several c-vlans and want to map groups of them to different s-vlans, and on top of that you seem to want to change that mapping on an intermediate equipment.

Can you give a practical example where this would be purposeful?

paulct · Mon Jun 11, 2018 12:43 pm

which is not ideal in all cases, unless ALL your switches can do this.

e.g Switch A (some other brand or such - managed switch but no q-in-q) - one uplink to switch B (mikrotik) - and then to switch C (Mikrotik) - handover point.
So on switch B you wont be able to say c-tag 10-30 goes to s-tag 100 and c-tag 40-100 goes to s-tag 120.

But either way - nice to know Mikrotik supports it and maybe one day can do the above.
Sorry, I didn't get your point here. How Mikrotik's "more native" support of this "selective tagging" than using the setup I've suggested would change the fact that the other vendor's equipment cannot do it? And, to extend my mental horizon, what would be the application scenario?

The usual application scenario is that the ISP provides L2 WAN service to several unrelated customers and uses s-vlans to isolate from one another the traffic of these customers, which comes as trunks of c-vlans. In such arrangement, there is little use for c-vlan to s-vlan mapping inside the ISP's network, the traffic is normally s-tagged on the border switch, port-based.

Already @CZFan's application scenario is quite far from typical, as he

uses c-vlans to isolate his customers from one another instead of s-vlans (but that's exactly what saves him from the need to use rc which would be needed to support vlan-filtering of s-vlans), and

hands over the result already s-tagged to his carrier ISP while the usual approach is that the carrier ISP s-tags the traffic on their own equipment.

Your scenario is even more unusual, as you take several c-vlans and want to map groups of them to different s-vlans, and on top of that you seem to want to change that mapping on an intermediate equipment.

Can you give a practical example where this would be purposeful?

It's great that Mikrotik supports it, but there are scenarios where selective q-in-q in certain networks is needed. Not Mikrotik's fault as I do realise it is an unusual setup.

e.g say you have 100 buildings - and use a different brand of switch. Why? Well maybe until very recently Mikrotik had no multi ( > 24 port) SFP switches or 48 port ethernet switches. One would not want to replace them all, and they do not support q-in-q. One would simply tag the various ports and then install one "master" Mikrotik switch which can do selective q-in-q.

Why would you want to do this? Maybe to enable open access on your own network.

CZFan · Mon Jun 11, 2018 9:52 pm

Thank you sindy, all

CZFan · Sun Jun 17, 2018 8:35 pm

If I may throw in another curve ball, I am new to the "Nuts & Bolts" of Vlan, so please bear with me.

Have a MT CCR with ether1 as a routed port (Stand alone, not part of a bridge / switch group)
Attach Vlan 10 to ether1, and attach vlan's 20, 30 & 40 to vlan 10, creating QinQ

Ether2 - ethernth in a bridge
Configure bridge for Vlan filter=yes
In Bridge Vlan, ether2 to ether5 & bridge tagged for vlan-ids=20,30,40

Trying to get frames coming in tagged on ether2 - 5 with vlan id ether 20, or 30 or 40, tag must stay on frame and go out via relevant vlan 20, 30 or 40, tunneled into vlan 10 out on ether1?

Hope it makes sense

sindy · Sun Jun 17, 2018 8:48 pm

Drawings are better, but I assume you want ether1 to carry the QinQ frames where VID 10 is the outer one (service-vlan, or S-vlan, ethertype 0x88a8) and VID 20,30,40 are the inner ones (customer-vlan, C-vlan, ethertype 0x8100). If so, the method suggested above is still valid, just modify the picture (and configuration) from

      bridge-x
ether1===|---s-vlan50===ether4

to

     bridge-vlan
ether2===|
ether3===|---s-vlan10===ether1
ether4===|
ether5===|

CZFan · Sun Jun 17, 2018 10:16 pm

@sindy, thank you very much. Apologies, was in a rush, but you are 100% correct, drawings will make it more clear, so below drawing and explanation, also explaining how I see it, so please correct me if I am wrong.

Vlans coming into CCR already tagged for 20, 30 & 40.
CCR is configured:
- ether1 is stand alone, router port, not part of any switch group / bridge, used for routing to WAN.
--- on ether1 I have vlan 10 attached to ether1 and vlan's 20, 30, 40 attached to vlan 10 (QinQ Tunneling)
- ether 2 - ether4 configured as part of a bridge
--- under /bridge vlan, I have selected the bridge, with Vlan-Ids=20, 30, 40, tagged=bridge, ether2, ether3, ether4 and untagged=none

So if my understanding is correct, packets/frames will come into the CCR already tagged by CPE devices with relevant ISP Vlan ID, as they come into bridge via ether2, 3 & 4, they will remain tagged, go out of the bridge still tagged, will then be encapsualted with vlan 10, go accross WAN linek, get to Co-Loc, Vlan 10 gets stripped and based on the Vlan tag remaining, will be routed to relevant ISP.

I am of the understanding that if I do the above, it remains 802.1Q (ethertype 0x8100), should I enable the "use service tag", it will become 802.1ad (ethertype 0x88a8), is my understanding correct?
If so, will it work on 802.1Q?

QinQ VLAN.JPG

sindy · Mon Jun 18, 2018 12:05 am

I have not tried an actual QinQ, only "Qinad", but I don't see why it should not work with the outer tag being a Q one (0x8100, use-service-tag=no).

CZFan · Mon Jun 18, 2018 7:57 pm

I cant get above working, anyone that can offer help, please

Mon Jun 18, 2018 8:18 pm

I have not tried an actual QinQ, only "Qinad", but I don't see why it should not work with the outer tag being a Q one (0x8100, use-service-tag=no).

yes, in some cases Q-in.Q works, in other cases you have to do Q-in-ad

CZFan · Mon Jun 18, 2018 8:21 pm

I have not tried an actual QinQ, only "Qinad", but I don't see why it should not work with the outer tag being a Q one (0x8100, use-service-tag=no).
yes, in some cases Q-in.Q works, in other cases you have to do Q-in-ad

I will try that in lab quickly, thx

CZFan · Mon Jun 18, 2018 8:25 pm

Just to be clear what I am trying to achieve, I want tagged vlans coming into a bridge, that must then go out of a routed interface still tagged, the routed interface is not part of the bridge, is that possible?

Mon Jun 18, 2018 8:50 pm

Just to be clear what I am trying to achieve, I want tagged vlans coming into a bridge, that must then go out of a routed interface still tagged, the routed interface is not part of the bridge, is that possible?

i think one way to do it is:

ether going to collocation with vlan and vlan in vlan interfaces

ether going in the other direction with vlan interfaces

make a bridge for each tagged vlan and add to that bridge as ports corresponding, for example:

bridge-20: ports: vlan20 ether1. vlan 20 ether2, vlan 20 ether3

in that way for each vlan

sindy · Mon Jun 18, 2018 9:03 pm

Just to be clear what I am trying to achieve, I want tagged vlans coming into a bridge, that must then go out of a routed interface still tagged, the routed interface is not part of the bridge, is that possible?

Now wait a bit. So you want that a packet comes in via VLAN 20 on interface A, gets routed (i.e. not bridged) to VLAN20 on interface B? So the IP subnet living in VLAN 20 on interface A is a different one than the IP subnet living in VLAN 20 on interface B (let's leave QinQ aside for the moment)?

peson · Mon Jun 18, 2018 9:50 pm

I am treading in waters I have not done before and it is a semi live network, so I need to get my ducks in a row, below is what I need:

Cust 1 ---- C-Vlan 10 -----
\
Cust 2 ---- C-Vlan 20 -------\--- CCR1036 -- S-Vlan 50 ---- Co Loc for ISP's
/
Cust 3 ---- C-Vlan 30 ---- /

Customers coming in on their relevant Vlan's to the CCR1036, then encapsulate the C-Vlan's into one Service Vlan to the co location where the S-Vlan will be stripped again and then be routed accordingly to the relevant ISP based on original Vlan's.

I have read somewhere on the Wiki that only "initial" support for this is available on bridges from 6.43RC14 I think it was (can't find it now). I have some concerns running Release Candidate versions in production.

Is above possible at the moment without running RC version, if so, can you provide some guidance on how I can achieve this?

Thanking you in advance

Are the customers behind the same physical interface?
What's on the other end of the CCR, a Mikrotik, Cisco...?
The CCR only acts as a L2 bridge for customers VLANs and S-Vlan them upstream?

CZFan · Mon Jun 18, 2018 9:57 pm

@chechito, thx, but tried that, then I can get the the Vlan20 on the far side router, but no devices behind it

Here is exports of what I have done so far in my lab, maybe it will make more sense:

Router1:

# jan/02/1970 00:44:46 by RouterOS 6.42.3

#
# model = RouterBOARD 931-2nD

/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface bridge
add fast-forward=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=bridge-vlan20 vlan-id=20
add interface=ether1 name=vlan10 use-service-tag=yes vlan-id=10
add interface=vlan10 name=vlan10-vlan20 vlan-id=20
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 interface=ether2 pvid=20
/interface bridge vlan
add bridge=bridge1 tagged=bridge1 untagged=ether2 vlan-ids=20
/ip address
add address=10.0.0.1/24 interface=bridge1 network=10.0.0.0
add address=10.10.10.1/24 interface=vlan10 network=10.10.10.0
add address=10.10.20.1/24 interface=bridge-vlan20 network=10.10.20.0
/system routerboard settings
set silent-boot=no

Router2:

# jan/02/1970 00:45:18 by RouterOS 6.42.3

#
# model = 951Ui-2HnD

/interface bridge
add fast-forward=no name=bridge1 vlan-filtering=yes
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface vlan
add interface=bridge1 name=bridge-vlan20 vlan-id=20
add interface=ether1 name=vlan10 use-service-tag=yes vlan-id=10
add interface=vlan10 name=vlan10-vlan20 vlan-id=20
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 interface=ether2 pvid=20
/interface bridge vlan
add bridge=bridge1 tagged=bridge1 untagged=ether2 vlan-ids=20
/ip address
add address=10.0.0.2/24 interface=bridge1 network=10.0.0.0
add address=10.10.10.2/24 interface=vlan10 network=10.10.10.0
add address=10.10.20.2/24 interface=bridge-vlan20 network=10.10.20.0
/system routerboard settings
set silent-boot=no

peson · Mon Jun 18, 2018 10:30 pm

Those configs won't work for you.
Please, try to explain the whole chain of what you're trying to achieve.
Review my questions above.
Is it something like this:
Cust2 -- untagged -- CCR:ether2 -- some core -- tagged 20@ISP router

CZFan · Mon Jun 18, 2018 10:46 pm

Those configs won't work for you.
Please, try to explain the whole chain of what you're trying to achieve.
Review my questions above.
Is it something like this:
Cust2 -- untagged -- CCR:ether2 -- some core -- tagged 20@ISP router

@peson, did you read my post, viewtopic.php?f=2&t=135504&p=669035#p668784

sindy · Mon Jun 18, 2018 10:49 pm

@CZfan, can you respond to post #20?

CZFan · Mon Jun 18, 2018 10:58 pm

@CZfan, can you respond to post #20?

@sindy, my answer to that was post 22, with the lab config, I just tried with untagged, as I do not have equipment to provide tagged data

In a nutshell, from CPE comes tagged with vlan20 to CCR (Bridge) then out with Vlan20 still tagged, gets tunneld into vlan10 and out of ether1 to other side. Ether1 is not poart of bridge, but a stahndalone port on the CCR.

Make sense?

CZFan · Mon Jun 18, 2018 10:59 pm

@sindy, apologies, accidentally accepted your last post as solved, meant to click on quotation marks to quote your post

peson · Mon Jun 18, 2018 11:05 pm

Those configs won't work for you.
Please, try to explain the whole chain of what you're trying to achieve.
Review my questions above.
Is it something like this:
Cust2 -- untagged -- CCR:ether2 -- some core -- tagged 20@ISP router
@peson, did you read my post, viewtopic.php?f=2&t=135504&p=669035#p668784

Sorry, wasn't logged in the first time, so I couldn't see the drawing.
So it's a L2 configuration you need.
Is the Co-Loc an MT or some other equipment?

CZFan · Mon Jun 18, 2018 11:08 pm

@peson, Sorry, cant answer for equipment other side, I asked numerous occasions but still did not get answer, have meeting with collocation engineers in the morning at 10:00

peson · Mon Jun 18, 2018 11:13 pm

@peson, Sorry, cant answer for equipment other side, I asked numerous occasions but still did not get answer, have meeting with collocation engineers in the morning at 10:00

But they request you to have a s-vlan 10 carrying the customers inside with c-vlan 20, 30 and 40?

CZFan · Mon Jun 18, 2018 11:16 pm

@peson, correct, it is a deployment / project that i got involved late, asked info and got very little, i.e. network design, etc

peson · Mon Jun 18, 2018 11:21 pm

@peson, correct, it is a deployment / project that i got involved late, asked info and got very little, i.e. network design, etc

Do you need to interfere with the customers vlans on layer 3. or is it only L2 tunneling you need.?
Any queuing or shaping involved?

sindy · Mon Jun 18, 2018 11:26 pm

@sindy, my answer to that was post 22, with the lab config, I just tried with untagged, as I do not have equipment to provide tagged data
In a nutshell, from CPE comes tagged with vlan20 to CCR (Bridge) then out with Vlan20 still tagged, gets tunneld into vlan10 and out of ether1 to other side. Ether1 is not poart of bridge, but a stahndalone port on the CCR.
Make sense?

I must be missing some point. I don't understand what "routed interface" means, that's the first point.

When talking about QinQ (or "Qinad"), we normally stay in the L2 domain, no routing is related.

In one of my first posts, there is a Wireshark dissection of a frame created using the configuration I've suggested - the tagless side of a local /interface vlan has an IP configuration attached to it, the tagged side of that /interface vlan uses bridge-x as its carrier interface and has use-service-tag set to no, so it adds 802.1Q tags to the frames coming to its tagless side before forwarding them to the bridge. Then, the tagless side of another /interface vlan is made a member port of bridge-x, that /interface vlan uses ether1 as the carrier interface at its tagged side and has use-service-tag set to yes, so it takes already 802.1Q tagged frames from bridge-x and adds 802.1ad tags to them before sending them out via ether1.

I've dissected an ARP packet because I did not configure a second 'Tik the same way, so I could not generate any other than ARP traffic (I've pinged an address in the same subnet to which the IP address attached to the first /interface vlan belongs, so the router generated an ARP packet to determine that address, and that's the packet I've sniffed and dissected).

Neither of your configuration exports matches this, plus you have attached IP configuration to the /interface vlan with pvid=10.

So no, for me the configuration export in post #22 did not clarify your intention, and was not an answer to my question in post #20.

If the intention is to take frames already tagged with 802.1Q tag with VIDs 20,30,40 and just tag them one more time with VID 10 (no matter whether with an 802.1Q or 802.1ad tag) before sending them towards the colocation, my configurations from post #2 do exactly that; if the intention is different, describe how the frame should look when it comes in to the CCR from the left side and how the frame should look like when it goes out to the right side, and whether you expect it to be routed inside the CCR.

CZFan · Mon Jun 18, 2018 11:48 pm

@sindy, what I mean by "routed Interface"

Take any MT router, lets say Hap AC Lite, usually you will configure ether 2 - 5 as members of a bridge, then ether 1 will be your WAN port.
Now imagine you have a VoIP phone and your SIP provider is connected to your ISP, but on Vlan 20, but your ISP requires you to send that Vlan 20 accross your WAN port tunneled into Vlan 10.
So you configure on the VoIP phone itself that it must use Vlan 20, so the phone will come into your bridge on your Hap AC Lite as tagged with vlan 20, must then be encapsualted / tunneled into Vlan 10 and out on the WAN port to your ISP.
At your ISP, the ISP will then decapsulate / strip Vlan 10, which will leave only Vlan 20 on the packet. The ISP sees, ahhh, Vlan 20 belongs to SIP Provider 1, send the packet to them

sindy · Tue Jun 19, 2018 12:18 am

What you describe is still a mere bridging with QinQ and has nothing to do with routing, so my suggestion from post #2 should work.

Falling asleep, let's see what those guys tell you tomorrow at ten.

CZFan · Tue Jun 19, 2018 12:28 am

What you describe is still a mere bridging with QinQ and has nothing to do with routing, so my suggestion from post #2 should work.

Falling asleep, let's see what those guys tell you tomorrow at ten.

Thx sindy, yes, will revert back tomorrow, also falling asleep here

peson · Tue Jun 19, 2018 4:52 am

This might be helpful:

/interface bridge
add name=br-QinQ vlan-filtering=yes
/interface vlan
add interface=ether1 name=vl10-QinQ use-service-tag=yes vlan-id=10
/interface bridge port
add bridge=br-QinQ interface=vl10-QinQ
add bridge=br-QinQ interface=ether2
add bridge=br-QinQ interface=ether3
add bridge=br-QinQ interface=ether4
/interface bridge vlan
add bridge=br-QinQ tagged=vl10-QinQ,ether2,ether3,ether4 vlan-ids=20,30,40

Use horizon values if you need to block traffic between customers.
If you need L3 termination in the CCR you have to add the vlan interfaces (/interface vlan add...)and the bridge itself in tagged vlan list.

CZFan · Tue Jun 19, 2018 8:11 pm

@sindy, @peson, thx for your feedback so far.

Info received after meeting today, the other side is apparently a Cisco ASR1002,

Today we have cut over to an additional / new layer 2 fibre link so all is currently working via that link. I have been told the older / existing link is also suppose to be a layer 2 link, so I am not sure why they originally configured ether1 as a "routed / WAN" interface. From my understanding, with the info supplied now, it should be a lot easier, as I can configure the CCR as a switch, i.e. bridge all ports, then the vlan issue becomes a bit simpler.

I will test in my lab tonight and take from there

CZFan · Wed Jun 20, 2018 2:42 pm

@sindy, thank you, your solution in post #2 worked, the reason I could not get it to work was I used my youngest son's laptop on the other end of my lab setup. Seems the OS on that laptop is broken and I am unable to ping it, not even directly from the router it connected to, hence I thought QinQ tunneling was not working when I tried to ping it from my laptop via QinQ setup.

sindy · Wed Jun 20, 2018 3:33 pm

It may not even be broken, many embedded firewalls do not respond to pings by default in "unknown" networks. I was quite confident it did work as my test has confirmed it, so I was expecting some misconfiguration rather than a mistake of the concept.

But this @peson's remark is worth considering given the application case you've described:

Use horizon values if you need to block traffic between customers.

I mean, if the c-vlans are used to connect customers' networks to different VoIP providers which possibly use their own private address spaces to avoid NAT-related problems, it makes sense to permit only traffic between the VoIP provider's access point and the customers but not from one customer to another.

Off topic, two weeks ago I've obtained a pair of glasses after all and it does make a difference

CZFan · Wed Jun 20, 2018 5:44 pm

Yes, I should give credit where due, thank you @peson, will definitely implement "horizon" config.

@sindy, I have disabled firewall on his laptop, still could not ping it. Many games and crap on that laptop...

off topic, whose name did I spell wrong this time...

sindy · Wed Jun 20, 2018 6:12 pm

Off topic, no victims this time, I've just used this topic to deliver the update because it is yours and solved

CZFan · Wed Jun 20, 2018 7:14 pm

off topic, FYI, I am at the stage already where I have 2 pairs of glasses, one for every day use and another for reading, it definitely helps, but unfortunately not in all scenarios

@peson, I read up a bit more on the horizon (I just wonder why these things are not covered in training / certification classes) and yes, will definitely use it.

peson · Wed Jun 20, 2018 9:39 pm

Split horizon is covered in the MTCINE training (https://mikrotik.com/pdf/MTCINE_Outline.pdf)
I teaching it in the MTCNA classes I have since it's a very useful feature when configuring port isolation which is common used in ISPs networks.

Of topic:
I still have only one pair of glasses, but I need to take them off when reading

CZFan · Wed Jun 20, 2018 9:51 pm

Split horizon is covered in the MTCINE training (https://mikrotik.com/pdf/MTCINE_Outline.pdf)
I teaching it in the MTCNA classes I have since it's a very useful feature when configuring port isolation which is common used in ISPs networks.

Of topic:
I still have only one pair of glasses, but I need to take them off when reading

Aaahhhh, that's good to know, if all works out well, I will be attending MTCINE end July 2018. Sounds like an awesome course looking at outline, very excited

deepmedia · Fri Apr 12, 2019 1:16 pm

Hi there,

I'm looking for a qinad solution aswell. Currently I'm running 2 x CRS317 with a metro vlan provider in between them. There are several vlans on both sides that require to be connected by the s-vlan together. I'm running the vlan aware bridge configuration on both CRS'es. Some of the vlans are entering the device as tagged and some as untagged. The complex part is ether4. I thought it would be a good idea to create seperate bridges, but ether4 cant be attached to both bridges at the same time. See also attached image.

802.1ad-example.png

Anyone got a configuration suggestion / example?

sindy · Fri Apr 12, 2019 9:41 pm

I'm afraid none of the currently available Mikrotik products can fulfil your requirement completely in hardware without using crude hacks, at least because you need to add/remove two tags on the path between ether2 and ether4, whilst all the bridge implementations cannot add more than one tag on ingress and remove more than one tag on egress. So if you need to make use of the 10 Gbit/s bandwidth of the uplink, you may have to stack two CRS317, unless the partitioning of the switch chip and VLAN filtering is so good that you can connect together two ports of the same switch belonging to different partitions without creating a forwarding loop (you need per-VLAN forwarding tables in the switch chip as a minimum, plus you probably need that no VLAN ID is used for both a C-VLAN and an S-VLAN as the switch chip likely only uses the VID as an index to the table, not the tag type).

The topology would be the following:

ascii-art code

              S-bridge                     C-bridge A
                  ║                             ║
                  ║                             ║---access---
                  ║---access---~~~~~====trunk===║   PVID 10
      hybrid      ║  PVID=100        VID 10,20  ║
===VIDs 100,999===║                             ║===trunk====
     PVID=999     ║                             ║   VID 20
                  ║
                  ║                        C-bridge B
                  ║                             ║
                  ║---access---~~~~~====trunk===║===trunk====
                  ║  PVID=999          VID 30   ║   VID 30

C-bridge B is only necessary to prevent ingress of frames tagged with any other VID than 30 as S-bridge only cares about S-tags.

If you don't need the full 10 Gbit/s speed (actually even just 1 Gbit/s may be too much to expect), you can implement a similar topology using software bridges and see whether the CPU will cope with the traffic or not. In this case, instead of cables between ports, you would use /interface vlan:

/interface vlan
add name=vcable-a interface=s-bridge vlan-id=100 use-service-tag=yes
add name=vcable-b interface=s-bridge vlan-id=999 use-service-tag=yes

and make the tagless sides of these /interface vlan member ports of c-bridge-a and c-bridge-b respectively:

/interface bridge port
add bridge=c-bridge-a interface=vcable-a
add bridge=c-bridge-b interface=vcable-b

peson · Sat Apr 13, 2019 12:17 am

Hi there,

I'm looking for a qinad solution aswell. Currently I'm running 2 x CRS317 with a metro vlan provider in between them. There are several vlans on both sides that require to be connected by the s-vlan together. I'm running the vlan aware bridge configuration on both CRS'es. Some of the vlans are entering the device as tagged and some as untagged. The complex part is ether4. I thought it would be a good idea to create seperate bridges, but ether4 cant be attached to both bridges at the same time. See also attached image.

802.1ad-example.png

Anyone got a configuration suggestion / example?

Haven't looked into this in Mikrotik, but is tag stacking per port usable?
https://wiki.mikrotik.com/wiki/Manual:B ... g_Stacking
/Paul

sindy · Sat Apr 13, 2019 3:24 pm

is tag stacking per port usable?

I understand the description in the manual you refer to in such a way that tag-stacking=yes only makes the /interface bridge port handling ignore the already existing tags on the ingress frames even if the topmost ethertype of the ingress frame matches the ether-type of the bridge and always act as an access port, i.e. add another tag in front of the existing one on ingress. But still it is just a single tag to be added on ingress and removed on egress, not two. Whereas in his post above, @deepmedia asks for adding a C-tag 10 to tagless frames received at ether2 and then adding also an S-tag 100 before sending them out ether4, so two ingress crossings of bridge (or switch) border are required.

So a single bridge approach could only work if you could add two tags in a single ingress handling step, but the switch chip of the CRS317 doesn't provide such (rarely required) functionality in hardware, so even if it was implemented to the bridge, the throughput would still be limited by the CPU power and by the bandwidth of the internal connection between the switch part of the chip and the CPU's packet interface.

I suspect from the feature overview on the respective manual page that the switch chips used in CRS1xx/2xx can add both a C-tag and an S-tag in a single ingress step, but 1) I don't have a possibility to test this practically and 2) these devices have just up to two 10 Gbit/s ports so the bandwidth limitation might still affect @deepmedia's use case.

deepmedia · Tue Apr 16, 2019 2:27 am

Also on advice of MT support I decided to go with CVID tag stacking instead of c-vlan within s-vlan.

I build following config:

/interface bridge
add ingress-filtering=yes name=bridge vlan-filtering=yes

/interface vlan
add interface=sfp-sfpplus1 name=vlan-gs-ser vlan-id=309
add interface=bridge name=vlan-mgmt vlan-id=20

/interface bridge port
add bridge=bridge ingress-filtering=yes interface=vlan-gs-ser tag-stacking=yes
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=sfp-sfpplus1

/interface bridge vlan
add bridge=bridge comment=transport-gs-ser tagged=bridge,sfp-sfpplus1,vlan-gs-ser vlan-ids=309
add bridge=bridge comment=mgmt tagged=bridge,vlan-gs-ser,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,trunk-ccr2-crs2 vlan-ids= 20

Without tag-stacking option on the vlan-gs-ser port everything seem to work fine, but as soon as I enable tag stacking it got broken. Anyone got an idea?

peson · Tue Apr 16, 2019 8:51 am

Also on advice of MT support I decided to go with CVID tag stacking instead of c-vlan within s-vlan.

I build following config:
Code: Select all
/interface bridge
add ingress-filtering=yes name=bridge vlan-filtering=yes

/interface vlan
add interface=sfp-sfpplus1 name=vlan-gs-ser vlan-id=309
add interface=bridge name=vlan-mgmt vlan-id=20

/interface bridge port
add bridge=bridge ingress-filtering=yes interface=vlan-gs-ser tag-stacking=yes
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=sfp-sfpplus1

/interface bridge vlan
add bridge=bridge comment=transport-gs-ser tagged=bridge,sfp-sfpplus1,vlan-gs-ser vlan-ids=309
add bridge=bridge comment=mgmt tagged=bridge,vlan-gs-ser,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8,trunk-ccr2-crs2 vlan-ids= 20
Without tag-stacking option on the vlan-gs-ser port everything seem to work fine, but as soon as I enable tag stacking it got broken. Anyone got an idea?

I think you missundersstand the tag-stacking feature.
From the example on page: https://wiki.mikrotik.com/wiki/Manual:B ... g_Stacking
"What we want to achieve is that regardless what is being received on ether2 and ether3, a new VLAN tag will be added to encapsulate the traffic that is coming from those ports. What tag-stacking does is forces a new VLAN tag, so we can use this property to achieve our desired setup. We are going to be using the same configuration as in the Trunk/Access port setup, but with tag stacking enabled on the access ports:"
In your example, if 309 is the outer tag and sfp-sfpplus1 is the port to metro provider, it would be something like:

/interface bridge
add name=bridge vlan-filtering=yes ether-type=0x8100
/interface bridge port
add bridge=bridge interface=sfp-sfpplus1
add bridge=bridge interface=sfp-sfpplus6 tag-stacking=yes pvid=309 (outer tag)
add bridge=bridge interface=sfp-sfpplus7 tag-stacking=yes pvid=309 (outer tag)
add bridge=bridge interface=sfp-sfpplus8 tag-stacking=yes pvid=309 (outer tag)
...
/interface bridge vlan
add bridge=bridge tagged=sfp-sfpplus1 untagged=sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8 vlan-ids=309

In "/interface vlan" you only add the VLAN that needs to be processed by the CPU, not the switch chip.

deepmedia · Tue Apr 16, 2019 11:28 am

The main reason I didnt use pvid 309 on the endports is the fact we have untagged traffic that needs both an inner and outer vlan tag. Is there a way to double tag them? Or do you suggest to always tag all traffic on the Customers side?

sindy · Tue Apr 16, 2019 1:30 pm

That's what I've tried to explain before in post #49 - it makes no difference whether you place an S-tag over an existing C-tag or whether you nest two C-tags, the real problem is that only a single tag can be added to a frame at ingress and none at egress, and just a single tag can be removed from a frame at egress and none at ingress. So when a tagless frame comes in through ether2, you can tag it with C-VID 10, but all the way to the wire connected to ether4 there is no place where you could add the other tag with S-VID or C-VID 100, unless you cascade two bridges as in my drawing.

If you can make all the customers send you only C-tagged frames, you'll be in a better position because you'll only need to add the outer tags, so in this case tag-stacking=yes will ensure that the existing C-tags of the ingress frames will be ignored and new C-tags will be added on ports configured as "untagged". But if you set the ether-type=0x88a8 on the bridge, you don't even need the tag-stacking=yes setting because the C-tags (0x8100) of ingress frames will be ignored because the bridge won't recognize frames with ethertype 0x8100 as tagged ones.

deepmedia · Tue Apr 16, 2019 1:36 pm

Sindy, that's exactly what I was already thinking. 1st of all I will start by making all my traffic towards and from our customers tagged. We'll see if that will work out. Thanks!

Verstuurd vanaf mijn Pixel 3 met Tapatalk

deepmedia · Fri Jun 07, 2019 11:49 pm

To get a workaround I bought 2 additional CRS317's (I needed them anyway)

I built the following setup:

Untitled.png

The upper CRS is configured using the following config:

/interface bridge
add ingress-filtering=yes name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] speed=10Gbps
set [ find default-name=sfp-sfpplus16 ] speed=10Gbps
/interface bridge port
add bridge=bridge ingress-filtering=yes interface=sfp-sfpplus1 pvid=309 tag-stacking=yes
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=sfp-sfpplus16
add bridge=bridge ingress-filtering=yes interface=ether1
/interface bridge vlan
add bridge=bridge tagged=bridge,sfp-sfpplus16,ether1 vlan-ids=80
add bridge=bridge tagged=bridge untagged=sfp-sfpplus1 vlan-ids=10

The lower CRS got the following config::

/interface bridge
add ingress-filtering=yes name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] speed=10Gbps
set [ find default-name=sfp-sfpplus16 ] speed=10Gbps
/interface vlan
add interface=bridge name=vlan10 vlan-id=10
add interface=bridge name=vlan80 vlan-id=80
/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=sfp-sfpplus1
add bridge=bridge frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=sfp-sfpplus16
/interface bridge vlan
add bridge=bridge tagged=bridge,sfp-sfpplus1 vlan-ids=10
add bridge=bridge tagged=bridge,sfp-sfpplus16 vlan-ids=80
/ip address
add address=1.1.1.1/24 interface=vlan10 network=1.1.1.0
add address=2.2.2.2/24 interface=vlan80 network=2.2.2.0

Whenever all cables are connected only the double tagged traffic gets forwarded to the wireshark (see image)

allportsenabled.png

When disconnect sfp1 cable the single tagged traffic gets forwarded (see image)

disconnectcablesfp1.png

Is there any workaround you guys might think of? The only requirement is the fact that single and double tagged traffic needs to be on the same cable (because of the carrier who is providing these vlans)

Thanks for your suggestions.

sindy · Sat Jun 08, 2019 12:29 am

I'd have to see your setup hands-on (but I don't plan to drive by any time soon), but I think you are missing one basic thing about L2 - in addition to VLAN-tagged traffic, there are also VLAN-agnostic protocols running on switches, among which STP is the one which causes this issue. Whenever you connect two L2 devices together by more than a single logical link (a bond of several physical links behaves as a single logical link), you have to address the omission of the L2 protocol designers which is the absence of any TTL field in the L2 header. In another words, you have to prevent loops in the logical topology by dynamically disabling all logical links except one. So although you only permit one VLAN on the physical link between sfpplus1 and only permit another VLAN on the physical link between sfpplus16, the RSTP protocol which runs on the bridges by default still only permits one of these links to be active at a time whenever both are physically connected.

So setting protocol-mode at both bridges (upper and lower) to none might resolve your issue, but it may also cause a broadcast storm if eventually some frames leak the wrong way and loop back through the other link. So if this happens, you'll have to set up also split horizon, preventing any frame received on sfpplus1 from being sent out via sfpplus16 and vice versa, whereas ether4 on the upper CRS and any port except sfpplus1 and sfpplus16 on the lower CRS will be able to forward frames to/from both sfpplus1 and sfpplus16.

Other than that, I'm afraid that in order to make it work in both directions, the second row in /interface bridge vlan in the upper CRS configuration has to be changed to add bridge=bridge tagged=bridge untagged=sfp-sfpplus1 vlan-ids=309. The pvid parameter of /interface bridge port row controls the ingress handling; the position of the interface on the tagged or untagged list on the vlan-ids row in /interface bridge vlan controls the egress handling. For the upper CRS, VLAN 10 doesn't exist at all, it only knows about VLAN 309 and VLAN 80.

deepmedia · Sat Jun 08, 2019 8:27 am

Your suggestion turning off RSTP works... see image! Allthough I'm very excited to implement this on the network, on the other hand I am a little scared for broadcast storms as you suggested. Is there a way to verify / monitor this? And if this is the case, how to prevent it?

afterrstp-none.png

Oh and btw, In my previous post the last rule of the following config fell off by copy pasting, so current config about vlan statements is:

/interface bridge vlan
add bridge=bridge tagged=bridge,sfp-sfpplus16,ether1 vlan-ids=80
add bridge=bridge tagged=bridge untagged=sfp-sfpplus1 vlan-ids=10
add bridge=bridge tagged=bridge,ether1 vlan-ids=309

sindy · Sat Jun 08, 2019 12:06 pm

Your suggestion turning off RSTP works... see image! Allthough I'm very excited to implement this on the network, on the other hand I am a little scared for broadcast storms as you suggested. Is there a way to verify / monitor this? And if this is the case, how to prevent it?

Better to prevent it than to monitor

Check this.

Oh and btw, In my previous post the last rule of the following config fell off by copy pasting, so current config about vlan statements is:
Code: Select all
/interface bridge vlan
add bridge=bridge tagged=bridge,sfp-sfpplus16,ether1 vlan-ids=80
add bridge=bridge tagged=bridge untagged=sfp-sfpplus1 vlan-ids=10
add bridge=bridge tagged=bridge,ether1 vlan-ids=309

Replace the last two lines above by the single one I've suggested - add bridge=bridge tagged=bridge,ether1 untagged=sfp-sfpplus1 vlan-ids=309. As said, VLAN 10 is invisible to the upper CRS, it is just a payload inside VLAN 309 like any other.

deepmedia · Sat Jun 08, 2019 12:43 pm

In the configuration-to-be all ports will be added to a single bridge, with the protocol-mode set to none. No other bridges, no tagless or untagged (except for the stacked ones) will be accepted onto the switch. I am probably safe when all connections are managed and connected by myself

By the way: it servers as a core/metro switch to connect the other datacentre to the primary one.

Current config of the upper one:

/interface bridge vlan
add bridge=bridge tagged=bridge,sfp-sfpplus16,ether1 vlan-ids=80
add bridge=bridge tagged=bridge,ether1 untagged=sfp-sfpplus1 vlan-ids=309

Still working flawless. No loops (traffic still 0bps and cpu ilde)

Thanks for your support, appreciated greatly!

sindy · Sat Jun 08, 2019 3:13 pm

No other bridges, no tagless or untagged (except for the stacked ones) will be accepted onto the switch. I am probably safe when all connections are managed and connected by myself

Well, I'm not really sure how exactly the switch chip handles ingress frames which are not Ethernet II but 802.2 (i.e. the first two bytes following the MAC addresses represent frame size, not content type), i.e. whether it deems them "untagged" or rather "neither tagged nor untagged". And there are not just STP frames, some switch vendors use their proprietary frames for loop detection, which may take the long path provider_switch -> upperCRS.ether4 -> upperCRS.sfpplus1 -> lowerCRS.sfpplus1 -> lowerCRS.sfpplus16 -> upperCRS.sfpplus16 -> upperCRS.ether4 -> provider_switch and make the provider switch shut down the port. In worse case, the adjacent switch would not detect a loop and disable the port but frames which escape the ingress filtering would circulate there forever (as they have no TTL field to be used to count hops), gradually seizing all the bandwidth. Candidates are not just broadcast/multicast frames but also frames towards unicast MAC addresses from which an ingress frame never comes so the switch cannot associate them with a port.

So to prevent this from happening, the port isolation needs to be set at the more distant switch from the source (so against loop detection/eternally circulating frames coming from the provider switch, the lowerCRS needs to have sfpplus1 and sfpplus16 isolated from each other; against loop detection/eternally circulating frames coming from the customer switch, the upperCRS needs to have sfpplus1 and sfpplus16 isolated from each other).

CZFan · Sat Jun 08, 2019 8:20 pm

@deepmedia
As a side note, I assume the 1.1.1.1, etc addresses are loopback addresses, anyway, personally I will stay away from them as they are routable on internet

deepmedia · Thu Jun 13, 2019 4:07 pm

Those we're used in a disconnected environment, no worries

nichky · Wed Apr 27, 2022 8:39 am

So you want it like this, right?
Code: Select all
Ethernet II, Src: Routerbo_78:5a:37 (64:d1:54:78:4a:37), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
IEEE 802.1ad, ID: 50
    000. .... .... .... = Priority: 0
    ...0 .... .... .... = DEI: 0
    .... 0000 0011 0010 = ID: 50
    Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 10
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = DEI: Ineligible
    .... 0000 0000 1010 = ID: 10
    Type: ARP (0x0806)
Address Resolution Protocol (request)
I haven't tried yet, but from what the wiki says, the "initial support of QinQ" is relevant to using vlan-filtering on frames with 802.1ad tags on a bridge, so unless you need a bridge hosting several s-vlans, you don't need that feature. So if you want to avoid rc in production and you have enough CPU for adding the s-tag using /interface vlan, it is possible to do it the following way:
Code: Select all
                 bridge-x
ether1---tag(c10)===|
ether2---tag(c20)===|---s-vlan50===ether4
ether3---tag(c30)===|
(--- ... tagless side, === ... tagged side)
i.e.
Code: Select all
/interface vlan
add interface=ether4 name=s-vlan50 use-service-tag=yes vlan-id=50

/interface bridge
add name=bridge-x vlan-filtering=yes

/interface bridge port
add bridge=bridge-x interface=s-vlan50
add bridge=bridge-x interface=ether1 pvid=10
add bridge=bridge-x interface=ether2 pvid=20
add bridge=bridge-x interface=ether3 pvid=30

/interface bridge vlan
add bridge=bridge-x tagged=bridge-x,s-vlan50 untagged=ether1 vlan-ids=10
add bridge=bridge-x tagged=bridge-x,s-vlan50 untagged=ether2 vlan-ids=20
add bridge=bridge-x tagged=bridge-x,s-vlan50 untagged=ether3 vlan-ids=30
I haven't tested it completely, so be careful.

Of course if the CCR already gets c10,c20,c30 tagged in a single trunk, all you need is
Code: Select all
      bridge-x
ether1===|---s-vlan50===ether4
so all shrinks down to just
Code: Select all
/interface vlan
add interface=ether4 name=s-vlan50 use-service-tag=yes vlan-id=50

/interface bridge
add name=bridge-x vlan-filtering=yes

/interface bridge port
add bridge=bridge-x interface=s-vlan50
add bridge=bridge-x interface=ether1

/interface bridge vlan
add bridge=bridge-x tagged=bridge-x,ether1,s-vlan50 vlan-ids=10,20,30

Hi sindy,

for some reason this method does not work for v7, but it does well for v6.
Have you noticed that?

sindy · Wed Apr 27, 2022 9:33 am

I only touch the 7.x softly so far, no deep diving. So no, I haven't.

nichky · Sat Jul 16, 2022 3:05 pm

@MikroTik - can you please let us know why the config that @sindy presented here doesn't work on v7, are you aware of this?

peson · Sat Jul 16, 2022 5:40 pm

@MikroTik - can you please let us know why the config that @sindy presented here doesn't work on v7, are you aware of this?

Have you tried the VLAN stacking feature on bridge ports?
Take a look at: https://help.mikrotik.com/docs/display/ ... agStacking

nichky · Sun Jul 17, 2022 7:22 am

@MikroTik - can you please let us know why the config that @sindy presented here doesn't work on v7, are you aware of this?
Have you tried the VLAN stacking feature on bridge ports?
Take a look at: https://help.mikrotik.com/docs/display/ ... agStacking

yes, but that not even close to qinq on v6 , correct me if i'm wrong

sindy · Sun Jul 17, 2022 7:56 am

Tag stacking is QinQ (or Qinad, depending on the setup).

The reason I've provided my "manual implementation" was that @CZFan has started this topic when tag stacking was only available in beta. And then @deepmedia wanted to add both tags on a single pass (i.e. a frame that was completely tagless on ingress to have two tags at egress, which cannot be achieved using a single bridge with tag stacking enabled, let alone in hardware).

So depending on your particular needs, tag stacking may or may not be sufficient.

Which says nothing about why the "manual implementation" doesn't work in V7.

nichky · Sun Jul 17, 2022 10:19 am

Tag stacking is QinQ (or Qinad, depending on the setup).

The reason I've provided my "manual implementation" was that @CZFan has started this topic when tag stacking was only available in beta. And then @deepmedia wanted to add both tags on a single pass (i.e. a frame that was completely tagless on ingress to have two tags at egress, which cannot be achieved using a single bridge with tag stacking enabled, let alone in hardware).

So depending on your particular needs, tag stacking may or may not be sufficient.

Which says nothing about why the "manual implementation" doesn't work in V7.

help me @sindy , acording from the link

https://help.mikrotik.com/docs/display/ ... agStacking

how the conf would be on the router, the exsisting config is for the switch

nichky · Sun Jul 17, 2022 10:27 am

@sindy

im using Tag stacking this way:

that issues that i'm having if i can't add Mgm vlan on R3 becouse of Tag stacking

sindy · Sun Jul 17, 2022 11:04 am

how the conf would be on the router, the exsisting config is for the switch

Bridge configuration is identical on a switch and on a router, the difference is performance - if you only configure L2 features that the switch chip used in the device supports, you can get wirespeed bridging, otherwise you get a software one, which usually severely degrades throughput on switches as their CPUs are dimensioned for device management, not for traffic forwarding.

im using Tag stacking this way:
...
that issues that i'm having if i can't add Mgm vlan on R3 becouse of Tag stacking

From what I can see, ether2 on R2 adds an 802.1Q tag with VID 100 on ingress and strips it on egress no matter what other tags are eventually present in the frame. On R3, tag stacking is not used at all. But the following questions remain open:

How are R2 and R3 interconnected? R2/ether1 is connected to R3/ether1? If so, are they connected directly or via some network that manipulates VLAN tags?
Which of the VLANs on R3 should be the management one, VLAN 10 or VLAN 20?
Do you want to access the management VLAN of R3 from R2?

Provide a full description or diagram to explain what your goal is. Do not refer to some other descriptions or diagrams, describe your particular network setup.

nichky · Sun Jul 17, 2022 11:16 am

which ethernet or goes where you can see from the topology.

add bridge=br-vlan interface=eth2 pvid=10
That is untagged vlan to the clients

Everything works fine, just im wondering how can i add Mngm vlan on R3 since im using Tag stacking on R2 (interface pointing ro R3)

nichky · Sun Jul 17, 2022 11:36 am

put this way, i need something similar to your "manual implementation" on v7.

peson · Sun Jul 17, 2022 11:44 am

which ethernet or goes where you can see from the topology.

add bridge=br-vlan interface=eth2 pvid=10
That is untagged vlan to the clients

Everything works fine, just im wondering how can i add Mngm vlan on R3 since im using Tag stacking on R2 (interface pointing ro R3)

Some more information of your setup is needed.
What VLID is the mgmt VLAN?
From where do you want to access the mgmt VLAN?
Provide bridge configuration of the routers. (/interface/bridge export)

sindy · Sun Jul 17, 2022 12:05 pm

So in your topology, R2 acts as a provider switch in the QinQ concept - whatever it receives on ether2 gets an outer 802.1Q tag with VID 100 and is forwarded double-tagged to ether1. So if you add a management VLAN to the bridge at R3, it will be treated the same way like any other VLAN at R2, i.e. it will also get the outer tag.

You can prevent frames belonging to the management VLAN from ever reaching R2 by excluding eth1 from the tagged list for the management VLAN at R3. But if the intention is the reverse one, i.e. to manage R2 using the management VLAN from R3, you have to attach a stack of /interface vlan to the bridge at R2. Assuming the management VLAN ID is 333, it would look as follows:

/interface/bridge/vlan
set [find bridge=br-vlan vlan-ids=100] tagged=br-vlan,ether1
/interface/vlan
add interface=br-vlan vlan-id=100 name=br-vlan.100
add interface=br-vlan.100 vlan-id=333 name=br-vlan.100.333

Now you can attach the management address from the subnet in R3's VLAN 333 to R2.

Just tested that on a pair of CHRs running ROS 7.3, works fine.

peson · Sun Jul 17, 2022 12:14 pm

@sindy "But if..." Is why I'm asking for more information to provide help.
Assumptions make misunderstandings.

sindy · Sun Jul 17, 2022 12:25 pm

Assumptions make misunderstandings.

I fully agree with you, and I've attempted twice to get a more precise description. However, quite some years in the customer support field have taught me that sometimes it is faster to suggest options for the other guy to choose from than to just keep asking for more details. And in this particular case, knowing that @nichky is down under and hence his time is 6 to 8 hours ahead of ours, I've chosen this approach

nichky · Sun Jul 17, 2022 1:15 pm

btw there is the config on R1. vl100 is carries vl10 and vl20.

/interface vlan
add interface=e2vl100 name=e2vl10 vlan-id=10
add interface=e2vl100 name=e2vl20 vlan-id=20
add interface=ether2 name=e2vl100 vlan-id=100

[admin@R1] /interface vlan> /interface bridge export
/interface bridge
add name=br-vlan vlan-filtering=yes
/interface bridge port
add bridge=br-vlan interface=ether2
/interface bridge vlan
add bridge=br-vlan tagged=br-vlan,ether2 vlan-ids=100

@sindy ill try that

peson · Sun Jul 17, 2022 1:47 pm

@nichky
Why do you need QinQ in this configuration?
Does vlid 10 and 20 interfere with other vlans in R2?
From this drawing and the configurations, why not use 10 and 20 as tagged vlan from R3 to R1?

nichky · Sun Jul 17, 2022 1:48 pm

@sindy

unfortunately that doesn't work becouse of that:

R2:
/interface bridge port
add bridge=br-vlan interface=ether1
add bridge=br-vlan interface=ether2 pvid=100 tag-stacking=yes
/interface bridge vlan
add bridge=br-vlan tagged=ether1 untagged=ether2 vlan-ids=100

i can make it work as u sugester , but than i will lose qinq to the cl1 and cl2

u got all the connfig to all routers

sindy · Sun Jul 17, 2022 1:55 pm

i can make it work as u sugester , but than i will lose qinq to the cl1 and cl2

Why should you lose QinQ? What I suggest doesn't touch the path from ether2 to ether1 at all, it simply adds local access to VLAN 100 (by making the "router-facing port" of bridge br-vlan a member of the outermost VLAN 100), and then it adds local access to the inner VLAN. What you've posted now does not match what I've suggested.

nichky · Sun Jul 17, 2022 1:55 pm

i'm not using vlan 100 on R3, i don't need that, even i do still not working, see that issush that im having?

u can modigicate as much as u want, im happy to try

nichky · Sun Jul 17, 2022 2:01 pm

@sindy

what u are saying is absolutely right ,my apologies

sindy · Sun Jul 17, 2022 2:17 pm

Maybe better to call... the whole setup is a QinQ one at R2, where VLAN 100 only exists; R3 doesn't know anything about VLAN 100 or about QinQ, whereas R2 basically knows nothing about VLANS 10 and 20 and just forwards them transparently, encapsulated into VLAN 100.

Now I am again not sure what you actually want to achieve. Is the intention to have access to management of both R2 and R3 via the same subnet (and VLAN) or something else?

peson · Sun Jul 17, 2022 2:28 pm

Now I am again not sure what you actually want to achieve. Is the intention to have access to management of both R2 and R3 via the same subnet (and VLAN) or something else?

Agree,
@nichky
Please explain.
Does c1 and c2 need to send both untagged and tagged packets?
Does VLAN 10 and 20 interfere with other VLANs in R2?
Is R1 just a router?
From/to where do you want management?

nichky · Sun Jul 17, 2022 2:29 pm

i want to expand Mgm vlan to all the routers, whether i can get access from cl1 and cl2 i didn't try yet,but in my case i dont need it.
Just i need from Rx to get access to Rz that is all i want.
And also i needed qinq + Mng-vlanx

nichky · Sun Jul 17, 2022 2:43 pm

Agree,
@nichky
Please explain.

Does c1 and c2 need to send both untagged and tagged packets?

NO, they are just only the end users

Does VLAN 10 and 20 interfere with other VLANs in R2?

vlan10 & vlan20 are part of qinq

Is R1 just a router?

yes one of the router in my cloud

From/to where do you want management?

Mgm vlan-x between the Rx
[/quote]

peson · Sun Jul 17, 2022 2:46 pm

Agree,
@nichky
Please explain.
Does c1 and c2 need to send both untagged and tagged packets?
NO, they are just only the end users

Does VLAN 10 and 20 interfere with other VLANs in R2?
vlan10 & vlan20 are part of qinq
Is R1 just a router?
yes one of the router in my cloud
From/to where do you want management?
Mgm vlan-x between the Rx

Why do you need QinQ between R2 and R1?

sindy · Sun Jul 17, 2022 2:47 pm

Okay. Each bridge only understands a single VLAN tag in a frame - the outermost one. So br-vlan on R2 only knows about VLAN 100, the inner tags of VLAN 10 and VLAN 20 are invisible to it. That's because tag stacking is enabled at ether2, i.e. even if an already tagged frame (with VLAN 10 or VLAN 20) arrives from the wire, the port treats it as if it was tagless and adds the tag with VID 100 to it. This is the basic functionality of R2 you want, so you cannot change it.

Hence to get access to the tagless contents of a frame that R3 has sent tagged over the wire, you have to peel not only the tag the R3 has added, but also (first!) the tag that R2 has tagged. Which is what my setup does. What happens when your management PC connected to an access port to management VLAN at R3 sends a tagless frame:

R3 tags that frame with the ID of the management VLAN (as the corresponding pvid is configured on the access port)
R3 sends the frame with a single tag (mgmt vid) via ether1
R2 receives the frame and attaches an outer tag (VID 100) to it because pvid of ether2 is 100 and tag-stacking is set to yes
the frame gets to the "router-facing port" of br-vlan, tagged with outer tag 100 and inner tag mgmt vid
the /interface vlan named br-vlan.100 in my configuration, attached to the "router-facing port" of br-vlan, peels off the outer tag
the /interface vlan named br-vlan.100.333 in my configuration, attached to the tagless end of /interface vlan named br-vlan.100, peels of the remaining (inner) tag and makes the untagged contents of the packet available to the router at its own tagless end.

nichky · Sun Jul 17, 2022 2:48 pm

Why do you need QinQ between R2 and R1?

there is no qinq between R1 and R2, where have u seen that?

peson · Sun Jul 17, 2022 2:52 pm

Why do you need QinQ between R2 and R1?
there is no qinq between R1 and R2, where have u seen that?

R1 extracts VLAN 100 to get access to VLAN 10 and 20, that's QinQ.
The QinQ starts at R2 and extracts in R1

nichky · Sun Jul 17, 2022 2:58 pm

R1 extracts VLAN 100 to get access to VLAN 10 and 20, that's QinQ.
The QinQ starts at R2 and extracts in R1

see the config on R1, vl100 is carries vl10 and vl20
and on R2 or all the way to the end users im tagging only vl100

nichky · Sun Jul 17, 2022 3:05 pm

@sindy has explained everything on #89.

ill say against, everything was working just fine (qinq) , i only wasn't sure how to do Mng-vlan on the Router which i have already untag vlan-x

peson · Sun Jul 17, 2022 3:07 pm

R1 extracts VLAN 100 to get access to VLAN 10 and 20, that's QinQ.
The QinQ starts at R2 and extracts in R1

see the config on R1, vl100 is carries vl10 and vl20
and on R2 or all the way to the end users im tagging only vl100

That's why I say that the QinQ starts in R2 and exracts in R1
All incoming traffic to ether2 on R2 will have a tag added (100) and all outgoing traffic on ether2 will have vlan 100 tag removed.

nichky · Sun Jul 17, 2022 3:24 pm

That's why I say that the QinQ starts in R2 and exracts in R1

other way around, i had to upload all in once , i thought that u can assume.

peson · Sun Jul 17, 2022 3:36 pm

That's why I say that the QinQ starts in R2 and exracts in R1
other way around, i had to upload all in once , i thought that u can assume.

Ok, this depends from where you look.
I still wonder why you need to put vlid10 and 20 inside vlid 100 between R1 and R2?
I fully agree to the expanation in #89, but is it really needed?
Your configuration, as presented, do not need to have QinQ between R1 and R2, this is cost of extra cpu cycles.
If this is just a lab testing for a live environment with other devices included, then I understand.
I'm aking this questions to help you in the right direction.

nichky · Sun Jul 17, 2022 3:54 pm

I still wonder why you need to put vlid10 and 20 inside vlid 100 between R1 and R2?

that why u miss the point, that is between R2 and R3
fyi Rx==vl100(vl10&vl20)====>R2==vl100==>R3=vl10=>clients

I fully agree to the expanation in #89, but is it really needed?

agre, u got millions of options

I'm aking this questions to help you in the right direction.

Thanks, its all done

JBLTelecom · Tue Jul 09, 2024 4:02 pm

So you want it like this, right?
Code: Select all
Ethernet II, Src: Routerbo_78:5a:37 (64:d1:54:78:4a:37), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
IEEE 802.1ad, ID: 50
    000. .... .... .... = Priority: 0
    ...0 .... .... .... = DEI: 0
    .... 0000 0011 0010 = ID: 50
    Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 10
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = DEI: Ineligible
    .... 0000 0000 1010 = ID: 10
    Type: ARP (0x0806)
Address Resolution Protocol (request)
I haven't tried yet, but from what the wiki says, the "initial support of QinQ" is relevant to using vlan-filtering on frames with 802.1ad tags on a bridge, so unless you need a bridge hosting several s-vlans, you don't need that feature. So if you want to avoid rc in production and you have enough CPU for adding the s-tag using /interface vlan, it is possible to do it the following way:
Code: Select all
                 bridge-x
ether1---tag(c10)===|
ether2---tag(c20)===|---s-vlan50===ether4
ether3---tag(c30)===|
(--- ... tagless side, === ... tagged side)
i.e.
Code: Select all
/interface vlan
add interface=ether4 name=s-vlan50 use-service-tag=yes vlan-id=50

/interface bridge
add name=bridge-x vlan-filtering=yes

/interface bridge port
add bridge=bridge-x interface=s-vlan50
add bridge=bridge-x interface=ether1 pvid=10
add bridge=bridge-x interface=ether2 pvid=20
add bridge=bridge-x interface=ether3 pvid=30

/interface bridge vlan
add bridge=bridge-x tagged=bridge-x,s-vlan50 untagged=ether1 vlan-ids=10
add bridge=bridge-x tagged=bridge-x,s-vlan50 untagged=ether2 vlan-ids=20
add bridge=bridge-x tagged=bridge-x,s-vlan50 untagged=ether3 vlan-ids=30
I haven't tested it completely, so be careful.

Of course if the CCR already gets c10,c20,c30 tagged in a single trunk, all you need is
Code: Select all
      bridge-x
ether1===|---s-vlan50===ether4
so all shrinks down to just
Code: Select all
/interface vlan
add interface=ether4 name=s-vlan50 use-service-tag=yes vlan-id=50

/interface bridge
add name=bridge-x vlan-filtering=yes

/interface bridge port
add bridge=bridge-x interface=s-vlan50
add bridge=bridge-x interface=ether1

/interface bridge vlan
add bridge=bridge-x tagged=bridge-x,ether1,s-vlan50 vlan-ids=10,20,30

@Sindy I need your guidance to resolve this. We are trying to implement a QnQ on our network due to the exhaustion of Vlans. This is what we want to achieve, there are three package routers.

>Router A
C-Vlans 167,169,177,185 and 238
S-Vlan 4091
>Router B
C-Vlans 240, 768, 850, 920, and 1021
S-Vlan 4092
>Router C
C-Vlans 230, 546, 650, 680 and 1242
S-Vlan 4093
The C-Vlans are sub-interfaces of the S-Vlans in the respective routers. The aim is to pass the S-Vlans across the intermediary switches de-encapsulate the S-Vlans on a router at the station then send the C-Vlans to a Switch and tag them on the right ports. The C-Vlans of the clients will be tagged on the rightful port with which the client AP is connected.

The reason for this architecture is that the routers at the client end don't have VLAN capabilities hence taking this route. Please find the diagram of the setup below.

Your guidance and input is highly welcomed. Decapsulating the S-Vlans on the router is where I am having the issue.

sindy · Tue Jul 09, 2024 8:50 pm

Decapsulating the S-Vlans on the router is where I am having the issue.

So let me rephrase that - on Router@Station, you want to "unpack" the C-VLANs from the respective S-VLANs from Switch B and forward them to Switch@Station on a common trunk without encapsulation into any S-VLAN. This can indeed be done, but bear in mind that the C-VLANs must be unique (which seems to be the case) and, more important, think about the capacity of the MAC mapping tables on the switches - it might be better to use VXLANs to encapsulate the C-VLANs as that way, the MAC addresses that live inside the C-VLANs will not be visible to switches A and B. With VLAN tag stacking (QinQ), all three switches will have to be able to deal with all the MAC addresses in all VLANs, otherwise they would fall into broadcast mode.

As for the decapsulation - you need to attach three /interface/vlan (subinterfaces if you want) with use-service-tag=yes and vlan-id=409x to ether1:
/interface/vlan
add use-service-tag=yes vlan-id=4091 interface=ether1 name=ether1.s4091
add use-service-tag=yes vlan-id=4091 interface=ether1 name=ether1.s4092
add use-service-tag=yes vlan-id=4091 interface=ether1 name=ether1.s4093

Next, make their tagless ends member ports of a bridge with vlan-filtering=yes whose other member port is ether2, and set ingress-filtering=yes and frame-types=admit-only-vlan-tagged on all of them:
/interface/bridge/port
add bridge=bridge interface=ether2 ingress-filtering=yes frame-types=admit-only-vlan-tagged
add bridge=bridge interface=ether1.s4091 ingress-filtering=yes frame-types=admit-only-vlan-tagged
add bridge=bridge interface=ether1.s4092 ingress-filtering=yes frame-types=admit-only-vlan-tagged
add bridge=bridge interface=ether1.s4093 ingress-filtering=yes frame-types=admit-only-vlan-tagged

Lastly, in the /interface/bridge/vlan table, set the appropriate mapping of the C-VLANs to the four member ports of the bridge:
/interface/bridge/vlan
add bridge=bridge vlan-ids=167,169,177,185,238 tagged=ether2,ether1.s4091
add bridge=bridge vlan-ids=1240,768,850,920,1021 tagged=ether2,ether1.s4092
add bridge=bridge vlan-ids=230,546,650,680,1242 tagged=ether2,ether1.s4093

That should be it.

JBLTelecom · Wed Jul 10, 2024 5:22 pm

Thanks for the guide @Sindy but still unable to get traffic across to a client end. Please find below the configs for the devices participating in network topology starting from Router A to the Switch @ Station.

# jan/02/1970 03:06:11 by RouterOS 7.9
# software id = W5KW-24NI
#
# model = RB760iGS
# serial number = E1F10E894A35
/interface ethernet
set [ find default-name=ether5 ] poe-out=off
/interface vlan
add interface=ether5 name=Ses_CnC use-service-tag=yes vlan-id=4091
add interface=Ses_CnC name=\
    act_0167_Aburime_Natty_Favour_AF0077_SES_Oyarifa_Access_Radio vlan-id=167
add interface=Ses_CnC name=\
    act_0169_Suleman_Useni_Sani_SU0217_SES_Dodowa_Acc_rad vlan-id=169
add interface=Ses_CnC name=\
    act_0177_Nosa_Aiamouwosa_NA3559_SES_Abokobi_Access_Radio vlan-id=177
add interface=Ses_CnC name=\
    act_0185_Bettey_Acquah_BA7743_SES_Oyibi_Access_Radio vlan-id=185
add interface=Ses_CnC name=\
    act_0195_Emeka_Samuel_Okpara_ES3657_SES_Teiman_Madina_Access_Radio \
    use-service-tag=yes vlan-id=195
add interface=Ses_CnC name="act_0238_Micheal_Elvis_Omoegbe_ME5880_SES_Pantang-\
    Village-Road_Access_Radio" vlan-id=238
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/port
set 0 name=serial0
/ip address
add address=20.30.0.121/30 comment=Aburime_Natty_Favour_AF0077_0167 \
    interface=act_0167_Aburime_Natty_Favour_AF0077_SES_Oyarifa_Access_Radio \
    network=20.30.0.120
add address=20.30.5.69/30 comment=Micheal_Elvis_Omoegbe_ME5880_0238 \
    interface="act_0238_Micheal_Elvis_Omoegbe_ME5880_SES_Pantang-Village-Road_\
    Access_Radio" network=20.30.5.68
add address=20.30.21.1/30 comment=Nosa_Aiamouwosa_NA3559_0177 interface=\
    act_0177_Nosa_Aiamouwosa_NA3559_SES_Abokobi_Access_Radio network=\
    20.30.21.0
add address=20.30.36.81/30 comment=Emeka_Samuel_Okpara_ES3657_0195 interface=\
    act_0195_Emeka_Samuel_Okpara_ES3657_SES_Teiman_Madina_Access_Radio \
    network=20.30.36.80
add address=20.30.0.217/30 comment=Suleman_Useni_Sani_SU0217_0169 interface=\
    act_0169_Suleman_Useni_Sani_SU0217_SES_Dodowa_Acc_rad network=20.30.0.216
/system identity
set name="Router A"
/system note
set show-at-login=no
/tool romon
set enabled=yes

The Config on Switch A

# 1970-01-04 00:15:00 by RouterOS 7.15.1
# software id = 9MX6-K0FH
#
# model = RB951Ui-2HnD
# serial number = HF509E77WXX
/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface ethernet
set [ find default-name=ether5 ] poe-out=off
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether1
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether2
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether4
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=bridge1 comment=S-Vlan-4091 tagged=ether3,ether4 vlan-ids=4091
add bridge=bridge1 comment=S-Vlan-4092 tagged=ether2,ether4 vlan-ids=4092
add bridge=bridge1 comment=S-Vlan-4093 tagged=ether1,ether4 vlan-ids=4093
/system identity
set name="Switch A"
/system note
set show-at-login=no
/tool romon
set enabled=yes]

Config on Switch B

# 1970-01-02 02:13:34 by RouterOS 7.15.2
# software id = FJ40-CZ0I
#
# model = RB951Ui-2HnD
# serial number = 4AC704B7B29E
/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether1
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether2
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether4
/interface bridge vlan
add bridge=bridge1 comment=S-Vlan-4091 tagged=ether1,ether2 vlan-ids=4091
add bridge=bridge1 comment=S-Vlan-4092 tagged=ether1,ether2 vlan-ids=4092
add bridge=bridge1 comment=S-Vlan-4093 tagged=ether1,ether2 vlan-ids=4093
/system identity
set name="Switch B"
/system note
set show-at-login=no
/tool romon
set enabled=yes

Config on Router @ Station

# 1970-01-05 04:17:05 by RouterOS 7.15.2
# software id = F84T-IXUP
#
# model = RB760iGS
# serial number = HDJ08XEY2ZP
/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge1 port-cost-mode=short \
    vlan-filtering=yes
/interface ethernet
set [ find default-name=ether5 ] poe-out=off
/interface vlan
add interface=ether1 name=vlan.s4091 use-service-tag=yes vlan-id=4091
add interface=vlan.s4091 name=vlan167 vlan-id=167
add interface=vlan.s4091 name=vlan169 vlan-id=169
add interface=vlan.s4091 name=vlan177 vlan-id=177
add interface=vlan.s4091 name=vlan185 vlan-id=185
add interface=vlan.s4091 name=vlan238 vlan-id=238
add interface=ether1 name=vlans.4092 use-service-tag=yes vlan-id=4092
add interface=ether1 name=vlans.4093 use-service-tag=yes vlan-id=4093
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/ip smb users
set [ find default=yes ] disabled=yes
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2
/routing ospf area
add disabled=yes instance=default-v2 name=backbone-v2
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether2
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=vlan.s4091
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=vlans.4092
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=vlans.4093
/ip firewall connection tracking
set udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=bridge1 comment=S-Vlan-4091 tagged=ether2,vlan.s4091 vlan-ids=\
    167,169,177,185,238
add bridge=bridge1 comment=S-Vlan-4092 tagged=ether2,vlans.4092 vlan-ids=\
    768,850,920,1021,1240
add bridge=bridge1 comment=S-Vlan-4093 tagged=ether2,vlans.4093 vlan-ids=\
    230,546,650,680,1242
/interface ovpn-server server
set auth=sha1,md5
/ip address
add address=10.20.0.94/30 interface=*D network=10.20.0.92
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=10.20.0.93
/ip smb shares
set [ find default=yes ] directory=/flash/pub
/routing bfd configuration
add disabled=no interfaces=all min-rx=200ms min-tx=200ms multiplier=5
/system identity
set name="Router @ Station"
/system note
set show-at-login=no
/tool romon
set enabled=yes

Config on Switch @ Station

# 1970-01-02 01:34:41 by RouterOS 7.14.3
# software id = YKQ7-P7I2
#
# model = RB750Gr3
# serial number = CC230F36139A
/interface bridge
add name=bridge1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether1
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether2
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether4
/interface bridge vlan
add bridge=bridge1 comment="AP - Sector A" tagged=ether1,ether3 vlan-ids=\
    167,177,850,920,1021
add bridge=bridge1 comment="AP - Sector B" tagged=ether1,ether2 vlan-ids=\
    169,185,238,650,1242
add bridge=bridge1 comment="AP - Sector C" tagged=ether1,ether4 vlan-ids=\
    230,546,680,768,1240
/system identity
set name="Switch @ Station"
/system note
set show-at-login=no
/tool romon
set enabled=yes

Please review the configs above and let me know what i am doing wrong. Currently i am just using the vlans on Router A. As in both the S-Vlans and the C-Vlans

sindy · Wed Jul 10, 2024 5:53 pm

On both switch A and switch B:
/interface bridge set bridge1 ether-type=0x88a8
The thing is that the frames arrive S-tagged (i.e. the VLAN tag starts with ethertype 0x88a8) but since the bridge looks for C-tags (ethertype 0x8100) by default, it handles them as untagged ones and therefore it does not let them in.

JBLTelecom · Wed Jul 10, 2024 7:27 pm

@Sindy configuration was affected but having the same issue traffic doesn't transverse to the client end.

Switch A

# 1970-01-04 02:42:32 by RouterOS 7.15.1
# software id = 9MX6-K0FH
#
# model = RB951Ui-2HnD
# serial number = HF509E77WXX
/interface bridge
add ether-type=0x88a8 frame-types=admit-only-vlan-tagged name=bridge1 \
    vlan-filtering=yes
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface ethernet
set [ find default-name=ether5 ] poe-out=off
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether1
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether2
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether4
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=bridge1 comment=S-Vlan-4091 tagged=ether3,ether4 vlan-ids=4091
add bridge=bridge1 comment=S-Vlan-4092 tagged=ether2,ether4 vlan-ids=4092
add bridge=bridge1 comment=S-Vlan-4093 tagged=ether1,ether4 vlan-ids=4093
/system identity
set name="Switch A"
/system note
set show-at-login=no
/tool romon
set enabled=yes

Switch B

# 1970-01-02 04:39:14 by RouterOS 7.15.2
# software id = FJ40-CZ0I
#
# model = RB951Ui-2HnD
# serial number = 4AC704B7B29E
/interface bridge
add ether-type=0x88a8 frame-types=admit-only-vlan-tagged name=bridge1 \
    vlan-filtering=yes
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether1
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether2
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether4
/interface bridge vlan
add bridge=bridge1 comment=S-Vlan-4091 tagged=ether1,ether2 vlan-ids=4091
add bridge=bridge1 comment=S-Vlan-4092 tagged=ether1,ether2 vlan-ids=4092
add bridge=bridge1 comment=S-Vlan-4093 tagged=ether1,ether2 vlan-ids=4093
/system identity
set name="Switch B"
/system note
set show-at-login=no
/tool romon
set enabled=yes

sindy · Wed Jul 10, 2024 7:37 pm

Set hw=no on all rows of the /interface bridge port table of all devices - it should happen automatically on these devices when vlan-filtering is set to yes but just for the case. Then, run /tool sniffer quick mac-address=xx:xx:xx:xx:xx:xx, matching on a known MAC address xx:xx:xx:xx:xx:xx, to see how far the frames get. I suppose the "tower" end is where wireless PPPoE clients are connected and "Router A, B, C" is where you hand their traffic over to PPPoE servers?

JBLTelecom · Wed Jul 10, 2024 9:13 pm

The result of sniffing from Router A to the Mac address (C4:AD:34:F6:5D:0D) on the client device interface with a VLAN ID of 169.

sindy · Wed Jul 10, 2024 9:30 pm

OK, and? This shows that various frames from that MAC address did make to Router A. Where in the network is C4:AD:34:F6:5D:0D connected? All of those frames are broadcast or multicast ones. So maybe you should try pinging 20.30.0.217 from 20.30.0.218 and sniff on all the devices between them to see how far the ARP requests and responses, and possibly echo requests and responses, get?

JBLTelecom · Fri Jul 12, 2024 10:55 pm

The Mac address C4:AD:34:F6:5D:0D is the terminating interface on the client router. I have tried recommendations but still experiencing the same Issue. packets don't traverse to the client router. Find attached the Wireshark capture of the sniffing test done on Router A to the client end (20.30.0.218).

[img

Wireshark.PNG

][/img]

sindy · Sat Jul 13, 2024 1:50 pm

You apparently did not get my idea. Running /tool sniffer quick mac-address=xx:xx:xx:xx:xx:xx directly on each device, without any additional filter conditions, will show you the complete path the frames from or to that MAC address pass on the device - physical port(s), bridge(s), and VLAN interface(s). In normal case, you should see the frame to arrive in via one physical interface, pass through the various internal interfaces, and finally leave via another physical interface. If this is not the case, seeing the last interface the frame did pass through is the hint where exactly to look into the configuration for issues (or the device is an endpoint in the communictaion so it is correct that only one physical interface is used).

I did not see anything wrong in the configs you posted after you've fixed the ether-type on the bridges, that's why I wanted to understand where the frames get lost. As you haven't shown the filter you have used when sniffing on Router A, it is not possible to conclude whether the ARP frame generated by Router A itself on interface act_0169_Suleman_Useni_Sani_SU0217_SES_Dodowa_Acc_rad did not make it even to the underlying interface, which is Ses_CnC, or whether it just could not be sniffed there because the filter has prevented that. Plus if you have actually connected your PC running Wireshark to ether5 rather than sniffing on the Mikrotik itself into a file and opening that file in Wireshark afterwards, which seems to be the case, the VLAN tags cannot be seen in the Wireshark capture because Windows networking drivers strip any VLAN tags from incoming packets. So the ARP frame you can see here may well have the stacked VLAN tags on the wire but Wireshark won't see that if fed via Windows networking.

JBLTelecom · Tue Jul 16, 2024 8:57 pm

@Sindy find attached the various sniffs on all the devices between the connected client device (C4:AD:34:F6:5D:0D)

Router A Sniff.

Switch A Sniff.

Switch B Sniff

Router @ Station Sniff

Switch @ Station Sniff

sindy · Tue Jul 16, 2024 9:52 pm

Maybe easier to copy-paste that as text, to avoid these weird issues with posting images...

JBLTelecom · Tue Jul 16, 2024 10:05 pm

sindy · Tue Jul 16, 2024 10:21 pm

Well, the results of the sniffing show that the IPv4 and IPv6 MNDP packets (UDP to port 5678) from the client device C4:AD:34:F6:5D:0D did make it all the way to Router A alright, and as far as I can see, with properly stacked tags. There is not much reason why the reverse direction should be an issue, but nevertheless, pinging from the client device to something that should be reachable via Router A should generate frames in both directions. If we can see ping requests to come from the client device but no ping responses, the next phase of analysis will focus on for routing at Router A.

JBLTelecom · Wed Jul 17, 2024 1:33 pm

Routing from Router A, there is no much going on in Router A. Per the configs as shown below Router A has nothing different to the router @ Station.

# 1970-01-05 06:17:58 by RouterOS 7.15.2
# software id = W5KW-24NI
#
# model = RB760iGS
# serial number = E1F10E894A35
/interface vlan
add interface=ether5 name=Ses_CnC use-service-tag=yes vlan-id=4091
add disabled=yes interface=Ses_CnC name=\
    act_0167_Aburime_Natty_Favour_AF0077_SES_Oyarifa_Access_Radio vlan-id=167
add interface=Ses_CnC name=\
    act_0169_Suleman_Useni_Sani_SU0217_SES_Dodowa_Acc_rad vlan-id=169
add disabled=yes interface=Ses_CnC name=\
    act_0177_Nosa_Aiamouwosa_NA3559_SES_Abokobi_Access_Radio vlan-id=177
add disabled=yes interface=Ses_CnC name=\
    act_0185_Bettey_Acquah_BA7743_SES_Oyibi_Access_Radio vlan-id=185
add disabled=yes interface=Ses_CnC name=\
    act_0195_Emeka_Samuel_Okpara_ES3657_SES_Teiman_Madina_Access_Radio \
    use-service-tag=yes vlan-id=195
add disabled=yes interface=Ses_CnC name="act_0238_Micheal_Elvis_Omoegbe_ME5880\
    _SES_Pantang-Village-Road_Access_Radio" vlan-id=238	
/ip address
add address=20.30.0.121/30 comment=Aburime_Natty_Favour_AF0077_0167 \
    interface=act_0167_Aburime_Natty_Favour_AF0077_SES_Oyarifa_Access_Radio \
    network=20.30.0.120
add address=20.30.5.69/30 comment=Micheal_Elvis_Omoegbe_ME5880_0238 \
    interface="act_0238_Micheal_Elvis_Omoegbe_ME5880_SES_Pantang-Village-Road_\
    Access_Radio" network=20.30.5.68
add address=20.30.21.1/30 comment=Nosa_Aiamouwosa_NA3559_0177 interface=\
    act_0177_Nosa_Aiamouwosa_NA3559_SES_Abokobi_Access_Radio network=\
    20.30.21.0
add address=20.30.36.81/30 comment=Emeka_Samuel_Okpara_ES3657_0195 interface=\
    act_0195_Emeka_Samuel_Okpara_ES3657_SES_Teiman_Madina_Access_Radio \
    network=20.30.36.80
add address=20.30.0.217/30 comment=Suleman_Useni_Sani_SU0217_0169 interface=\
    act_0169_Suleman_Useni_Sani_SU0217_SES_Dodowa_Acc_rad network=20.30.0.216
/system identity
set name="Router A"
/system note
set show-at-login=no
/tool romon
set enabled=yes

That is basically the configs on router A

sindy · Wed Jul 17, 2024 1:42 pm

What I had in mind was that I could not see anything wrong even in the sniffs you've provided yesterday, so I suspect the issue you observe ("traffic not getting through") is related to something else than bridge&vlan configuration.

So since the two routers (A and Station) share a /30 subnet in C-VLAN 169, I would first ping one from the other within that subnet; if the ping would show timeouts rather than responses, I would sniff again, starting from the device being the ping responder, to see why the pings get no response.

If the pings get responded, we know that the L2 path is working in both directions and we may dive deeper into what "traffic not getting through" means in detail, i.e. sniff while attempting to push that very traffic that is not getting through.

JBLTelecom · Wed Jul 17, 2024 3:49 pm

@Sindy configuring the C-VLAN 169 on the Router @ Station produces a ping response with a sniff as attached below,

Ping 4RM Rter@Stn.PNG

The above sniff shows that the layer 2 config from Switch A through to Router@Station is okay if so what might i be missing on Router@Station and the Switch@Station.

Thanks very very much for your guidance and help i do appreciate it,

sindy · Wed Jul 17, 2024 4:46 pm

So you have disabled 20.30.0.218/30 on the customer router (or maybe not) and put it up on Router@Station instead, and then the ping from Router A got responded, is that correct?

If so:
first, I have missed that vlan-filtering is set to no on switch1. In RouterOS 6, it only means that everything in /interface bridge vlan and the all the VLAN-related stuff in /interface bridge port is ignored, i.e. the traffic is not filtered per VLAN ID and port and tagging&untagging is not performed on bridge/switch ports. Since you only need the tagged frames to pass between the Ethernet ports unchanged, that's not an issue. But maybe in RouterOS 7 the default configuration of the switch chip is different, so try setting vlan-filtering to yes. I do realize that doing so will make the traffic flow through the CPU and thus restrict the throughput, but for the diagnostics it is a necessary step I am afraid.

Then, disable (or remove) the 20.30.0.218/30 on Router@Station, and try pinging that addres from Router A again. If it still fails, add the following configuration to Switch@Station:
/interface vlan add interface=bridge1 vlan-id=169 name=bridge1.169
/interface bridge vlan set [find where vlan-ids~"169"] tagged=ether1,ether2,bridge1
/ip address add address=20.30.0.218/30 interface=bridge1.169
Then try pinging 20.30.0.218 from Router A and sniffing again.

If pinging while the IP address is up on the Switch@Station works as well, remove the 20.30.0.218/30 from Switch@Station again, and instead of an ICMP ping, try an ARP "ping" from Router A to 20.30.0.218 (which is normally the address of a client device if I get it right):
:ping 20.30.0.218 arp=yes interface=act_0169_Suleman_Useni_Sani_SU0217_SES_Dodowa_Acc_rad

The thing is that the firewall on the customer's router may block pings by mistake, but blocking ARP requires more qualification from the customer than messing up the IP firewall.

Other than that, I hope that the mapping between Routers A,B,C and Sectors A,B,C on Station is indeed not indended to be 1:1, i.e. that it is correct that C-VLAN 169 is served by Router A but connected to Sector B at the Station end?

And yet another question, totally unrelated to the "traffic not passing through" issue - your configuration suggests that you spend 4 public addresses to provide a single one to a customer, is that indeed the case?

JBLTelecom · Wed Jul 17, 2024 7:51 pm

So you have disabled 20.30.0.218/30 on the customer router (or maybe not) and put it up on Router@Station instead, and then the ping from Router A got responded, is that correct?

Yes,

If so:
first, I have missed that vlan-filtering is set to no on switch1. In RouterOS 6, it only means that everything in /interface bridge vlan and the all the VLAN-related stuff in /interface bridge port is ignored, i.e. the traffic is not filtered per VLAN ID and port and tagging&untagging is not performed on bridge/switch ports. Since you only need the tagged frames to pass between the Ethernet ports unchanged, that's not an issue. But maybe in RouterOS 7 the default configuration of the switch chip is different, so try setting vlan-filtering to yes. I do realize that doing so will make the traffic flow through the CPU and thus restrict the throughput, but for the diagnostics it is a necessary step I am afraid.

Switch A has its bridge vlan filtering set to yes

Then, disable (or remove) the 20.30.0.218/30 on Router@Station, and try pinging that addres from Router A again. If it still fails, add the following configuration to Switch@Station:
/interface vlan add interface=bridge1 vlan-id=169 name=bridge1.169
/interface bridge vlan set [find where vlan-ids~"169"] tagged=ether1,ether2,bridge1
/ip address add address=20.30.0.218/30 interface=bridge1.169
Then try pinging 20.30.0.218 from Router A and sniffing again.

If pinging while the IP address is up on the Switch@Station works as well, remove the 20.30.0.218/30 from Switch@Station again, and instead of an ICMP ping, try an ARP "ping" from Router A to 20.30.0.218 (which is normally the address of a client device if I get it right):
:ping 20.30.0.218 arp=yes interface=act_0169_Suleman_Useni_Sani_SU0217_SES_Dodowa_Acc_rad

Configuration is done on the switch at station but ping wasn't successful.

The thing is that the firewall on the customer's router may block pings by mistake, but blocking ARP requires more qualification from the customer than messing up the IP firewall.

The firewall on the client router has been deleted.

Other than that, I hope that the mapping between Routers A,B,C and Sectors A,B,C on Station is indeed not indended to be 1:1, i.e. that it is correct that C-VLAN 169 is served by Router A but connected to Sector B at the Station end?

The C-Vlans can be tagged on any AP or port on the switch at Station depending on the AP client is connected.

And yet another question, totally unrelated to the "traffic not passing through" issue - your configuration suggests that you spend 4 public addresses to provide a single one to a customer, is that indeed the case?

No Doing CGNAT on each router.

sindy · Wed Jul 17, 2024 8:37 pm

I have missed that vlan-filtering is set to no on switch1. ...
Switch A has its bridge vlan filtering set to yes

Sorry, instead of "switch1" it should have read "Switch@Station"

If it still fails, add the following configuration to Switch@Station:
/interface vlan add interface=bridge1 vlan-id=169 name=bridge1.169
/interface bridge vlan set [find where vlan-ids~"169"] tagged=ether1,ether2,bridge1
/ip address add address=20.30.0.218/30 interface=bridge1.169
Then try pinging 20.30.0.218 from Router A and sniffing again.
...
Configuration is done on the switch at station but ping wasn't successful.

OK, so that localizes the issue somewhere between Router@Station and Switch@Station. Since I've mislead you regarding where to look, you probably haven't enabled vlan-filtering on Switch@Station yet? If that's the case, can you enable it and repeat the previous step (keeping 20.30.0.218 only on the customer device) and if it fails, repeat this one (putting the address up on Switch@Station)? If that also fails, sniff at Switch@Station to see whether the request comes in from Router@Station and how is it encapsulated.

it is correct that C-VLAN 169 is served by Router A but connected to Sector B at the Station end?
The C-Vlans can be tagged on any AP or port on the switch at Station depending on the AP client is connected.

OK, let me rephrase it - is the client device with address 20.30.0.218 indeed connected via sector B?

your configuration suggests that you spend 4 public addresses to provide a single one to a customer, is that indeed the case?
No Doing CGNAT on each router.

So what makes you use the (Microsoft!) public addresses as the /30 ones?

JBLTelecom · Wed Jul 17, 2024 10:15 pm

Switch A has its bridge vlan filtering set to yes
Sorry, instead of "switch1" it should have read "Switch@Station"

Configuration is done on the switch at station but ping wasn't successful.

OK, so that localizes the issue somewhere between Router@Station and Switch@Station. Since I've mislead you regarding where to look, you probably haven't enabled vlan-filtering on Switch@Station yet? If that's the case, can you enable it and repeat the previous step (keeping 20.30.0.218 only on the customer device) and if it fails, repeat this one (putting the address up on Switch@Station)? If that also fails, sniff at Switch@Station to see whether the request comes in from Router@Station and how is it encapsulated.
Sniffing was done on switch@Station and there was no request from Router@Station. No mac trace to router@Station as shown below with sniff.
Switch@StationSniff.PNG

it is correct that C-VLAN 169 is served by Router A but connected to Sector B at the Station end?
The C-Vlans can be tagged on any AP or port on the switch at Station depending on the AP client is connected.

OK, let me rephrase it - is the client device with address 20.30.0.218 indeed connected via sector B?

Yes the client is connected via sector B on port 2 on the switch@Station

your configuration suggests that you spend 4 public addresses to provide a single one to a customer, is that indeed the case?
No Doing CGNAT on each router.

So what makes you use the (Microsoft!) public addresses as the /30 ones?

We are just using that ip as our internal IP, since it won't be available on the internet via our end there won't be any problem or?

sindy · Wed Jul 17, 2024 10:53 pm

Sniffing was done on switch@Station and there was no request from Router@Station. No mac trace to router@Station as shown below with sniff.
Switch@StationSniff.PNG

It becomes a mystery... in the previous sniffs, the frames from C4:AD:34:F6:5D:0D/20.30.0.218 did pass from ether2@Switch@Station through ether1@Switch@Station to ether2@router@station, so why don't the frames in the opposite direction pass the reverse way?

What does /ip arp print where address=20.30.0.218 show at Router A? The train of thought is the following: in order that Router A could send a ping packet to 20.30.0.218, it must first determine its MAC address as the route at Router A has the interface act_0169_Suleman_Useni_Sani_SU0217_SES_Dodowa_Acc_rad as the gateway for 20.30.0.216/30. To determine the MAC address, it must use an ARP request saying "who has 20.30.0.218, tell 20.30.0.217" which is sent to MAC address ff:ff:ff:ff:ff:ff so you wouldn't see it if the sniff output if you filtered by MAC address C4:AD:34:F6:5D:0D, but your screenshot above shows that you did not use any filter at all. While sniffing at both Router@Station and Switch@Station and pinging 20.30.0.218 from Router A, can you use filter mac-protocol=!35007 to prevent the heavy traffic (that I guess to be CAPsMANv3) from obfuscating the traffic we are really interested in?

We are just using that ip as our internal IP, since it won't be available on the internet via our end there won't be any problem or?

The only problem will arise if your client will need to connect to a real server listening on that address, as it will land on your Router A instead. At the moment 20.30.0.217 does not seem to be even routable from the internet, but it may change in future.

JBLTelecom · Thu Jul 18, 2024 5:24 pm

Sniffing was done on switch@Station and there was no request from Router@Station. No mac trace to router@Station as shown below with sniff.
Switch@StationSniff.PNG
It becomes a mystery... in the previous sniffs, the frames from C4:AD:34:F6:5D:0D/20.30.0.218 did pass from ether2@Switch@Station through ether1@Switch@Station to ether2@router@station, so why don't the frames in the opposite direction pass the reverse way?
What does /ip arp print where address=20.30.0.218 show at Router A? The train of thought is the following: in order that Router A could send a ping packet to 20.30.0.218, it must first determine its MAC address as the route at Router A has the interface act_0169_Suleman_Useni_Sani_SU0217_SES_Dodowa_Acc_rad as the gateway for 20.30.0.216/30. To determine the MAC address, it must use an ARP request saying "who has 20.30.0.218, tell 20.30.0.217" which is sent to MAC address ff:ff:ff:ff:ff:ff so you wouldn't see it if the sniff output if you filtered by MAC address C4:AD:34:F6:5D:0D, but your screenshot above shows that you did not use any filter at all. While sniffing at both Router@Station and Switch@Station and pinging 20.30.0.218 from Router A, can you use filter mac-protocol=!35007 to prevent the heavy traffic (that I guess to be CAPsMANv3) from obfuscating the traffic we are really interested in?
/ip arp print where address=20.30.0.218 returns nothing as shown below on RouterA

IParpPrint.PNG
/ip arp print where address=20.30.0.218 with pings on RouterA

IParpSniff.PNG
Find below sniff on Router@Station.

Router@StationSniff.PNG

Find the sniff on Switch@Station below.

Switch@StationSniff.PNG

We are just using that ip as our internal IP, since it won't be available on the internet via our end there won't be any problem or?
The only problem will arise if your client will need to connect to a real server listening on that address, as it will land on your Router A instead. At the moment 20.30.0.217 does not seem to be even routable from the internet, but it may change in future.

As stated it's not routable on the internet hence there wont be any issues secondly we don't advertise this to any of our ISPs so i think we should be okay.

sindy · Thu Jul 18, 2024 5:56 pm

Find below sniff on Router@Station.
Find the sniff on Switch@Station below.

What does /interface/bridge/port/monitor [find] show on Router@Station and on Switch@Station? It looks as if some other path between these two devices, not shown on the diagram, exists in the network, and RSTP has chosen to cut the loop by blocking traffic on ether2@Router@Station.

it's not routable on the internet

Today it is not, tomorrow it may be

It is assigned to Microsoft and it is their decision whether and when they start actually using it.

we don't advertise this to any of our ISPs

Not advertising them definitely prevents you from "breaking the internet" (as in "draining the traffic towards these addresses from nearby network regions to yourself"), that's good. But once those addresses come into use, your customers will be unable to talk to them even if you don't advertise them to the internet - for your customers, you are the default route.

JBLTelecom · Thu Jul 18, 2024 8:07 pm

Find below sniff on Router@Station.
Find the sniff on Switch@Station below.

What does /interface/bridge/port/monitor [find] show on Router@Station and on Switch@Station? It looks as if some other path between these two devices, not shown on the diagram, exists in the network, and RSTP has chosen to cut the loop by blocking traffic on ether2@Router@Station.

Port Monitor on Router@Station below.

Router@StationPortMonitor.PNG

Port monitor on Switch@Station Below

Switch@StatiionPortMonitor.PNG

it's not routable on the internet
Today it is not, tomorrow it may be It is assigned to Microsoft and it is their decision whether and when they start actually using it.

Indeed, we will change it but not now.

Not advertising them definitely prevents you from "breaking the internet" (as in "draining the traffic towards these addresses from nearby network regions to yourself"), and that's good. But once those addresses come into use, your customers will be unable to talk to them even if you don't advertise them to the Internet - for your customers, you are the default route.

We will find a way as the aim is to introduce IPv6 in our network. That is the plan so this will curb or prevent any unforeseeable issues.

sindy · Thu Jul 18, 2024 9:23 pm

So both ports are in forwarding state, there is no bridge filter in either configuration, but nevertheless the ARP requests do not make it from Router@Station to Switch@Station. Incredible. Please use /interface bridge port set [find interface=ether2] hw=no on Router@Station, and /interface bridge port set [find] hw=no on Switch@Station, then start sniffing with just mac-protocol=!35007 at both at almost the same time, and ping 20.30.0.218 from Router A. There used to be some issues with sniffing in ROS 6 when hw was set to yes.

And give me /ip neighbor print detail from both Router@Station and Switch@Station, please.

sindy · Fri Jul 19, 2024 12:52 pm

I did not see it as important, and the output of /interface bridge port monitor suggests the same, but maybe it actually does have an impact? viewtopic.php?p=1086400#p1086400

On Router@Station, port-cost-mode is set to short, try changing it to the default long.

JBLTelecom · Fri Jul 19, 2024 2:21 pm

So both ports are in forwarding state, there is no bridge filter in either configuration, but nevertheless the ARP requests do not make it from Router@Station to Switch@Station. Incredible. Please use /interface bridge port set [find interface=ether2] hw=no on Router@Station, and /interface bridge port set [find] hw=no on Switch@Station, then start sniffing with just mac-protocol=!35007 at both at almost the same time, and ping 20.30.0.218 from Router A. There used to be some issues with sniffing in ROS 6 when hw was set to yes.

And give me /ip neighbor print detail from both Router@Station and Switch@Station, please.

Please are you saying the above is not important with your response today?

sindy · Fri Jul 19, 2024 2:32 pm

Please are you saying the above is not important with your response today?

If changing the port-cost-mode to long solves the issue, it is not, otherwise the suggestions from previous post stay important.

JBLTelecom · Fri Jul 19, 2024 3:16 pm

Please are you saying the above is not important with your response today?
If changing the port-cost-mode to long solves the issue, it is not, otherwise the suggestions from previous post stay important.

The port-cost mode did not solve the issue. Please find below screenshots of /ip neighbor print on the Router@Station and Switch@Station.

Router@Station IP Neighbor Print below.

Router@StationIPNeig.PNG

Switch@Station IP Neighbor Print below.

Switch@StationIPNei.PNG

sindy · Fri Jul 19, 2024 4:03 pm

What looks suspicious to me is that whilst the neighbor information row for Station@Switch at Router@Station says interface-name=bridge1/ether1, at Switch@Station, the row for Router@Station says just interface-name=ether2, without mentioning bridge1. The settings of neighbor discovery at Router@Station say discover-interface-list=!dynamic, which means that the bridge should be indicated in the LLDP messages as well. But it may be a glitch of 7.15.2 - I could test 7.14.3 as the recipient but I cannot easily connect any 7.15.2 to it to see the complete chain.

This somehow points in the same direction like the previous sniff, indicating that the ARP packet to ff:ff:ff:ff:ff:ff did not leave through ether2 - that's why I have asked you for sniffing again with hw set to no on the relevant interfaces.

Or, if it is a test setup as the name "Client Router" suggests, try rebooting the Router@Station - if that helps, there is a bug

JBLTelecom · Fri Jul 19, 2024 9:21 pm

What looks suspicious to me is that whilst the neighbor information row for Station@Switch at Router@Station says interface-name=bridge1/ether1, at Switch@Station, the row for Router@Station says just interface-name=ether2, without mentioning bridge1. The settings of neighbor discovery at Router@Station say discover-interface-list=!dynamic, which means that the bridge should be indicated in the LLDP messages as well. But it may be a glitch of 7.15.2 - I could test 7.14.3 as the recipient but I cannot easily connect any 7.15.2 to it to see the complete chain.

This somehow points in the same direction like the previous sniff, indicating that the ARP packet to ff:ff:ff:ff:ff:ff did not leave through ether2 - that's why I have asked you for sniffing again with hw set to no on the relevant interfaces.

Or, if it is a test setup as the name "Client Router" suggests, try rebooting the Router@Station - if that helps, there is a bug

Reboot did not solve the issue. Are we saying there is a bug in the RouterOS 7.15.2? It's the field setup which is being tested in-house.

sindy · Fri Jul 19, 2024 10:55 pm

Reboot did not solve the issue. Are we saying there is a bug in the RouterOS 7.15.2?

I am not as audacious as to specify a particular ROS version affected. Since the reboot has not helped, please do the sniffing with the settings as required in my last post related to the sniffing setup. Without that, we cannot be sure that the ARP requests indeed do not leave through ether2 of Router@Station.

It's the field setup which is being tested in-house.

What I had in mind was that you could afford to reboot Router@Station without affecting real customers, which you have just confirmed

JBLTelecom · Fri Jul 19, 2024 11:16 pm

Reboot did not solve the issue. Are we saying there is a bug in the RouterOS 7.15.2?
I am not as audacious as to specify a particular ROS version affected. Since the reboot has not helped, please do the sniffing with the settings as required in my last post related to the sniffing setup. Without that, we cannot be sure that the ARP requests indeed do not leave through ether2 of Router@Station.

It's the field setup which is being tested in-house.
What I had in mind was that you could afford to reboot Router@Station without affecting real customers, which you have just confirmed

So both ports are in forwarding state, there is no bridge filter in either configuration, but nevertheless the ARP requests do not make it from Router@Station to Switch@Station. Incredible. Please use /interface bridge port set [find interface=ether2] hw=no on Router@Station, and /interface bridge port set [find] hw=no on Switch@Station, then start sniffing with just mac-protocol=!35007 at both at almost the same time, and ping 20.30.0.218 from Router A. There used to be some issues with sniffing in ROS 6 when hw was set to yes.

And give me /ip neighbor print detail from both Router@Station and Switch@Station, please.

Please are you referring to the above post?

sindy · Fri Jul 19, 2024 11:18 pm

Please are you referring to the above post?

Yes, I do.

JBLTelecom · Fri Jul 19, 2024 11:53 pm

Please are you referring to the above post?
Yes, I do.

Please find the requested information below.

Router@Station IP Neigbhor Print Details.

Router@StationIPNei.PNG

Switch@Station IP Neigbhor Print Details.

Switch@StationIPNei.PNG

I am hoping we will get closure on this today.

sindy · Sat Jul 20, 2024 6:28 am

Instead of sending sniffing results, you have sent the ip neighbor print results again.

sindy · Sat Jul 20, 2024 8:38 am

I have just spotted another misconfiguration on Router@Station that may be related:

/interface vlan
add interface=ether1 name=vlan.s4091 use-service-tag=yes vlan-id=4091
add interface=vlan.s4091 name=vlan167 vlan-id=167
add interface=vlan.s4091 name=vlan169 vlan-id=169
add interface=vlan.s4091 name=vlan177 vlan-id=177
add interface=vlan.s4091 name=vlan185 vlan-id=185
add interface=vlan.s4091 name=vlan238 vlan-id=238
add interface=ether1 name=vlans.4092 use-service-tag=yes vlan-id=4092
add interface=ether1 name=vlans.4093 use-service-tag=yes vlan-id=4093

Either remove the rows that contain interface=vlan.s4091 completely (because you would anyway only need them if you wanted to create a local IP interface on this router in those C-VLANs), or replace vlan-s4091 by bridge1 on all of them. Ifa an interface (in this case, vlan.s4091) is a member port of a bridge, any other configuration (IP address, VLAN interface) must not be attached directly to it, but RouterOS for some reason doesn't warn you about that. In this particular case, the frames that arrive to the tagless end of the vlan.s4091 interface seem to be "consumed" by the tagged end of the vlan169 interface and never make it to the bridge, so the bridge cannot forward them to ether2.

JBLTelecom · Mon Jul 22, 2024 12:48 pm

I have just spotted another misconfiguration on Router@Station that may be related:

/interface vlan
add interface=ether1 name=vlan.s4091 use-service-tag=yes vlan-id=4091
add interface=vlan.s4091 name=vlan167 vlan-id=167
add interface=vlan.s4091 name=vlan169 vlan-id=169
add interface=vlan.s4091 name=vlan177 vlan-id=177
add interface=vlan.s4091 name=vlan185 vlan-id=185
add interface=vlan.s4091 name=vlan238 vlan-id=238
add interface=ether1 name=vlans.4092 use-service-tag=yes vlan-id=4092
add interface=ether1 name=vlans.4093 use-service-tag=yes vlan-id=4093

Either remove the rows that contain interface=vlan.s4091 completely (because you would anyway only need them if you wanted to create a local IP interface on this router in those C-VLANs), or replace vlan-s4091 by bridge1 on all of them. Ifa an interface (in this case, vlan.s4091) is a member port of a bridge, any other configuration (IP address, VLAN interface) must not be attached directly to it, but RouterOS for some reason doesn't warn you about that. In this particular case, the frames that arrive to the tagless end of the vlan.s4091 interface seem to be "consumed" by the tagged end of the vlan169 interface and never make it to the bridge, so the bridge cannot forward them to ether2.

Noted, making the changes and revert with the result.

JBLTelecom · Mon Jul 22, 2024 2:03 pm

I have just spotted another misconfiguration on Router@Station that may be related:

/interface vlan
add interface=ether1 name=vlan.s4091 use-service-tag=yes vlan-id=4091
add interface=vlan.s4091 name=vlan167 vlan-id=167
add interface=vlan.s4091 name=vlan169 vlan-id=169
add interface=vlan.s4091 name=vlan177 vlan-id=177
add interface=vlan.s4091 name=vlan185 vlan-id=185
add interface=vlan.s4091 name=vlan238 vlan-id=238
add interface=ether1 name=vlans.4092 use-service-tag=yes vlan-id=4092
add interface=ether1 name=vlans.4093 use-service-tag=yes vlan-id=4093

Either remove the rows that contain interface=vlan.s4091 completely (because you would anyway only need them if you wanted to create a local IP interface on this router in those C-VLANs), or replace vlan-s4091 by bridge1 on all of them. Ifa an interface (in this case, vlan.s4091) is a member port of a bridge, any other configuration (IP address, VLAN interface) must not be attached directly to it, but RouterOS for some reason doesn't warn you about that. In this particular case, the frames that arrive to the tagless end of the vlan.s4091 interface seem to be "consumed" by the tagged end of the vlan169 interface and never make it to the bridge, so the bridge cannot forward them to ether2.

@Sindy it worked. attached are screenshots of the Pings from RouterA to the IP 20.30.0.218 on the client router and a sniff of Mac Address C4:AD:34:F6:5D:0D.

Pings

RouterApings.PNG

Sniff

RouterAsniff.PNG

Thank you very much i appreciate your help and guidance.

sindy · Mon Jul 22, 2024 2:09 pm

it worked.

Yup, sorry it took so long, these subtle differences from proper configuration are the hardest ones to spot.

JBLTelecom · Mon Jul 22, 2024 2:21 pm

it worked.
Yup, sorry it took so long, these subtle differences from proper configuration are the hardest ones to spot.

Don't worry it has been a learning curve for me. Thanks again.