Status of ROS V7 for BGP, MPLS, VPLS

azg · Sun Nov 19, 2023 1:07 pm

Hello Everyone:
I run a small AS with a handful of BGP peers, and internally OSPF, MPLS/LDP, L3VPN and VPLS. Some of the (MPLS-) links are GREs or L2TP, with IPsec. No BFD so far. Hardware is a mix of physical devices, and CHR instances. On ROS 6.49.x the setup is rock stable - for example some BGP sessions via VPLS have uptimes of a year (resp. since the last ROS update..)
Question: Is 7.12 now ready for use in such a network? I understand there are still BGP limitations (number of prefixes received from peer), but beyond inconveniences, are we at the point where ROS V7 is stable and reliable for such a setup??
Thanks --

joegoldman · Sun Nov 19, 2023 4:05 pm

As what everyone should do - set up a testing environment using CHR and GNS3 - should be able to test your desired config and conduct failover/stability testing.

azg · Sun Nov 19, 2023 4:09 pm

Thanks joegoldman - that would then be exactly the next step. Takes 2-3 days of work though, that is why I ask here first.

kryztoval · Thu Nov 23, 2023 10:15 pm

I may be doing something wrong (configuration wise) but alas. This is my experience: I have a wifi link (ax, wifi6) between two AX3, and over this link I run a MPLS/VPLS link.

The Wifi link is unstable - it sometimes freezes for no reason and does not reconnect unless something is done to it like switching is direction (set the ap to station and the station to ap).

When the link returns the MPLS reconnects 100% of the time without issues

But the VPLS does not reconnect automatically. I am not sure if it timesout, or it tries to connect too fast and goes into passive mode, or what is going on. But frequently enough whenever the link does fail VPLS will fail to reconnect. The foolproof way to make it reconnect is to disable-enable the VPLS on both devices. Eventhough sometimes just doing this in one of the sides is enough.

This has happened for me in all versions of ROS7 up to 7.11.2

I currently have version 7.12,1 installed and I am waiting for the link to fail to see if VPLS reconnects or not.

ericsooter · Fri Dec 01, 2023 3:56 pm

We have 7.12 running on border and core routers in a few of our main datacenters (probably about 5-8GB) traffic. Running OSFP and BGP and using a mix of 2216 and 2100 routers. It's been solid over the last 3 months. I haven't been brave enough to try hardware offload yet. But that will be our plan as the firmware matures. Also, we aren't doing much with VPLS. But I've heard of issues with VPLS and hardware offload when running high traffic.

clambert · Fri Dec 01, 2023 5:00 pm

There is hardware offload support for label switching in RouterOS for now.

Wed Dec 06, 2023 5:40 am

There is hardware offload support for label switching in RouterOS for now.

There is no hardware MPLS support in RouterOS v7 at this point.

There is however a FastPath MPLS modules that will bypass the Linux Kernel for P routers and provide higher performance.

DarkNate · Wed Dec 06, 2023 8:27 am

There is no hardware MPLS support in RouterOS v7 at this point.

It's strange, isn't it? The Marvell ASICs that MikroTik uses supports MPLS/VXLAN/EVPN in hardware, but MikroTik decided it was a terrible idea to support these three on the ASICs.

mrz · Wed Dec 06, 2023 10:34 am

but MikroTik decided it was a terrible idea to support these three on the ASICs.

Hate to tell you, but your "inside source" is not trustworthy.

loloski · Wed Dec 06, 2023 10:51 am

Wow, that's good news but the million dollar question is when this going to see the light of the day

most of the Chinese cheapos switches now a days support this like Rujie/Maipu et al, please add Q-in-Q in hardware in the pipeline please

Wed Dec 06, 2023 4:11 pm

but MikroTik decided it was a terrible idea to support these three on the ASICs.

I am pretty sure it's just a case of "Good things take time" rather than any decision not to support them.

DarkNate · Wed Dec 06, 2023 7:56 pm

Hate to tell you, but your "inside source" is not trustworthy.

Ha, what inside source? The last company, I'd want an “inside source” from is MikroTik. I was poking sarcasm at the obvious fact that MikroTik ROSv7 has been a mess, and you're all very slow in bringing the hardware offloading that your hardware actually supports, to the table. But there's a lot of focus on containers, storage features etc. But where's EVPN/MPLS/VXLAN on the ASICs? Nope, nothing, nada, only unicorns and rainbow.

DarkNate · Wed Dec 06, 2023 7:59 pm

I am pretty sure it's just a case of "Good things take time" rather than any decision not to support them.

Explain to me, how Cumulus, SONiC, OcNOS supports hardware offloading for most of this stuff in 2023 and MikroTik (a company that started in 1996), doesn't?

Heck MikroTik's own (potentially) largest partner IPArchitechs have been touting OcNOS in the public domain:
https://iparchitechs.com/ecosystem/ipinfusion/

Thu Dec 07, 2023 1:42 am

Explain to me, how Cumulus, SONiC, OcNOS supports hardware offloading for most of this stuff in 2023 and MikroTik (a company that started in 1996), doesn't?

Each of those OS is quite focused on service providers and cloud providers... RouterOS is a universal networking operating system trying to meet the needs of customers from home users through SMB's right up into enterprise and ISP's.

Also, can they provide all of the functionality of RouterOS at under 20watts DC ?

Heck MikroTik's own (potentially) largest partner IPArchitechs have been touting OcNOS in the public domain:
https://iparchitechs.com/ecosystem/ipinfusion/

IPArchitechs are an independent consulting company, they can use/sell/promote whatever they want. Right tool for the job, Horses for courses

azg · Thu Dec 07, 2023 7:38 pm

Hm, maybe some participants here should start their own networking equipment company, if they know so much better? Good luck.
Meanwhile, MT will continue to deliver a surprisingly robust and versatile platform.

Is anyone running a production network with MPLS/LDP and VPLS already under 7.12? MT testlab ???

thx! a.

Fri Dec 08, 2023 4:35 am

Hm, maybe some participants here should start their own networking equipment company, if they know so much better? Good luck.
Meanwhile, MT will continue to deliver a surprisingly robust and versatile platform.

I think this would be a much better idea than antagonizing Mikrotik and it's customers.

Is anyone running a production network with MPLS/LDP and VPLS already under 7.12? MT testlab ???

I have quite a complex test lab running and it is stable. However, my main focus has been on RSVP-TE and VPNv4

buvarbeno · Tue Dec 12, 2023 4:22 pm

I tried VPLS between RB951G-2HnD and cAP AC, it works but every 5-10 minutes the RB951G-2HnD had kernel failure and reboot. Between two reboot I can reach bridged devices trough VPLS but would be better to rise this time to infinity

Is there any known issue with MIPSBE and LDP/VPLS?

PS: I tried with 7.13beta3, and no problems with EoIPV6 tunnel.

uCZBpmK6pwoZg7LR · Mon Dec 18, 2023 5:21 pm

Don`t use MPLS VPN4 ROS 7 because you CPE will be completely open for remote side of tunnel.
Firewall fail to detect inbound interface and mark it as unknown and if you filter something using :
add action=drop chain=input in-interface=<mpls interface> traffic will reach you CPE without any limitation.
This firewall rule will not work. Same will happens with forward. Mikrotik firewall on PE just blind for transit VPN4 traffic.
VPN4 in ROS 7 completely not secure due to blindness of firewall. Bug was reported to mkt and confirmed but they prefer to fix docker containers.

DarkNate · Tue Dec 19, 2023 8:45 am

Don`t use MPLS VPN4 ROS 7 because you CPE will be completely open for remote side of tunnel.
Firewall fail to detect inbound interface and mark it as unknown and if you filter something using :
add action=drop chain=input in-interface=<mpls interface> traffic will reach you CPE without any limitation.
This firewall rule will not work. Same will happens with forward. Mikrotik firewall on PE just blind for transit VPN4 traffic.
VPN4 in ROS 7 completely not secure due to blindness of firewall. Bug was reported to mkt and confirmed but they prefer to fix docker containers.

Is this ChatGPT? What the hell are you talking about? An ISP should never put any kind of data plane firewall in an MPLS core. MPLS core P and PE routers only have firewall rules for the control plane of the P and PE routers. What dumb approach is this?

CE devices should have localised firewall as any other normal CE router in the world. The CE doesn't run MPLS, you either sell L3VPN or L2VPN services to the CE, the CE only either sends a tagged/untagged VLAN or it configures a direct IP addressing on the port connected to the PE.

uCZBpmK6pwoZg7LR · Tue Dec 19, 2023 12:04 pm

Is this ChatGPT? What the hell are you talking about? An ISP should never put any kind of data plane firewall in an MPLS core. MPLS core P and PE routers only have firewall rules for the control plane of the P and PE routers. What dumb approach is this?

CE devices should have localised firewall as any other normal CE router in the world. The CE doesn't run MPLS, you either sell L3VPN or L2VPN services to the CE, the CE only either sends a tagged/untagged VLAN or it configures a direct IP addressing on the port connected to the PE.

Most probably you is ChatGPT guy and not understand how VPN4 MPLS works. I suggest for you to learn at 1st what is PE router in MPLS VPN. Have a nice day. Please don`t answer to me.

PS: most probabaly i confused with example firewall rule it have to be not mpls interface which looking to core it have to be VRF interface which looking to CE.

mrz · Tue Dec 19, 2023 12:07 pm

7.14 will have exposed vrf interface and loopback interface ,so you will be able to match in firewall traffic looped to vrf interface.

loloski · Tue Dec 19, 2023 1:25 pm

@mrz, if you could be so kind could you please confirm if MP-BGP/EVPN + VXLAN is now on horizon since IS-IS was in too? just a nugget please because this will be very critical to us in near feature

evellin · Tue Dec 19, 2023 1:48 pm

Hi, Can we hope to have BGP Filter working with VRF in 7.14 (Like we have in v6) ?

Tue Dec 19, 2023 6:04 pm

@mrz, if you could be so kind could you please confirm if MP-BGP/EVPN + VXLAN is now on horizon since IS-IS was in too? just a nugget please because this will be very critical to us in near feature

RFC4684 support is absolutely critical to eVPN and L3VPN use in the real world.

I hope Mikrotik are working on both RFC4684 and eVPN support.

loloski · Tue Dec 19, 2023 7:02 pm

Yeah, Q3 next year if MT can't still produce a decent implementation for all of this critical technologies in ISP space we are going to re-think our strategies, If only LAC mode not just LNS is readily available today we can duct tape our network and still can still wait for another 3 years more, even this was not available

selavi

clambert · Tue Dec 19, 2023 7:35 pm

We are still waiting for support for 6vpe.

spippan · Tue Dec 19, 2023 8:56 pm

...or inter-VRF route leaking via RD with import/export on ROSv7 like it was useable in v6 or since xx-years on cisco ios (yeah that one, which also powered 2800 and 1800 routers...)

DarkNate · Tue Dec 19, 2023 11:09 pm

Yeah, Q3 next year if MT can't still produce a decent implementation for all of this critical technologies in ISP space we are going to re-think our strategies, If only LAC mode not just LNS is readily available today we can duct tape our network and still can still wait for another 3 years more, even this was not available selavi

But why use LAC/LNS/L2TP/PPPoE in modern day fibre (or even wireless) networks though? Migrate to MPLS/VPLS (on MikroTik) or MPLS/EVPN/Pseudowire (on other vendors) and run DHCP directly on the BNG.

DarkNate · Tue Dec 19, 2023 11:10 pm

...or inter-VRF route leaking via RD with import/export on ROSv7 like it was useable in v6 or since xx-years on cisco ios (yeah that one, which also powered 2800 and 1800 routers...)

I don't just understand why ROSv7 is lacking a lot of the generic features dating back to the early 2000s that we can see on other vendors, not only Cisco.

spippan · Wed Dec 20, 2023 12:11 am

...or inter-VRF route leaking via RD with import/export on ROSv7 like it was useable in v6 or since xx-years on cisco ios (yeah that one, which also powered 2800 and 1800 routers...)
I don't just understand why ROSv7 is lacking a lot of the generic features dating back to the early 2000s that we can see on other vendors, not only Cisco.

neither do i

loloski · Wed Dec 20, 2023 4:31 am

@DarkNate

As a band aid solution whilst we are still waiting for proper EVPN/VXLAN to come in Mikrotik, our tech stack revolves around mikrotik for 3 years now lots of investment already from hardware to people training and we don't want to go back to pure Juniper shop if we can fight for it for cost reasons.

Wed Dec 20, 2023 4:50 am

...or inter-VRF route leaking via RD with import/export on ROSv7 like it was useable in v6 or since xx-years on cisco ios (yeah that one, which also powered 2800 and 1800 routers...)
I don't just understand why ROSv7 is lacking a lot of the generic features dating back to the early 2000s that we can see on other vendors, not only Cisco.

I thought with the new routing engine in RouterOS v7 that they would be able to add missing functionality rapidly. However we are still not at a point where we have feature parity with RouterOS v6.

mrz · Wed Dec 20, 2023 1:23 pm

inter-VRF route leaking via RD with import/export on ROSv7 like it was useable in v6

That never existed in v6 either. There was just a workaround where you could establish bgp session between vrfs on a single router and then redistribute. In theory you already can do the same in v7 too.

spippan · Wed Dec 20, 2023 6:16 pm

inter-VRF route leaking via RD with import/export on ROSv7 like it was useable in v6
That never existed in v6 either. There was just a workaround where you could establish bgp session between vrfs on a single router and then redistribute. In theory you already can do the same in v7 too.

that is the point in vrf route leaking. same on cisco in the internal, underlying structure

and how is it done in ros7? i tested it in v6 and v7
6 worked
7 didn't

DarkNate · Thu Dec 21, 2023 2:37 am

@DarkNate

As a band aid solution whilst we are still waiting for proper EVPN/VXLAN to come in Mikrotik, our tech stack revolves around mikrotik for 3 years now lots of investment already from hardware to people training and we don't want to go back to pure Juniper shop if we can fight for it for cost reasons.

I learnt this mistake early on. Currently, for all new business operations, that's just starting, I ensure to design the network in a way to avoid relying on MikroTik-specific features or implementations, making it easy to migrate to Juniper, once we have a revenue stream and large number of customers.

Think of it this, way, use MikroTik as a bootstrap kit for business, dump it when the business is no longer in bootstrap mode.

If you can get investments coming in from outside, even better, dump MikroTik network-wide and switch to Juniper.

I limit MikroTik only for out of band management network. Other migrate to Juniper, or Nokia depending on use-case and situation.

If you need good reason on why to dump MikroTik, well look at their staff's attitude, not any one person, but all of them on this forum and on the support portal.

loloski · Thu Dec 21, 2023 7:42 am

@Darknate

I can feel you and I can clearly see your point and that was really obvious, but I don't need reasons to ditch MT because the company I work for already accept that fact that MT as a company is not perfect, my personal only sour grape with them is they don't layout their roadmap on what they want to be and this really affect us in direct way because every fiscal year we have to make a budget For X equipment with X Feature and every year as far as MT is concern are heads is spinning because they don't have clear roadmap.

Thu Dec 21, 2023 8:27 am

I feel that RouterOS v7 is very slowly heading in the right direction, unlike RouterOS v6 routing that had no development for almost 10 years, I can see regular development happening in RouterOS v7 routing. We already have IS-IS, MPLS Fast Re-Route, MPLS ECMP that did not exist in RouterOS v6.

However RouterOS v7 routing is still not at a point that I would trust it to replace RouterOS v6 for a MPLS based network, maybe in 6 months it will be, but not right now.

loloski · Thu Dec 21, 2023 8:51 am

Care to share how much MPLS traffic you have at peak and is it in tile arch?, we have a pilot MPLS implementation base on v6 (mpls atom/pseudowire) in one of our PoP and just running < 500mb at peak

DarkNate · Thu Dec 21, 2023 10:42 am

@Darknate

I can feel you and I can clearly see your point and that was really obvious, but I don't need reasons to ditch MT because the company I work for already accept that fact that MT as a company is not perfect, my personal only sour grape with them is they don't layout their roadmap on what they want to be and this really affect us in direct way because every fiscal year we have to make a budget For X equipment with X Feature and every year as far as MT is concern are heads is spinning because they don't have clear roadmap.

I guess you're in the engineering team and not management, in my case, I'm in management and I take final decisions on these type of issues, I've seen too many businesses whining about their MikroTik-only stack and increased OpEx due to hacks/workarounds and bs over time.

My advice, convince your management to opt for multivendor environment. Using Juniper-only model is also terrible idea.

If CapEx wasn't an issue here's what I would do for an ISP network:
MikroTik for OOB/MGMT network
Juniper for edge and core routers
Possibly Arista for layer 3 distribution routers
x64 with DPDK for CGNAT
Juniper ACX or some Nokia for BNG and traffic shaping + stateless IPv6 routing
Juniper or Nokia for MPLS core, also for MPLS-TP on inter-site fibre pathways
Nokia for OLT/ONT

Top it off with Netbox + Ansible for network device config and automation + Python for scripting or custom functions + CI/CD pipeline built by qualified software engineer + IPv6 native everywhere including management backbone.

In such model, there's no SPOF of vendor.

loloski · Thu Dec 21, 2023 11:55 am

You are right and spot on, I'm responsible with Engineering In perfect world I got the final say on most things related to network from Core,CO,Pop down to Last mile, but still can be vetoed once there was a big Asian money at stake down to a drain pipe, Cap-ex is hard to come by in emerging market that's why Mikrotik really fit the bill for us, we are using Juniper too in Core and Edge period but some areas in the business relies heavily on Chinese brand from OLT,EDFA down to ONU's

Thanks for sharing your genuine thoughts on most of the random post here

Thu Dec 21, 2023 3:03 pm

Care to share how much MPLS traffic you have at peak and is it in tile arch?, we have a pilot MPLS implementation base on v6 (mpls atom/pseudowire) in one of our PoP and just running < 500mb at peak

In excess of 20Gbit worth of L3VPN traffic per CCR1072 router.

buvarbeno · Thu Dec 21, 2023 3:08 pm

I tried VPLS between RB951G-2HnD and cAP AC, it works but every 5-10 minutes the RB951G-2HnD had kernel failure and reboot. Between two reboot I can reach bridged devices trough VPLS but would be better to rise this time to infinity
Is there any known issue with MIPSBE and LDP/VPLS?

PS: I tried with 7.13beta3, and no problems with EoIPV6 tunnel.

I upgraded my RB951G-2HnD to 7.14 beta and gave a try to VPLS again. Now It is little better, the router is stable for hours with VPLS, but still have kernel faults and reboots.

DarkNate · Fri Dec 22, 2023 12:07 pm

I upgraded my RB951G-2HnD to 7.14 beta and gave a try to VPLS again. Now It is little better, the router is stable for hours with VPLS, but still have kernel faults and reboots.

Do a fresh netinstall of 7.13, with no-default-config, ensure RouterBOARD firmware is also on 7.13.

This will resolve your kernel panics and crashes.

spippan · Mon Dec 25, 2023 2:37 am

That never existed in v6 either. There was just a workaround where you could establish bgp session between vrfs on a single router and then redistribute. In theory you already can do the same in v7 too.
that is the point in vrf route leaking. same on cisco in the internal, underlying structure

and how is it done in ros7? i tested it in v6 and v7
6 worked
7 didn't

bump.

spippan · Thu Dec 28, 2023 7:02 pm

inter-VRF route leaking via RD with import/export on ROSv7 like it was useable in v6
That never existed in v6 either. There was just a workaround where you could establish bgp session between vrfs on a single router and then redistribute. In theory you already can do the same in v7 too.

this particular issue has its own thread yet
viewtopic.php?p=1044901#p1044901

please help out how this is achieved in rOSv7
so i can "un-shelf" 9 CCRs already (which are only collecting dust atm.)

uCZBpmK6pwoZg7LR · Sat Dec 30, 2023 7:19 pm

Care to share how much MPLS traffic you have at peak and is it in tile arch?, we have a pilot MPLS implementation base on v6 (mpls atom/pseudowire) in one of our PoP and just running < 500mb at peak

In excess of 20Gbit worth of L3VPN traffic per CCR1072 router.

How you got it ? I cannot pass 1gb/s . Can you please share your conf and traffic profile(avg size of packets )?

DarkNate · Wed Jan 03, 2024 12:35 pm

How you got it ? I cannot pass 1gb/s . Can you please share your conf and traffic profile(avg size of packets )?

Follow the MTU section from the Edge/BNG guide. I use jumbo frames network-wide:
9k L3 MTU, maxed L2 MTU on physical ports/interfaces.
VPLS L2 MTU capped to 9100.
Single bridge config as per MikroTik official docs.

Layer 3 VLAN termination L3 MTU depends on customer, if customer behind VLAN is 1500 then I set 1500 on L3 MTU of the VLAN sub-interface, otherwise it's 9000 MTU end-to-end there as well.

No problem pushing 1Gbps+ with low CPU usage on CCR2004 with VPLS (I don't have more than 1Gbps+ aggregate traffic yet, but still this is decent considering hardware model).

Wed Jan 03, 2024 6:17 pm

How you got it ? I cannot pass 1gb/s . Can you please share your conf and traffic profile(avg size of packets )?

Easy! 9136 MTU on backbone, traffic profile is extremely mixed and the key thing is that this was tens of thousands of sessions in L3VPN.

If it was VPLS tunnels, or elephant flows there is absolutely no way we could reach that level of performance on a Tilera based CCR.

clambert · Wed Jan 03, 2024 10:59 pm

the key thing is that this was tens of thousands of sessions in L3VPN

Can Tilera CCRs distribute L3VPN traffic between multiple cores? I have found it impossible to achieve on ARM and ARM64 devices.

DarkNate · Thu Jan 04, 2024 11:18 am

Can Tilera CCRs distribute L3VPN traffic between multiple cores? I have found it impossible to achieve on ARM and ARM64 devices.

Probably not. In 2024, you're better off with hardware that has ASICs.

CPU can't do much unless there's XDP for ingress and DPDK for egress, both are non-existent on MikroTik platform.

"FastTrack" exists but it's a precursor to modern-day XDP/DPDK or VPP, it's not good enough for near line rate.

azg · Thu Jan 25, 2024 11:58 pm

Coming back to my original question: I converted part of my network to ROS V7.13.3 (and some testing with 7.14b8).
V7 seems fundamentally expanded to serve as the basis for years of features to come. Overall it seems close to being ready for the features I use.

Below is some random feedback of what I ran into.
Setup is: Network core routers with public BGP are on V6 still, two access routers are on V7. Config has underlying OSPF, MPLS on all routers, with LDP via Ethernet, GRE/IPSec and L2TP/IPsec interfaces.
- VPLS pseudowires: They work versus V6 endpoints, but not versus V7. When I lay RSVP-TE (CSPF or manual path) on top, then VPLS works. Could be config error. Might have something to do with LDP Neighbors having the Passive flag.
- RSVP-TE works with a simple test, using CSPF as well as manual (strict and loose) hops. Both Primary and Secondary pathes come up, but then the router gets confused about what primary and secondary is.
- Nasty: TE/Tunnel Path: The Hops do not work in Winbox - they do not generate correct syntax. CLI only. Grr... this one cost time.
- RSVP-TE without LDP? Should be possible, right?
- Issues with VPLS pseudowires not delivering traffic into bridges: I had to disable Fast Forward etc.
- Not clear: Is it still necessary to stop & start an LDP instance so that it picks up changes? I had the impression that was the case in V6.
- 7.14beta versions: meaning of the VRF interfaces is not clear. Adding it to the VRF then made things work (not related to MPLS etc.)
- BGP: NRLI accept filtering based on prefixes works for most simple cases, but is not enough. Prefix count is another one missing. MT denial can last for years... ... I know.

- I have not yet tested VPN4, as there seem to be issues with the Route Reflector (discussed elsewhere in this forum).

Greetings
a.

mrz · Fri Jan 26, 2024 12:23 am

Few comments:

- VPLS PW works fine between v7 routers, either there is something specific or misconfiguration.
- Yes, RSVP-TE does not require LDP, those are independent label distribution methods.
- You might need to restart LDP when you change advertise/accept filters.
- Lo and VRF interfaces are exposed from linux kernel, if you want to know how vrfs work in linux and what is vrf interface, this article is good place to start:
https://www.dasblinkenlichten.com/worki ... inux-vrfs/
- prefix count is implemented for quite some time now.
- There are no known problems with VPNv4 and route reflectors. There is a known/not yet fixed problem with VPLS and route reflectors.

uCZBpmK6pwoZg7LR · Fri Jan 26, 2024 12:46 pm

- There are no known problems with VPNv4 and route reflectors. There is a known/not yet fixed problem with VPLS and route reflectors.

What about silent firewall ignore on CPE for traffic which came from VPN4 to VRF (SUP-141699) ?

azg · Fri Jan 26, 2024 1:30 pm

Thanks mrz!

Regarding the VPLS PW: I fixed a config bug (never underestimate the effect of Masquerading on IP Multicasts etc... ) ...
however I seem to be able to reproduce a race condition:

I have three VPLS configured on my R020 router (V7.13.3):
- PW10 and PW11 terminate on CHR/x86 instances under ROS V6 in a data center circa 11ms away. Path is via Ethernet locally to R015, which has GRE tunnels to the data center. R015 runs ROS V7.13.3 too.
- PW12 terminates on R015, i.e. is a VPLS PW between two local ROS V7.13.3 routers, connected via Ethernet (via ROS Bridges on both routers).

Now... here are the strange effects:
- When I disable PW12 then also PW10 and PW11 go down for about a second, before they come back.
- When I wait until PW10 and PW11 are back up, and then enable PW12, then PW12 comes up (and works)
- When I quickly re-enable PW12 before PW10 and PW11 have come back up, then PW12 will not come up, no matter how long I wait. Interesting: In this case, PW12 on the other end (R015) will show "running". On both R020 and R015, there are (identical) Remote and Local Labels assigned in Status.

The next question then was this: When I change PW12, do PW10 and PW11 go down only cosmetically? I tested with 50ms pinging through PW10, and in three tests lost 40, then 3, then approx 30 packets. So PW10 was indeed down for 1-2 seconds typically. NOT good that one VPLS PW being disabled causes other VPLS PW to lose connectivity. Was this always the case??

Obviously there is other cross-dependency between VPLS PWs. I test with Winbox, and the click rate is rather high. When testing on the CLI, probably one can not that quickly change and provoke issues. Also, kryztoval reported in this thread earlier (Nov23) about VPLS not coming back up over Wifi links -- might be related. Contributing possibly is R020: Its a slow hAP ac lite, causing different timing compared to a test network where all routers are equally fast.

Greetings a.

spippan · Fri Jan 26, 2024 2:03 pm

- There are no known problems with VPNv4 and route reflectors. There is a known/not yet fixed problem with VPLS and route reflectors.

do you consider not being able to leak routes properly between local VRFs (bgp-vpnv4 local) "no known problems"?

nichky · Fri Jan 26, 2024 2:28 pm

@mrz

why when i'm doing VRRP + BGP (v7) in order to make it work, i need to tick multihop

also with BGP-VPLS can you stop dynamic names on that?
I want to use the one that i'll specification

Network5 · Sat Jan 27, 2024 11:11 am

@azg

Now... here are the strange effects:
- When I disable PW12 then also PW10 and PW11 go down for about a second, before they come back.
- When I wait until PW10 and PW11 are back up, and then enable PW12, then PW12 comes up (and works)

I have the same behaviour for VPLS terminating at the same endpoints.

After a reboot it takes from 5 to 20 second for the tunnels to come up. Supposing that is waiting OSPF to redistribute the routes and LDP to start.

azg · Sun Jan 28, 2024 3:56 pm

Regarding RSVP-TE: Yes I can confirm that BGP signaled VPLS works with LDP disabled.

The only issue with BGP VPLS (compared to LDP signaled VPLS) is that the PW name is dynamic, and the PW therefore can be used only in the intended full mesh with dynamic addition to a Bridge. At first glance that is all fine, but the effect is that - due to the VPLS PW being dynamically added - it is quite difficult to work with VLANs. Yes, VLAN Filtering can be enabled on the Bridge used by BGP VPLS, but as the dynamically added VPLS PWs change name, keeping VLAN settings up to date would require scripting. Unfortunately it is not possible to add a bridge (e.g. the one used for BGP VPLS, with VLAN filtering off) to an other bridge (that has VLAN filtering on).

Comparably, LDP VPLS PWs appear as L2 interfaces similar to Ethernet, and they can be added to Bridges with no restrictions.

Not sure if I was able to describe the issue above. Anyone sees a workaround?
(a physical loopback using two ethernet ports would probably work, or a tunnel on the router itself. Not pretty.)

What would be needed: A form of Template for dynamically adding a L2 transport to an existing bridge.
The issue is actually broader: Dynamically generated VPLS PW for example have ARP enabled. They themselves require some Template.

a.

clambert · Sun Jan 28, 2024 4:21 pm

The problem related to the assignment of vlans on dynamically created VPLS interfaces was reported in the following post:
viewtopic.php?t=163891

I also created a support ticket asking about this [SUP-79290] and they responded to me with the following:

Currently, dynamic VPLS interfaces added to the vlan-filtering bridge do not support any VLAN settings, so you cannot place these interfaces on certain untagged VLAN. The only available solution is to bridge "/interface vlan" with VPLS interfaces.

DarkNate · Sun Jan 28, 2024 7:14 pm

I hope they allow proper PVID/VLAN config for BGP signalled VPLS.

I have not yet adopted BGP signalled VPLS on MikroTik because of this PVID/VLAN problem, even though the BGP signalling itself works solid.

Raf44 · Fri Feb 02, 2024 3:18 pm

- There are no known problems with VPNv4 and route reflectors. There is a known/not yet fixed problem with VPLS and route reflectors.

by "no known problem with VPNv4 and RR" should I understand that my ticket SUP-134760 is not understood / processed ;
or that the problem is more to do with the Best-Path Selection process, not being applied correctly in a VRF after a filter has been applied to routes retrieved from a RR ?

uCZBpmK6pwoZg7LR · Thu Feb 08, 2024 8:29 am

I can say even more since ros 7.14 it will be no firewall in PE routers . PE-CE segment will be not protected. will be not possible to SNAT and DNAT VPN4 traffic on PE router.

mrz · Thu Feb 08, 2024 10:18 am

It is not entirely true, PE can still be protected and client behind PE as well. Only thing that you cannot do is destination nat on traffic from MPLS cloud to CE.

uCZBpmK6pwoZg7LR · Thu Feb 08, 2024 1:58 pm

It is not entirely true, PE can still be protected and client behind PE as well. Only thing that you cannot do is destination nat on traffic from MPLS cloud to CE.

DNAT to local interface(bridge assigned to vrf works) "works" is too loud word in that case, at least it reach PREROUTING, FORWARD in mangle and filter but not reach POSTROUTING. so not possible to do snat. It is weird that possible to do DNAT for bridge allocated to vrf but not possible to do SNAT for dnatted traffic which after such conversion quite far from MPLS.

Not working completely DNAT and Filter to connected routes(not assigned adresses to router itself) and to static routes.

Static routes not exported to VPN4 at all. I have gateway which cannot do it and doesn`t matter how many times i restart it.

VRF which contain none interface don`t work in VPN4 at all until you add any connected interface to vrf .

PS: And yes when you got ticket which describe that PE to VRF traffic does not follow packet flow diagram you just updated Packet flow diagram wiki and told that such packets is just an exclusion instead of real bug fix. And it is extremely weird now that packets which come from vpn to vrf have established state and in firewall only visible answers for packets which come from VPN.

" There are no known problems with VPNv4 and route reflectors. "©

mrz · Thu Feb 08, 2024 3:25 pm

It sure does work.

Setup

(111.15.0.1)CE1—-PE1——-PE2——CE2(111.13.0.1)

PE2 will be used for nat testing.
Relevant IPs on PE2:

1   111.13.0.2/24      111.13.0.0    sfp-sfpplus2      
;;; router-test
3   111.16.0.1/24      111.16.0.0    vrf-dummy   

/ip vrf
add interfaces=sfp-sfpplus2,vrf-dummy name=vrfTest

Relevant routes:

   DAc   dst-address=111.13.0.0/24 routing-table=vrfTest gateway=sfp-sfpplus2@vrfTest immediate-gw=sfp-sfpplus2 distance=0 scope=10 suppress-hw-offload=no 
         local-address=111.13.0.2%sfp-sfpplus2@vrfTest 
   DAy   dst-address=111.15.0.0/24 routing-table=vrfTest gateway=203.0.113.2 immediate-gw=111.11.0.1%sfp-sfpplus1 distance=200 scope=40 target-scope=30 suppress-hw-offload=no 
   DAc   dst-address=111.16.0.0/24 routing-table=vrfTest gateway=vrf-dummy@vrfTest immediate-gw=vrf-dummy distance=0 scope=10 suppress-hw-offload=no 
         local-address=111.16.0.1%vrf-dummy@vrfTest

Test1: SRC NAT on packets from CE

/ip firewall nat
add action=src-nat chain=srcnat disabled=yes src-address=111.13.0.1 to-addresses=111.16.0.1


[admin@RB5009] /log> /ping 111.15.0.1 src-address=111.13.0.1
  SEQ HOST                                     SIZE TTL TIME       STATUS                                                                                     
    0 111.15.0.1                                 56  63 327us     
    1 111.15.0.1                                 56  63 343us     

10:58:14 firewall,info prerouting: in:sfp-sfpplus2 out:(unknown 0), connection-state:established,snat src-mac dc:2c:6e:46:f8:93, proto ICMP (type 8, code 0), 111.13.0.1->111.15.0.1, NAT (111.13.0.1->111.16.0.1)->111.15.0.1, len 56 
10:58:14 firewall,info forward: in:sfp-sfpplus2 out:sfp-sfpplus1, connection-state:established,snat src-mac dc:2c:6e:46:f8:93, proto ICMP (type 8, code 0), 111.13.0.1->111.15.0.1, NAT (111.13.0.1->111.16.0.1)->111.15.0.1, len 56 
10:58:14 firewall,info postrouting: in:sfp-sfpplus2 out:sfp-sfpplus1, connection-state:established,snat src-mac dc:2c:6e:46:f8:93, proto ICMP (type 8, code 0), 111.13.0.1->111.15.0.1, NAT (111.13.0.1->111.16.0.1)->111.15.0.1, len 56 

10:58:14 firewall,info prerouting: in:vrfTest out:(unknown 0), connection-state:established,snat src-mac d2:03:6d:b0:44:be, proto ICMP (type 0, code 0), 111.15.0.1->111.16.0.1, NAT 111.15.0.1->(111.16.0.1->111.13.0.1), len 56 
10:58:14 firewall,info forward: in:vrfTest out:sfp-sfpplus2, connection-state:established,snat src-mac d2:03:6d:b0:44:be, proto ICMP (type 0, code 0), 111.15.0.1->111.13.0.1, NAT 111.15.0.1->(111.16.0.1->111.13.0.1), len 56 
10:58:14 firewall,info postrouting: in:vrfTest out:sfp-sfpplus2, connection-state:established,snat src-mac d2:03:6d:b0:44:be, proto ICMP (type 0, code 0), 111.15.0.1->111.13.0.1, NAT 111.15.0.1->(111.16.0.1->111.13.0.1), len 56

Test2: DSTNAT on traffic from mpls cloud to local VRF IP dstnated and forwarded to CE

/ip firewall nat
add action=dst-nat chain=dstnat disabled=yes dst-address=111.16.0.1 to-addresses=111.13.0.1

[admin@CCR2004_2S+] /ip/address> /ping 111.16.0.1 src-address=111.15.0.1
  SEQ HOST                                     SIZE TTL TIME       STATUS                                                                                       
    0 111.16.0.1                                 56  62 336us     
    1 111.16.0.1                                 56  62 290us   

11:04:15 firewall,info prerouting: in:vrfTest out:(unknown 0), connection-state:established,dnat src-mac d2:03:6d:b0:44:be, proto ICMP (type 8, code 0), 111.15.0.1->111.16.0.1, NAT 111.15.0.1->(111.16.0.1->111.13.0.1), len 56 
11:04:15 firewall,info forward: in:vrfTest out:sfp-sfpplus2, connection-state:established,dnat src-mac d2:03:6d:b0:44:be, proto ICMP (type 8, code 0), 111.15.0.1->111.13.0.1, NAT 111.15.0.1->(111.16.0.1->111.13.0.1), len 56 
11:04:15 firewall,info postrouting: in:vrfTest out:sfp-sfpplus2, connection-state:established,dnat src-mac d2:03:6d:b0:44:be, proto ICMP (type 8, code 0), 111.15.0.1->111.13.0.1, NAT 111.15.0.1->(111.16.0.1->111.13.0.1), len 56 

11:04:15 firewall,info prerouting: in:sfp-sfpplus2 out:(unknown 0), connection-state:established,dnat src-mac dc:2c:6e:46:f8:93, proto ICMP (type 0, code 0), 111.13.0.1->111.15.0.1, NAT (111.13.0.1->111.16.0.1)->111.15.0.1, len 56 
11:04:15 firewall,info forward: in:sfp-sfpplus2 out:sfp-sfpplus1, connection-state:established,dnat src-mac dc:2c:6e:46:f8:93, proto ICMP (type 0, code 0), 111.13.0.1->111.15.0.1, NAT (111.13.0.1->111.16.0.1)->111.15.0.1, len 56 
11:04:15 firewall,info postrouting: in:sfp-sfpplus2 out:sfp-sfpplus1, connection-state:established,dnat src-mac dc:2c:6e:46:f8:93, proto ICMP (type 0, code 0), 111.13.0.1->111.15.0.1, NAT (111.13.0.1->111.16.0.1)->111.15.0.1, len 56

Test3: SRCNAT on packets from MPLS to CE

/ip firewall nat
add action=src-nat chain=srcnat src-address=111.15.0.1 to-addresses=111.16.0.1

[admin@CCR2004_2S+] /ip/address> /ping 111.13.0.1 src-address=111.15.0.1
  SEQ HOST                                     SIZE TTL TIME       STATUS                                                                                       
    0 111.13.0.1                                 56  62 294us     
    1 111.13.0.1                                 56  62 282us      


11:11:50 firewall,info output: in:(unknown 0) out:sfp-sfpplus2, connection-state:established,snat src-mac d2:03:6d:b0:44:be, proto ICMP (type 8, code 0), 111.15.0.1->111.13.0.1, NAT (111.15.0.1->111.16.0.1)->111.13.0.1, len 56 
11:11:50 firewall,info postrouting: in:vrfTest out:sfp-sfpplus2, connection-state:established,snat src-mac d2:03:6d:b0:44:be, proto ICMP (type 8, code 0), 111.15.0.1->111.13.0.1, NAT (111.15.0.1->111.16.0.1)->111.13.0.1, len 56 

11:11:50 firewall,info prerouting: in:sfp-sfpplus2 out:(unknown 0), connection-state:established,snat src-mac dc:2c:6e:46:f8:93, proto ICMP (type 0, code 0), 111.13.0.1->111.15.0.1, NAT 111.13.0.1->(111.16.0.1->111.15.0.1), len 56 
11:11:50 firewall,info forward: in:sfp-sfpplus2 out:sfp-sfpplus1, connection-state:established,snat src-mac dc:2c:6e:46:f8:93, proto ICMP (type 0, code 0), 111.13.0.1->111.15.0.1, NAT 111.13.0.1->(111.16.0.1->111.15.0.1), len 56 
11:11:50 firewall,info postrouting: in:sfp-sfpplus2 out:sfp-sfpplus1, connection-state:established,snat src-mac dc:2c:6e:46:f8:93, proto ICMP (type 0, code 0), 111.13.0.1->111.15.0.1, NAT 111.13.0.1->(111.16.0.1->111.15.0.1), len 56

As you can see NAT works as expected and logs reflect that.

And yes when you got ticket which describe that PE to VRF traffic does not follow packet flow diagram you just updated Packet flow diagram wiki and told that such packets is just an exclusion instead of real bug fix.

As it was mentioned already in your other topic on this problem, it is linux implementation, and yes documentation will be updated to reflect that.

And yes, there are no known problems with route reflectors. What you mentioned here is completely irrelevant to how route reflector is reflecting routes.

nichky · Fri Feb 09, 2024 6:41 am

@mrz

what is the purpose of adding ip-address of the vrf?

That is something new that i've seen on v7 (v7.14 beta 10)

mrz · Fri Feb 09, 2024 8:35 am

what do you mean?

nichky · Fri Feb 09, 2024 11:29 am

see attached

haven't seen in other version, that could be some useful, which i'm not familiar with

nichky · Fri Feb 23, 2024 2:30 am

@mrz
can U advice please?

azg · Fri Feb 23, 2024 3:52 pm

Thanks for all the posts.
Here are some updates from my recent tests under 7.14rc1:

First, my network is a mix of CHR, x86, and physical routers. OSPF with LDP, LDP advertisements filtered to loopbacks only.
MPLS MTU is 1200, to accomodate L2TP and GRE tunnels to DSL CPEs. eBGP runs on CHR/x86 with 2x full table, plus an IX as well as some peers. BGP sessions on the CHR run on VPLS links, which terminate on physical routers. The network is not productive, so I have the luxury to be able to shut down. Voluntary or not.. see below.

So meanwhile the network is migrated to 7.14rc1. As indicated in a previous post, I had a few bugs on my side (never underestimate the effect of NAT... on LDP in this case). LDP seems robust. RSVP too, but I use it less.

One issue is with L2TP&IPsec. In some cases I was unable to get the tunnels up at all, no matter what. Without IPsec they come up immediately. I replaced some L2TP with Wireguard, but WG does not carry MPLS. Smells like multiple issues concurrently. Unresolved for now.

I then cautiously migrated BGP sessions to V7, and at some point had a fully mixed ROS V6 / V7 network, without issues. Yes mrz, the number of BGP prefixes received is nicely displayed, and there are good tools under /routing/bgp/advertisements and in /routing/route
Overall Winbox needs a bugfix here and there, but on the CLI things are good.

I have one issue that I can not explain, it is described below in detail. It occured twice, caused loss of VPLS on one node, and subsequent loss of public BGP sessions. The effect is among three involved routers:

R001 (RB3011) has a 1G link to a VMhost with R007 (CHR).
R001 links with 1G to R003 (RB750gr3)
R003 has a 1G link to a VMhost with R008 (x86)
all routers run LDP, with advertisement limited to loopback addresses.

R007 is a BGP router, with VPLS carrying BGP to R001, on R001 bridged to external peers.
R008 is a BGP router, with VPLS carrying BGP to R003, on R003 bridged to external peers.
R007 has a VPLS to R008, but the BGP session here was disabled.
This worked with a handful of BGP peers for nearly a week. Stability testing only.

THEN... I enabled the BGP session between R007 and R008, and BOOOM all VPLS on R007 go down, and with that all BGP peers on R007.
VPLS remained down. I found an issue on R001: Here the LDP Local Transport address changed for peer R007, from the loopback address of R001 (198.18.0.1) to another IP address found under /ip/address. When shown in Winbox & sorted by IP address, the first enabled IP address was taken - may be a coincidence. Here are LDP neighbors:

[admin@R001] /mpls/ldp/neighbor> print
Flags: D - DYNAMIC; O - OPERATIONAL, W - PASSIVE-WAIT; t - SENDING-TARGETED-HELLO; v - VPLS; p - PASSIVE
Columns: TRANSPORT, LOCAL-TRANSPORT, PEER, ADDRESSES
# TRANSPORT LOCAL-TRANSPORT PEER ADDRESSES
0 DWtvp 198.18.0.7 172.31.19.190 198.18.0.7:0
1 DOtvp 198.18.0.3 198.18.0.1 198.18.0.3:0 x.x.x.x.
...

As a result, the LDP debug log complains about "198.18.0.7:0 wrong address for passive connection 198.18.0.7->198.18.0.1, expected 198.18.0.7->172.31.19.190" due to R007(198.18.0.7) attempting VPLS.
And yes, my Local Transport is given in the LDP instance, and only there.

ANY hint anyone? Why would a Local Transport address change? It smells like bug, but again as usual, 80 to 90% of the suspected MT bugs turn out to be config errors on the side of the network admin.. : /

greetings
a.

quantumx · Wed Mar 13, 2024 5:13 pm

Upgraded a 6.9.13 production config on a test router today. Config was a vanilla implementation as per reference design in documentation, of OSPF / MPLS / LDP / BGP signalled VPLS in a P/PE role. No VRF4 implemented. MPLS MTU set to 2000

Upgraded using the WebUI Packages menu, CCR1009 hardware.

Upgrade path 6.9.13 -> 7.12.1 (upgrade) -> 7.14.1

Problems encountered:

Update to 7.12.1 broke MPLS LDP Filters:

Before:

/mpls ldp advertise-filter> print
Flags: X - disabled
# PREFIX NEIGHBOR ADVERTISE
0 10.1.255.0/24 all yes
1 0.0.0.0/0 all no

After:
/mpls/ldp/advertise-filter> print detail
Flags: X - disabled
0 prefix=10.1.255.0/24 neighbor=0.0.0.0 advertise=yes

1 prefix=0.0.0.0/0 neighbor=0.0.0.0 advertise=no

This matched only peers with IP 0.0.0.0. ie. no match

Update to 7.12.1 lost configuration for MPLS MTU

Before:
/mpls interface> print detail
Flags: X - disabled, * - default
0 * interface=all mpls-mtu=2000

After:
/mpls interface> print detail

Flags: X - disabled; * - builtin

Otherwise process was reasonably smooth. Hopefully this saves someone a little time when upgrading.