VRRP with VLANS

t04s · Fri Aug 19, 2022 8:56 pm

Hi everyone,

Have been trying to add a second redundant Mikrotik router into an existing setup but following the VRRP configuration as per the Mikrotik docs doesn't work for me. I have several VLAN trunk uplinks from switches to the Mikrotik, and an onwards trunk to a firewall/WAN device. Mangle rules are applied to send Internet traffic out on the correct statically configured gateways to the firewall but this breaks once VRRP is enabled. The VRRP interfaces themselves work fine, but the configured routes become unreachable. I assume I have to adjust the mangle rules because with VRRP enabled I see a new dynamic route for the VRRP interface, alongside the usually dynamic routes for VLAN interfaces.

See the VLAN configuration...

/interface

Flags: D - dynamic; X - disabled, R - running; S - slave; P - passthrough 
 0  RS  name="eth01-lan" default-name="ether1" type="ether" mtu=1500 actual-mtu=1500 l2mtu=1580 
        max-l2mtu=10222 mac-address=48:8F:5A:D3:73:24 ifname="eth4" ifindex=10 id=5 
        last-link-up-time=may/16/2022 07:14:02 link-downs=0 

 3  RS  name="eth10-lan" default-name="ether10" type="ether" mtu=1500 actual-mtu=1500 l2mtu=1580 
        max-l2mtu=10222 mac-address=48:8F:5A:D3:73:2D ifname="eth13" ifindex=19 id=14 
        last-link-up-time=may/16/2022 07:14:02 link-downs=0 

 4  RS  name="eth11-lan" default-name="ether11" type="ether" mtu=1500 actual-mtu=1500 l2mtu=1580 
        max-l2mtu=10222 mac-address=48:8F:5A:D3:73:2E ifname="eth14" ifindex=20 id=15 
        last-link-down-time=may/25/2022 18:42:49 last-link-up-time=may/25/2022 18:43:57 link-downs=4 

 5  RS  name="eth12-lan" default-name="ether12" type="ether" mtu=1500 actual-mtu=1500 l2mtu=1580 
        max-l2mtu=10222 mac-address=48:8F:5A:D3:73:2F ifname="eth15" ifindex=21 id=16 
        last-link-down-time=may/20/2022 14:26:18 last-link-up-time=may/25/2022 17:00:09 link-downs=8 

16  R   name="br01" type="bridge" mtu=auto actual-mtu=1500 l2mtu=1580 mac-address=48:8F:5A:D3:73:24 
        ifname="br4" ifindex=65 id=52 last-link-down-time=may/16/2022 18:03:34 
        last-link-up-time=may/16/2022 18:03:34 link-downs=8 

17  R   name="vlan10-management" type="vlan" mtu=1500 actual-mtu=1500 l2mtu=1576 
        mac-address=48:8F:5A:D3:73:24 ifname="vlan53" ifindex=66 id=53 
        last-link-down-time=may/16/2022 18:03:34 last-link-up-time=may/16/2022 18:03:34 link-downs=8 

18  R   name="vlan11-voip" type="vlan" mtu=1500 actual-mtu=1500 l2mtu=1576 
        mac-address=48:8F:5A:D3:73:24 ifname="vlan54" ifindex=67 id=54 
        last-link-down-time=may/16/2022 18:03:34 last-link-up-time=may/16/2022 18:03:34 link-downs=8 

19  R   name="vlan12-data" type="vlan" mtu=1500 actual-mtu=1500 l2mtu=1576 
        mac-address=48:8F:5A:D3:73:24 ifname="vlan57" ifindex=69 id=57 
        last-link-down-time=may/16/2022 18:03:34 last-link-up-time=may/16/2022 18:03:34 link-downs=10 

20  R   name="vlan13-dmz" type="vlan" mtu=1500 actual-mtu=1500 l2mtu=1576 mac-address=48:8F:5A:D3:73:2>
        ifname="vlan56" ifindex=68 id=56 last-link-down-time=may/16/2022 18:03:34 
        last-link-up-time=may/16/2022 18:03:34 link-downs=8

/interface/bridge

Flags: X - disabled, R - running 
 0 R name="br01" mtu=auto actual-mtu=1500 l2mtu=1580 arp=enabled arp-timeout=auto 
     mac-address=48:8F:5A:D3:73:24 protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=yes 
     ageing-time=5m priority=0x2000 max-message-age=20s forward-delay=15s transmit-hold-count=6 
     vlan-filtering=yes ether-type=0x8100 pvid=1 frame-types=admit-only-vlan-tagged 
     ingress-filtering=yes dhcp-snooping=no

/interface/bridge/port

Flags: X - disabled, I - inactive; D - dynamic; H - hw-offload 
 0 I   interface=eth03-lan bridge=*18 priority=0x80 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=10 frame-types=admit-all ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no 

 1     interface=eth01-lan bridge=br01 priority=0x80 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none hw=yes auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=1 frame-types=admit-only-vlan-tagged ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no 

 2     interface=eth10-lan bridge=br01 priority=0x30 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none hw=yes auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=1 frame-types=admit-only-vlan-tagged ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no 

 3     interface=eth11-lan bridge=br01 priority=0x20 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none hw=yes auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=1 frame-types=admit-only-vlan-tagged ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no 

 4     interface=eth12-lan bridge=br01 priority=0x10 path-cost=10 internal-path-cost=10 edge=auto 
       point-to-point=auto learn=auto horizon=none hw=yes auto-isolate=no restricted-role=no 
       restricted-tcn=no pvid=1 frame-types=admit-only-vlan-tagged ingress-filtering=yes 
       unknown-unicast-flood=yes unknown-multicast-flood=yes broadcast-flood=yes tag-stacking=no 
       bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no

/interface/bridge/vlan

Flags: X - disabled, D - dynamic 
 0   bridge=br01 vlan-ids=10 tagged=br01,eth01-lan,eth10-lan,eth11-lan,eth12-lan untagged="" 
     current-tagged=br01,eth01-lan,eth10-lan,eth12-lan,eth11-lan current-untagged="" 

 1   bridge=br01 vlan-ids=11 tagged=br01,eth01-lan,eth10-lan,eth11-lan,eth12-lan untagged="" 
     current-tagged=br01,eth01-lan,eth10-lan,eth12-lan,eth11-lan current-untagged="" 

 2   bridge=br01 vlan-ids=12 tagged=br01,eth01-lan,eth10-lan,eth11-lan,eth12-lan untagged="" 
     current-tagged=br01,eth01-lan,eth10-lan,eth12-lan,eth11-lan current-untagged="" 

 3   bridge=br01 vlan-ids=13 tagged=br01,eth01-lan,eth10-lan,eth11-lan,eth12-lan untagged="" 
     current-tagged=br01,eth01-lan,eth10-lan,eth12-lan,eth11-lan current-untagged=""

The VRRP config is as follows...

/interface/vrrp

M - master, B - backup, F - failure 
 0 I   name="vrrp1" mtu=1500 mac-address=00:00:5E:00:01:01 arp=enabled 
       arp-timeout=auto interface=vlan10-management group-master="" vrid=1 
       priority=100 interval=1s preemption-mode=yes authentication=none 
       password="" on-backup="" on-master="" on-fail="" version=3 
       v3-protocol=ipv4 sync-connection-tracking=no 

 1 I   name="vrrp2" mtu=1500 mac-address=00:00:5E:00:01:02 arp=enabled 
       arp-timeout=auto interface=vlan12-voip group-master="" vrid=2 
       priority=100 interval=1s preemption-mode=yes authentication=none 
       password="" on-backup="" on-master="" on-fail="" version=3 
       v3-protocol=ipv4 sync-connection-tracking=no 

 2 I   name="vrrp3" mtu=1500 mac-address=00:00:5E:00:01:01 arp=enabled 
       arp-timeout=auto interface=vlan13-data group-master="" vrid=3 
       priority=100 interval=1s preemption-mode=yes authentication=none 
       password="" on-backup="" on-master="" on-fail="" version=3 
       v3-protocol=ipv4 sync-connection-tracking=no 

 3 I   name="vrrp4" mtu=1500 mac-address=00:00:5E:00:01:01 arp=enabled 
       arp-timeout=auto interface=vlan14-dmz group-master="" vrid=4 
       priority=100 interval=1s preemption-mode=yes authentication=none 
       password="" on-backup="" on-master="" on-fail="" version=3 
       v3-protocol=ipv4 sync-connection-tracking=no

/ip/address

   ;;; Management
     address=172.22.10.251/24 network=172.22.10.0 
     interface=vlan10-management actual-interface=vlan10-management 

 1   ;;; Data
     address=172.22.13.251/24 network=172.22.13.0 interface=vlan13-data 
     actual-interface=vlan13-data 

 2   ;;; VoIP
     address=172.22.12.251/24 network=172.22.12.0 interface=vlan12-voip 
     actual-interface=vlan12-voip 

 3 X ;;; Native
     address=172.22.99.1/24 network=172.22.99.0 interface=eth03-lan 
     actual-interface=eth03-lan 

 4   ;;; DMZ
     address=172.22.14.251/24 network=172.22.14.0 interface=vlan14-dmz 
     actual-interface=vlan14-dmz 

 5 X address=172.22.10.1/32 network=172.22.10.1 interface=vrrp1 
     actual-interface=vrrp1 

 6 X address=172.22.12.1/32 network=172.22.12.1 interface=vrrp2 
     actual-interface=vrrp2 

 7 X address=172.22.13.1/32 network=172.22.13.1 interface=vrrp3 
     actual-interface=vrrp3 

 8 X address=172.22.14.1/32 network=172.22.14.1 interface=vrrp4 
     actual-interface=vrrp4

/ip/route

1  As   dst-address=0.0.0.0/0 routing-table=VLAN13-Outbound pref-src="" 
         gateway=172.22.13.2 immediate-gw=172.22.13.2%vlan13-data 
         check-gateway=arp distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 2  As   dst-address=0.0.0.0/0 routing-table=VLAN10-outbound pref-src="" 
         gateway=172.22.10.2 immediate-gw=172.22.10.2%vlan10-management 
         check-gateway=arp distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 3  As   dst-address=0.0.0.0/0 routing-table=VLAN12-Outbound pref-src="" 
         gateway=172.22.12.2 immediate-gw=172.22.12.2%vlan12-voip 
         check-gateway=arp distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 4  As   dst-address=0.0.0.0/0 routing-table=VLAN14-Outbound pref-src="" 
         gateway=172.22.14.2 immediate-gw=172.22.14.2%vlan14-dmz 
         check-gateway=arp distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no

/ip/firewall/mangle

 3    ;;; Mark outbound traffic from VLAN100 for routing
      chain=prerouting action=mark-routing new-routing-mark=VLAN100-Outbound 
      passthrough=no src-address-list=Management-VLAN-subnet 
      dst-address-list=!private-address log=no log-prefix="" 

 4    ;;; Mark outbound traffic from VLAN102 for routing
      chain=prerouting action=mark-routing new-routing-mark=VLAN102-Outbound 
      passthrough=no src-address-list=VoIP-VLAN-subnet 
      dst-address-list=!private-address log=no log-prefix="" 

 5    ;;; Mark outbound traffic from VLAN103 for routing
      chain=prerouting action=mark-routing new-routing-mark=VLAN103-Outbound 
      passthrough=no src-address-list=Data-VLAN-subnet 
      dst-address-list=!private-address log=no log-prefix="" 

 6    ;;; Mark outbound traffic from VLAN104 for routing
      chain=prerouting action=mark-routing new-routing-mark=VLAN104-Outbound 
      passthrough=no src-address-list=DMZ-VLAN-subnet 
      dst-address-list=!private-address log=no log-prefix=""

Not sure why the routes become unreachable but some pointers would help me.

Thanks,
t04s

nichky · Sat Aug 20, 2022 1:15 am

we need your topology,

t04s · Sat Aug 20, 2022 1:58 am

Sure.

EDIT: with a better image...

This only shows the fully redundant part of the network, excluding any infrastructure none critical. Each have redundant links fully interconnected.

Pfsenses provide WAN/NAT and Internet firewall. Uplinks are trunks to allow a firewall policy per VLAN. They have CARP IPs and pfSync to provide HA on both WAN/LAN.

Mikrotiks are LAN routers and handle inter-VLAN routing, and firewalling. No NAT.

Switches provide LAN uplinks to Mikrotiks, with a primary root bridge and one as backup.

Hope this give more information. Let me know if you need more.

---
t04s

sindy · Sun Aug 21, 2022 8:16 pm

I must say I cannot imagine what exactly goes wrong when you enable the VRRP interfaces, as your configuration seems fine to me from this point of view. Can you post the result of /ip/route/print in that bad state, when the VRRP interfaces are enabled?

Regardless that, I am also not sure I understand the logical topology - are the /24 subnets in the individual VLANs used solely for communication between the Mikrotik routers and the pfsenses?

t04s · Mon Aug 22, 2022 12:40 pm

Hi, thanks for taking a look. I've replaced the diagram on the post above with much more detail. To answer your question directly, no, the /24 subnets are used for each VLAN on both the trunk uplink and trunk links to the lower switches into the rest of the network. See below for further information.

The whole idea is that each device is interconnected with a failover/backup device. Adding the trunks on the diagram makes it look more more complicated but in reality this is pretty simple.

For the purposes of explanation, for now, focus on the left side ignoring the right-side failover devices. From the Mikrotik perspective there are four connected ports which are all trunks. Port 1 (eth01-lan) is a trunk uplinked to the pfSense. This carries all the VLANs so there can be WAN firewall policies per VLAN. The pfSense is the Internet gateway on each VLAN as 172.22.10.2. 172.22.12.2, and so on. Ports 10,11 and 12 (eth10-lan, eth11-lan and eth12-lan) are trunks to the rest of the LAN network (you can discount eth12-lan as it's outside the scope of the question, as too are lower-level switches not shown). The Mikrotik handles inter-VLAN routing and inter-VLAN firewall.

Now, for failover each device on the left is crossed-over to each on the right, for example; Mikrotik2 > Switch1, Mikrotik1 > Switch2, Mikrotik1 > pfSense2, etc. At the pfSense, CARP IPs (VIPs) are employed (WAN and LAN-side) and pfSync to keep configs synced. Across devices, RSTP is enabled. 10G Switch1 is the root bridge for the network and port priorities on devices are set accordingly. All of this is working well. If anything is missing on the failover diagram or config outputs above, it's because it either isn't implemented, or is temporarily removed but this is the intended design.

Now, the final thing to resolve is the Mikrotik failover using VRRP. This network is operational so the plan was to configure VRRP on Mikrotik2 and switch over to it in a maintenance window, then configuring VRRP on Mikrotik1 as the primary to switch back. On the first maintenance switch over this route issue occurred. As I say, when I enable VRRP the routes become inaccessible. I now have this whole set up in GNS3 and I get the same issue. See the route output when the problem occurs;

/ip/route

#      DST-ADDRESS      GATEWAY             DISTANCE
0  IsH 0.0.0.0/0        172.22.10.2              1
  DAc  172.22.10.0/24  vlan10-management         0
  DAc  172.22.10.1/32  vrrp1                     0
  DAc  172.22.12.0/24  vlan12-voip               0
  DAc  172.22.12.1/32  vrrp2                     0
  DAc  172.22.13.0/24  vlan13-data               0
  DAc  172.22.13.1/32  vrrp3                     0
  DAc  172.22.14.0/24  vlan14-dmz                0
  DAc  172.22.14.1/32  vrrp4                     0
1  IsH 0.0.0.0/0        172.22.13.2              1
2  IsH 0.0.0.0/0        172.22.10.2              1
3  As  0.0.0.0/0        172.22.12.2              1
4  As  0.0.0.0/0        172.22.14.2              1

I did also notice odd behavior that enabling only one VRRP such as vrrp2 which is linked to vlan12-voip VLAN interface makes a different route become inaccessible such as 172.22.13.2 which is not the gateway for that VLAN.

---
t04s

t04s · Mon Aug 22, 2022 11:59 pm

Through further testing it appears there is possibly some conflict between CARP and VRRP. The vrid/vhid I'm using are the same on the Mikrotik (VRRP) and pfSense (CARP). Changing one of the vhid's on the pfSense gets the relevant route working.

I've found possibly related issues here and here.

Does anyone know if VRRP and CARP can co-exist? At the very least, must the vrid/vhid differ?

EDIT: this appears to confirm it's the case the IDs need to be unique.

---
t04s

t04s · Tue Aug 23, 2022 1:53 pm

Changed all VHIDs and VRIDs so they are distinct and all is well again.

Hopefully this helps someone else. The behavior experienced was odd because the VIPs worked and were accessible on the network to access both the Mikrotik and pfSense, but all outbound routes from Mikrotik > pfSense became disabled. As soon as the IDs were changed all routes immediately were accessible.

---
t04s

osimester · Sun Jan 29, 2023 6:54 pm

Hi,

I have a network with pfSense and Mikrotik. VRRP, CARP and bunch of VLANs. I have solutions what should help for you based on your topic:

At my network two pfsense act as firewall. Two mikrotik CCR2004 are the Core Router with VRRP. These devices connected to a Mikrotik switch. Each them with two SFP+ module with fiber cable. These interfaces are in bonding with LACP like this:

pfSense Firewall <-- 2 interface in bond --> Core switch <-- 2 interface in bond --> Core Router

Other members of the network connected to Core switch with vlans. All Mikrotik devices are on ROS 7.6.

pfSense and Core Router see each other in a separate VLAN (vlan030) through Core switch. All intern network's outbond and inbound traffic routed through on this VLAN.

The pfSense uses the vlan030 to handle CARP communications.

The Core Routers, bonding interfaces has IP address. The VRRP interface is on the bonding interface. VRRP has IP address with /32 netmask. If you are using for example /24, your router will create ECMP routes between vrrp interface and vlan interface which is not the best. Check your route table.

vrrp-vlan2.PNG.png

VLAN interfaces are on the VRRP interface with the Core Router's IP addresses. So if the VRRP has a failover situation, the VLANs IP addresses will be inactive and the other VRRP member will handle the traffic. Don't forget to use ConTrack Sync.

vrrp-vlan.PNG

With this topology, the CARP and VRRP communications are separated. CARP is in the tagged vlan030, VRRP is in the bond interface (untagged vlan if you like).

Two important details:

On the Mikrotik, MAC address of bonding interfaces must be forced. If the MAC address generated itself (based on the first running physical interface), there is a risk of network disturbances.

On the pfSense, MAC address is also important, because pfSense changing the MAC address of physical interfaces, even at a config sync of CARP. And it gives back XMLRPC errors which tells you nothing. So MAC address must be forced there too, on physical interfaces.

I hope it helps to you!

For Mikrotik

If somebody read it from Mikrotik, I would like to ask you to change the VRRP section of the help.mikrotik.com: there is a statement in the topic that VRRP IP must have /32 netmask:

vrrp-vlan3.PNG.png

BUT if the VRRP is running on a VLAN interface, the router is not reachable with /32 netmask. I bet it is the same with any non-physical interface.

Please add some informations about that to the VRRP section of help.mikrotik.com. Thank you!

VRRP with VLANS [SOLVED]

VRRP with VLANS

Re: VRRP with VLANS

Re: VRRP with VLANS

Re: VRRP with VLANS

Re: VRRP with VLANS

Re: VRRP with VLANS

Re: VRRP with VLANS [SOLVED]

Re: VRRP with VLANS