VRRP with single WAN and Single LAN Address
I am trying to set up a couple of CHR to run side by side for a streaming service that we offer. This requires that I have a single WAN and and a single LAN address that I can use. For best availability I need to have one of the routers running as the primary and the other as the secondary so that in the event the primary goes down my traffic will failover to the secondary router. These CHR are located in mirrored VMware datacenters so in the event of an issue at one site the second site can take over. I am not very well versed in Mikrotik so any help would be appreciated.
Re: VRRP with single WAN and Single LAN Address
I guess my initial question is if you have an investment in VMWare... wouldn't vMotion/etc work better? In that case you'd two identical CHRs, with VMWare controlling which one was active.
VRRP certainly be possible on LAN side pretty easily. And VRRP supports connection tracking sync which keep it hitless. But WAN side gets more tricky, and more specific to what's going on... And you may want to consider BGP instead of VRRP on WAN side.
VRRP certainly be possible on LAN side pretty easily. And VRRP supports connection tracking sync which keep it hitless. But WAN side gets more tricky, and more specific to what's going on... And you may want to consider BGP instead of VRRP on WAN side.
Re: VRRP with single WAN and Single LAN Address
I do not understand what a "mirrored datacenter" means. VRRP uses specific MAC addresses so the physical machines running the CHRs have to be interconnected on L2 level so that a single public address could migrate between them using VRRP (there are L3 methods that do not require L2 interconnection but have other complicated requirements). Plus the hypervisors must allow the VMs to use other MAC adddresses than the ones they have been assigned. On Mikrotik, you can configure one VRRP interface to track the state of another one, by means of adjusting the priority of the tracking one when the leading one changes state, using the on-master and on-backup items in its configuration that can hold scripts.
Re: VRRP with single WAN and Single LAN Address
Yeah that's kinda the central question here.I do not understand what a "mirrored datacenter" means.
Based on my now dated VMWare knowledge, but the general idea is that an "application" (i.e. CHR) didn't NOT have to be aware of redudency — since VMWare managed syncing all machine state and networks between clusters. i.e. VMWare mirrors memory, disk, and network for something like CHR... so in there is only one CHR from RouterOS POV. But this depends on much of VMWare kool-aid you've drank.... since how "mirrored" things are is a sliding scale and different features sets/licensing.
Re: VRRP with single WAN and Single LAN Address
Sorry, by mirrored datacenters I meant geographically redundant. I have two datacenters in two different physical locations, but each is running the exactly same hardware as the other.
Starting from the ground up I currently have the two CHR up and running and set up VRRP between them, but for some reason the one keep switching from master to remote constantly. For testing right now I have both routers built in the same datacenter. I created a new vlan for the Mikrotik VRRP, and have enable promiscuous mode for that vlan. It seemed to be working fine to start with and then I started to see the flip flop on the master/backup.
Starting from the ground up I currently have the two CHR up and running and set up VRRP between them, but for some reason the one keep switching from master to remote constantly. For testing right now I have both routers built in the same datacenter. I created a new vlan for the Mikrotik VRRP, and have enable promiscuous mode for that vlan. It seemed to be working fine to start with and then I started to see the flip flop on the master/backup.
Re: VRRP with single WAN and Single LAN Address
I have a problem to wrap my head around a scenario where you've got two georedundant datacenters between which a single IP address may freely migrate on an L2 segment (presumably a VxLAN) - in my understanding of networking, it would mean that there is a router in each of the datacenters that is able to advertise the subnet that lives inside that L2 VxLAN to the internet. Is that the case?
Other than that, please post the anonymized configuration exports of both CHRs as they look now, as you refer to switching "from master to remote" whereas the possible states of a VRRP interface are MASTER and BACKUP; what typically happens if there is a misconfiguration is that both machines become masters because they do not receive the VRRP messages from each other.
Other than that, please post the anonymized configuration exports of both CHRs as they look now, as you refer to switching "from master to remote" whereas the possible states of a VRRP interface are MASTER and BACKUP; what typically happens if there is a misconfiguration is that both machines become masters because they do not receive the VRRP messages from each other.
Re: VRRP with single WAN and Single LAN Address
vMotion is not L3 or Application redundancy. It just helps if the physical host dies.
You can use a vrrp-interface with on-master= and on-backup= scripts to enable/disable other interfaces, tunnels or ip-adresses.
You can use a vrrp-interface with on-master= and on-backup= scripts to enable/disable other interfaces, tunnels or ip-adresses.
Re: VRRP with single WAN and Single LAN Address
As far as the networking goes all the traffic in the two datacenters are set up in vlans. The datacenters are Cisco UCS/FI stacks being uplinked to our MPLS network through a pair of Cisco Nexus 9k and ASR 9ks for the MPLS ring. We have a video streaming solution we run at one of the datacenter locations on it's own hardware and that hardware connects to an ASR 9k which brings it into the network and allows my datacenters to get the streams. We bring those streams into the vmware stack on a particular vlan and drop it off to ether 2 on the CHR Mikrotik and then we share that out to our real world customers over the public IP addresses we have set on the Mikrotik via ether 1. I will post the config for the existing Mikrotik below.
I do now have two test CHRs up and running as a VRRP pair with internet access and if I disable the WAN ethernet port on either one my test box stays up and connected to the internet. The next part I need to figure out is how to make this work for HA. Right now both test boxes live in just one of the datacenters so I didn't have to build out the crossconnects through my MPLS ring. But the end goal is to have 1 CHR in each datacenter.
Hopefully that helps clarify things. I would draw out a diagram of the setup and post that as well, but am unsure of how I would upload that here.
I do now have two test CHRs up and running as a VRRP pair with internet access and if I disable the WAN ethernet port on either one my test box stays up and connected to the internet. The next part I need to figure out is how to make this work for HA. Right now both test boxes live in just one of the datacenters so I didn't have to build out the crossconnects through my MPLS ring. But the end goal is to have 1 CHR in each datacenter.
Hopefully that helps clarify things. I would draw out a diagram of the setup and post that as well, but am unsure of how I would upload that here.
Code: Select all
/interface ethernet
set [ find default-name=ether1 ] comment="VLAN 310 Internet"
set [ find default-name=ether2 ] comment="VLAN 116 Video"
/snmp community
set [ find default=yes ] addresses=y.y.65.18/32 name=mikrotik
/system logging action
set 1 disk-file-name=log
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip settings
set max-neighbor-entries=8192
/ip address
add address=x.x.116.1/24 comment="IP Address for Internal LAN" interface=ether2 network=x.x.116.0
add address=y.y.64.165/28 comment="IP Address for VidServer 3" interface=ether1 network=y.y.64.160
add address=y.y.64.162/28 comment="IP Address for VidServer App/Web Portal" interface=ether1 network=y.y.64.160
add address=y.y.64.166/28 comment="IP Address for VidServer 4" interface=ether1 network=y.y.64.160
add address=y.y.64.163/28 comment="IP Address for VidServer1" interface=ether1 network=y.y.64.160
add address=y.y.64.164/28 comment="IP Address for VidServer 2" interface=ether1 network=y.y.64.160
add address=y.y.64.167/28 comment="IP Address for VidServer 5" interface=ether1 network=y.y.64.160
/ip dns
set servers=y.y.64.11,y.y.64.12
/ip firewall filter
add action=accept chain=input comment="Allow Ping" protocol=icmp src-address=y.y.64.15
add action=accept chain=input comment="Allow Ping 2" protocol=icmp src-address=y.y.65.18
add action=accept chain=input comment="Allow ICMP from Other" protocol=icmp src-address=z.z.192.190
add action=accept chain=input comment="Allow ICMP from y.y.64.10" protocol=icmp
add action=accept chain=input comment="Allow port 8000 http access for mgmt" dst-port=8000 protocol=tcp src-address=y.y.65.0/24
add action=accept chain=input comment="Allow SSH access for Mgmt" dst-port=22 protocol=tcp src-address=y.y.65.0/24
add action=accept chain=input comment="Allow Established and Related Connections" connection-state=established,related log=yes
add action=accept chain=input comment=SNMP dst-port=161 protocol=udp src-address=y.y.65.18
add action=accept chain=input comment="VidBox 7022" dst-port=25069 protocol=tcp
add action=accept chain=input comment="VidBox 7022" dst-port=27514 protocol=udp
add action=drop chain=input comment="Drop Everything Else" in-interface=ether1 log=yes log-prefix=DROP
/ip firewall nat
add action=src-nat chain=srcnat comment="VidServer 1 - One to One NAT" out-interface=ether1 src-address=x.x.116.141 to-addresses=y.y.64.163
add action=src-nat chain=srcnat comment="VidServer 2 - One to One NAT" out-interface=ether1 src-address=x.x.116.142 to-addresses=y.y.64.164
add action=src-nat chain=srcnat comment="VidServer 3 - One to One NAT" out-interface=ether1 src-address=x.x.116.143 to-addresses=y.y.64.165
add action=src-nat chain=srcnat comment="VidServer 4 - One to One NAT" out-interface=ether1 src-address=x.x.116.144 to-addresses=y.y.64.166
add action=src-nat chain=srcnat comment="VidServer 5 - One to One NAT" out-interface=ether1 src-address=x.x.116.145 to-addresses=y.y.64.167
add action=dst-nat chain=dstnat comment="VidServer VRRP x.x.116.140 Port 443" dst-address=y.y.64.162 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.140 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer VRRP x.x.116.140 Port 80" dst-address=y.y.64.162 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.140 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 1 Port 443" dst-address=y.y.64.163 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.141 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 1 Port 80" dst-address=y.y.64.163 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.141 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 2 Port 443" dst-address=y.y.64.164 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.142 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 2 Port 80" dst-address=y.y.64.164 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.142 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 3 Port 443" dst-address=y.y.64.165 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.143 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 3 Port 80" dst-address=y.y.64.165 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.143 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 4 Port 443" dst-address=y.y.64.166 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.144 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 4 Port 80" dst-address=y.y.64.166 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.144 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 5 Port 443" dst-address=y.y.64.167 dst-port=443 in-interface=ether1 protocol=tcp to-addresses=x.x.116.145 to-ports=443
add action=dst-nat chain=dstnat comment="VidServer 5 Port 80" dst-address=y.y.64.167 dst-port=80 in-interface=ether1 protocol=tcp to-addresses=x.x.116.145 to-ports=80
add action=dst-nat chain=dstnat comment="VidServer 1 Port TCP 25069" dst-address=y.y.64.163 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.141 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 2 Port TCP 25069" dst-address=y.y.64.164 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.142 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 3 Port TCP 25069" dst-address=y.y.64.165 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.143 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 4 Port TCP 25069" dst-address=y.y.64.166 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.144 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 5 Port TCP 25069" dst-address=y.y.64.167 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.145 to-ports=25069
add action=dst-nat chain=dstnat comment="VidServer 1 Port UDP 27514" dst-address=y.y.64.163 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.141 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 2 Port UDP 27514" dst-address=y.y.64.164 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.142 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 3 Port UDP 27514" dst-address=y.y.64.165 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.143 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 4 Port UDP 27514" dst-address=y.y.64.166 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.144 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 5 Port UDP 27514" dst-address=y.y.64.167 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.145 to-ports=27514
add action=dst-nat chain=dstnat comment="VidServer 5041 authorization" dst-address=y.y.64.162 dst-port=25069 in-interface=ether1 protocol=tcp to-addresses=x.x.116.141 to-ports=25069
add action=dst-nat chain=dstnat comment="Vidserver 5041 syslog" dst-address=y.y.64.162 dst-port=27514 in-interface=ether1 protocol=udp to-addresses=x.x.116.141 to-ports=27514
/ip firewall service-port
set tftp disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip ipsec policy
set 0 disabled=yes
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=y.y.64.161
/ip service
set telnet disabled=yes
set ftp address=y.y.65.0/24 disabled=yes
set www address=y.y.65.0/24 port=8000
set ssh address=y.y.65.0/24
set www-ssl address=y.y.65.0/24 port=8080
set api disabled=yes
set winbox address=y.y.65.0/24 disabled=yes
set api-ssl disabled=yes
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/routing bfd configuration
add disabled=no interfaces=all min-rx=200us min-tx=200us multiplier=5
/snmp
set enabled=yes trap-generators=interfaces trap-version=2
/system clock
set time-zone-name=America/Denver
/system logging
set 3 action=memory
add topics=firewall
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=y.y.65.17
add address=y.y.65.18
/tool graphing interface
add allow-address=y.y.65.18/32 interface=ether1 store-on-disk=no
/tool sniffer
set filter-ip-protocol=0 filter-port=ircu
Re: VRRP with single WAN and Single LAN Address
Just curious, but why not fully utilize the VSM functionality since you already have a bunch of ASR 9Ks? I mean, why use CHRs as edge routers?
Btw, this is how you add an image to a post:
Btw, this is how you add an image to a post:
You do not have the required permissions to view the files attached to this post.
Re: VRRP with single WAN and Single LAN Address
He is the diagram. Not sure what you meant by VSM though.
You do not have the required permissions to view the files attached to this post.
Re: VRRP with single WAN and Single LAN Address
Thanks, but I have to admit I'm pretty confused by the network diagram as the image doesn’t seem to follow a clear visual logic and it’s hard to make sense of it without additional context.
For example, how does the red-dashed VRRP relate to the four nodes (VRRP1, VRRP2, CHR1, CHR2)? And what role does the blue-dashed CHR play in this layout? I'm also still curious about why use CHR instead of the powerfull ASR9k as CE, which btw already has that role on the right side?
Maybe you can provide an explanatory text and possibly a more logically organized network topology diagram?
For example, how does the red-dashed VRRP relate to the four nodes (VRRP1, VRRP2, CHR1, CHR2)? And what role does the blue-dashed CHR play in this layout? I'm also still curious about why use CHR instead of the powerfull ASR9k as CE, which btw already has that role on the right side?
Maybe you can provide an explanatory text and possibly a more logically organized network topology diagram?
Re: VRRP with single WAN and Single LAN Address
Ah, for me a "datacenter" normally means something provided by a 3rd partyThe datacenters are Cisco UCS/FI stacks being uplinked to our MPLS network through a pair of Cisco Nexus 9k and ASR 9ks for the MPLS ring.
So I figure the two ASRs use BGP to advertise the public subnet where the CHRs live to the internet, which the diagram is silent about. How exactly the L2 is tunneled between the two locations (MPLS, VxLAN, or something else) is not much relevant.The most important part, i.e. the VRRP configuration, is missing in that export, which kind of denies its purpose.I will post the config for the existing Mikrotik below.
But it dawned on me earlier today - as you are still a beginner with Mikrotik, could it be that you've mistook the RM indication next to the VRRP interface as ReMote whilst it actually indicates Running and Master?
Because in that export, I can see the usual omission - the rules in chain input of ip firewall filter drop the incoming VRRP packets, so the two CHRs do not know about each other and hence both become masters. Which wouldn't be a big deal if it wasn't for NAT.
Re: VRRP with single WAN and Single LAN Address
Sorry the existing router I took the export from is the Live router that is currently handling the traffic. The idea is to spin up two new CHR to replace that single and provide an HA type of solution for my current video offering. I am trying to get the two new CHR routers up and running in a test environment to prove the concept will work.Ah, for me a "datacenter" normally means something provided by a 3rd partyThe datacenters are Cisco UCS/FI stacks being uplinked to our MPLS network through a pair of Cisco Nexus 9k and ASR 9ks for the MPLS ring.
The most important part, i.e. the VRRP configuration, is missing in that export, which kind of denies its purpose.I will post the config for the existing Mikrotik below.
But it dawned on me earlier today - as you are still a beginner with Mikrotik, could it be that you've mistook the RM indication next to the VRRP interface as ReMote whilst it actually indicates Running and Master?
Because in that export, I can see the usual omission - the rules in chain input of ip firewall filter drop the incoming VRRP packets, so the two CHRs do not know about each other and hence both become masters. Which wouldn't be a big deal if it wasn't for NAT.
To clarify things maybe a bit more I work for an ISP. We offer an over the top option for video to be streamed to devices on our network. As part of this setup there is a URL that the steaming devices connect back to. This obviously resolves to a single IP that lives on the current CHR. Our goal is to be able to have two new CHR that can either individually (master/backup) or load balance that traffic coming in and in the event of a failure one of them can take over the entire load if needed. So we would prefer and active/active setup, but an active standby setup is agreeable to get going.
As for the transport layer it shouldn't play a huge part. We bring internet directly to the CHR on a vlan that gets stripped as the VMware host and the same for video so the CHR is never even aware of a vlan being used. So for all intents and purposes they have direct connections to the VidServer and the internet.
Hope that clarifies things some more. Sorry if I am not explaining things very well.
Re: VRRP with single WAN and Single LAN Address
That's the point - you have experienced the issue on the test pair of CHRs but you have posted an export of something else. So post the configuration of the two test devices together with the output of /interface vrrp print from both in the state you "do not like", so that we can debug and resolve the actual issue.Sorry the existing router I took the export from is the Live router that is currently handling the traffic.
...
Hope that clarifies things some more. Sorry if I am not explaining things very well.
Re: VRRP with single WAN and Single LAN Address
The red dashed connections are what I am hoping to try and get working and the blue dashed is what is currently in service. As for using the 9ks they are transparent to the VMs and VidServer they are the L2 transport network.Thanks, but I have to admit I'm pretty confused by the network diagram as the image doesn’t seem to follow a clear visual logic and it’s hard to make sense of it without additional context.
For example, how does the red-dashed VRRP relate to the four nodes (VRRP1, VRRP2, CHR1, CHR2)? And what role does the blue-dashed CHR play in this layout? I'm also still curious about why use CHR instead of the powerfull ASR9k as CE, which btw already has that role on the right side?
Maybe you can provide an explanatory text and possibly a more logically organized network topology diagram?
Re: VRRP with single WAN and Single LAN Address
Alright, got it. As for load balancing (ie vrrp load sharing) and grouping, have you checked if the ROS version has what you need? It might be worth a look, since it doesn’t have all the ‘bells and whistles’ of the Cisco IOS XR equivalent.
Re: VRRP with single WAN and Single LAN Address
The load balancing I can address down the road. My main concern for now is getting the single WAN IP to work with the dual router setup. The uplinks for the CHR are going to be 10Gb so as of right now we have plenty of bandwidth.Alright, got it. As for load balancing (ie vrrp load sharing) and grouping, have you checked if the ROS version has what you need? It might be worth a look, since it doesn’t have all the ‘bells and whistles’ of the Cisco IOS XR equivalent.
The subnet that the WAN IPs live in is a /28 of which 6 IPs are in used. So let say 100.100.64.162-167 in use. 161 is the gateway, and 168-174 are unused right now.
Since it was asked earlier. Here are the two test routers I have set up.
RTR 1
Code: Select all
/interface bridge
add name=lanbridge1
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
set [ find default-name=ether2 ] disable-running-check=no
set [ find default-name=ether3 ] disable-running-check=no name=ether4
/interface vrrp
add comment=LAN interface=ether1 name=vrrp1 preemption-mode=no priority=254 sync-connection-tracking=yes vrid=49
add comment=WAN disabled=yes interface=ether4 name=vrrp2 preemption-mode=no vrid=59
/ip pool
add name=dhcp_pool0 ranges=192.168.88.2-192.168.88.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface=lanbridge1 name=dhcp1
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=lanbridge1 interface=ether2
/ip firewall connection tracking
set enabled=yes
/ip address
add address=192.168.1.10/24 interface=ether1 network=192.168.1.0
add address=192.168.1.1 interface=vrrp1 network=192.168.1.1
add address=192.168.88.1/24 interface=lanbridge1 network=192.168.88.0
add address=172.20.120.254/24 comment=Internet interface=ether4 network=172.20.120.0
/ip dhcp-client
add disabled=yes interface=ether1
/ip dhcp-server network
add address=192.168.88.0/24 dns-server=192.168.88.1,8.8.8.8 gateway=192.168.88.1
/ip dns
set servers=8.8.8.8,8.8.4.4
/ip firewall nat
add action=masquerade chain=srcnat comment=Internet
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=172.20.120.1 routing-table=main suppress-hw-offload=no
/system clock
set time-zone-name=US/Mountain
/system note
set show-at-login=noCode: Select all
/interface bridge
add name=lanbridge1
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
set [ find default-name=ether2 ] disable-running-check=no
set [ find default-name=ether3 ] disable-running-check=no
/interface vrrp
add comment=LAN interface=ether1 name=vrrp1 preemption-mode=no sync-connection-tracking=yes vrid=49
add comment=WAN disabled=yes interface=ether3 name=vrrp2 preemption-mode=no priority=200 vrid=59
/ip pool
add name=dhcp_pool0 ranges=192.168.88.2-192.168.88.254
/ip dhcp-server
add address-pool=dhcp_pool0 interface=lanbridge1 name=dhcp1
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=lanbridge1 interface=ether2
/ip firewall connection tracking
set enabled=yes
/ip address
add address=192.168.88.1/24 interface=lanbridge1 network=192.168.88.0
add address=192.168.1.20/24 interface=ether1 network=192.168.1.0
add address=192.168.1.1 interface=vrrp1 network=192.168.1.1
add address=172.20.120.253/24 comment=Internet interface=ether3 network=172.20.120.0
/ip dhcp-client
add interface=ether1
/ip dhcp-server network
add address=192.168.88.0/24 dns-server=192.168.88.1 gateway=192.168.88.1
/ip dns
set servers=8.8.8.8,8.8.4.4
/ip firewall nat
add action=masquerade chain=srcnat comment=Internet
/ip route
add comment=Internet disabled=no dst-address=0.0.0.0/0 gateway=172.20.120.1 routing-table=main suppress-hw-offload=no
/system clock
set time-zone-name=US/Mountain
/system note
set show-at-login=noRe: VRRP with single WAN and Single LAN Address
OK, no firewall filter rules at all so VRRP packets from the other router can definitely get in if they make it through the LAN.Here are the two test routers I have set up.
With these configurations, what does /interface/vrrp/print where name=vrrp1 show at both test routers?
Re: VRRP with single WAN and Single LAN Address
After discussing the issue internally with some techs, our best guess at this point is that the flip-flop behavior might be caused by a VMware Virtual Network Adapter 'state change' which can happen for various reasons like network congestion, resource constraints, virtual switch misconfigurations, bandwidth cap mgmt tools reconfiguring network settings, etc. This state change is signaled to the guest OS, causing ROS to flip VRRP state.
If possible, enable logging in ESXi on both instances and check the ROS logs for VRRP events, then sync the timestamps to see when and why the issues occur (make sure time is synchronized beforehand).
If possible, enable logging in ESXi on both instances and check the ROS logs for VRRP events, then sync the timestamps to see when and why the issues occur (make sure time is synchronized beforehand).
Re: VRRP with single WAN and Single LAN Address
I wouldn't pay any attention to vrrp2 right now. I just created it this morning and have done nothing with it yet. But my intention is for that to be the WAN vrrp interface and vrrp1 is for the lan side.OK, no firewall filter rules at all so VRRP packets from the other router can definitely get in if they make it through the LAN.Here are the two test routers I have set up.
With these configurations, what does /interface/vrrp/print where name=vrrp1 show at both test routers?
RTR 1
Code: Select all
Flags: X - DISABLED; F - FAILURE
Columns: NAME, INTERFACE, MAC-ADDRESS, VRID, PRIORITY, INTERVAL, VERSION, V3-PROTOCOL, SYNC-CONNECTION-TRACKING
# NAME INTERFACE MAC-ADDRESS VRID PRIORITY INTERVAL VERSION V3-PROTOCOL SYNC-CONNECTION-TRACKING
;;; LAN
0 F vrrp1 ether1 00:00:5E:00:01:31 49 254 1s 3 ipv4 yes
;;; WAN
1 X vrrp2 ether4 00:00:5E:00:01:3B 59 100 1s 3 ipv4 noCode: Select all
[admin@MikroTik] > interface/vrrp/print
Flags: X - DISABLED; R - RUNNING; M - MASTER
Columns: NAME, INTERFACE, MAC-ADDRESS, VRID, PRIORITY, INTERVAL, VERSION, V3-PROTOCOL, SYNC-CONNECTION-TRACKING
# NAME INTERFACE MAC-ADDRESS VRID PRIORITY INTERVAL VERSION V3-PROTOCOL SYNC-CONNECTION-TRACKING
;;; LAN
0 RM vrrp1 ether1 00:00:5E:00:01:31 49 100 1s 3 ipv4 yes
;;; WAN
1 X vrrp2 ether3 00:00:5E:00:01:3B 59 200 1s 3 ipv4 no Re: VRRP with single WAN and Single LAN Address
It's the first time ever for me to see an F status of a VRRP interface on Mikrotik, I was unable to replicate it here using any settings I could think of.RTR 1Code: Select allFlags: X - DISABLED; F - FAILURE Columns: NAME, INTERFACE, MAC-ADDRESS, VRID, PRIORITY, INTERVAL, VERSION, V3-PROTOCOL, SYNC-CONNECTION-TRACKING # NAME INTERFACE MAC-ADDRESS VRID PRIORITY INTERVAL VERSION V3-PROTOCOL SYNC-CONNECTION-TRACKING ;;; LAN 0 F vrrp1 ether1 00:00:5E:00:01:31 49 254 1s 3 ipv4 yes
So please disable the VRRP interface that shows this status, enable logging of VRRP using /system logging add topics=vrrp, run /log print follow-only where topics~"vrrp", re-enable the VRRP interface, and wait until it shows the F status again. Then stop the /log print ... and post its output. Hopefully there will be something useful in it.
Re: VRRP with single WAN and Single LAN Address
If you don’t find any obvious reason in ROS for the VRRP state change, check the ESXi logs for the virtual NIC (referring to my previous post).
Re: VRRP with single WAN and Single LAN Address
I did get the flip flopping to stop by moving both to the same host. I can address that part with vmware in the future too.After discussing the issue internally with some techs, our best guess at this point is that the flip-flop behavior might be caused by a VMware Virtual Network Adapter 'state change' which can happen for various reasons like network congestion, resource constraints, virtual switch misconfigurations, bandwidth cap mgmt tools reconfiguring network settings, etc. This state change is signaled to the guest OS, causing ROS to flip VRRP state.
If possible, enable logging in ESXi on both instances and check the ROS logs for VRRP events, then sync the timestamps to see when and why the issues occur (make sure time is synchronized beforehand).
Re: VRRP with single WAN and Single LAN Address
Just curious if you happened to check the ESXi logs to find the root cause? Anyway, feel free to get back here if you find anything interesting for future reference.
Re: VRRP with single WAN and Single LAN Address
I did not. That is just a minor part of what I am trying to figure out. Pretty sure it has to do with the fact that I use 2 physical adapters for that simple switch and that is causing the issue. I ran into a similar issue with a Cisco ASA a while back.Just curious if you happened to check the ESXi logs to find the root cause? Anyway, feel free to get back here if you find anything interesting for future reference.
Still bashing my head on a wall trying to figure out how to use a single WAN IP address for the two routers I have created.
Re: VRRP with single WAN and Single LAN Address
The easiest is to use private addresses in the "WAN VRRP", and then NAT out the real public IP. Basically, the VRRP IP address does not have to be in the same subnet as the two members.Still bashing my head on a wall trying to figure out how to use a single WAN IP address for the two routers I have created.
If NAT is not desirable, I'm less sure what to suggest. I suppose you could use private address as the WAN IP on each, and make the VRRP address the public IP, with the real prefix (not /32) and route VRRP. Although this path I'm not as sure of myself, since at least in older RouterOS, using VRRP interface as an interface itself is slow, and there may be other artifacts in routing without NAT.
Re: VRRP with single WAN and Single LAN Address
At least in RouterOS 6, it was indeed possible that the address attached to the VRRP interface was totally unrelated to the address attached to the underlying "physical" one, so you could e.g. use private or APIPA addresses (169.254.x.y) to let the two devices talk VRRP to each other and a public address with a shorter prefix than /32 as the virtual one, so you could use dynamic routing protocols to advertise the router where the VRRP was currently in master mode as the gateway to that public subnet.
So also here, it should be possible to use just a single public address as the virtual one and let the devices negotiate which one will use it using private/APIPA addresses.
However, what I find much more problematic about VRRP than synchronizing multiple VRRP interfaces on the same device with one another is that a VRRP device does not become a master if the underlying physical interface stays up at L1 but the L2 path to the rest of the network is broken. Because if that happens, a VRRP interface always becomes a master as it stops receiving VRRP messages from the other VRRP routers, so if it also raises its priority on the other VRRP interface, it starts stealing the traffic and effectively blackholing it.
So once the WAN side VRRP interface becomes a master, it is very important to check that some canary destinations in the internet are indeed reachable through it before raising the LAN side VRRP interface priority to make it become a master.
On the "preferred master" router, the VRRP priority on both interfaces should be higher than on the "preferred backup" one and it should not be adjusted. On the "preferred backup", the fact that the WAN side VRRP becomes a master should trigger a check that internet is reachable through it and only if the result of that check is positive, the priority of the LAN side VRRP can be raised to make it beat the "preferred master" one. The same behavior is necessary also for the event that the LAN side VRRP becomes a master, but of course you have to add some additional checks to prevent a self-locking effect, i.e. each VRRP interface on the "preferred backup" side may only raise the priority of the other one if its own priority has not been raised yet.
So also here, it should be possible to use just a single public address as the virtual one and let the devices negotiate which one will use it using private/APIPA addresses.
However, what I find much more problematic about VRRP than synchronizing multiple VRRP interfaces on the same device with one another is that a VRRP device does not become a master if the underlying physical interface stays up at L1 but the L2 path to the rest of the network is broken. Because if that happens, a VRRP interface always becomes a master as it stops receiving VRRP messages from the other VRRP routers, so if it also raises its priority on the other VRRP interface, it starts stealing the traffic and effectively blackholing it.
So once the WAN side VRRP interface becomes a master, it is very important to check that some canary destinations in the internet are indeed reachable through it before raising the LAN side VRRP interface priority to make it become a master.
On the "preferred master" router, the VRRP priority on both interfaces should be higher than on the "preferred backup" one and it should not be adjusted. On the "preferred backup", the fact that the WAN side VRRP becomes a master should trigger a check that internet is reachable through it and only if the result of that check is positive, the priority of the LAN side VRRP can be raised to make it beat the "preferred master" one. The same behavior is necessary also for the event that the LAN side VRRP becomes a master, but of course you have to add some additional checks to prevent a self-locking effect, i.e. each VRRP interface on the "preferred backup" side may only raise the priority of the other one if its own priority has not been raised yet.
Re: VRRP with single WAN and Single LAN Address
I'm not sure the VRRP address has to be a /32 when it's NOT in same IP subnet?
Yeah... I had a good handle on how this all worked. And on LAN side, all the same. But with V7, the effects of the new routing engine on VRRP is just not well described and subtlety different. Now I don't use VRRP on WAN, but recall the pref-src= on routes didn't work the same when testing sync connections a while back - but could never really figure out a scheme to even use it on the WAN side.At least in RouterOS 6,
Re: VRRP with single WAN and Single LAN Address
So on the WAN interface we would use a private IP address to each router.At least in RouterOS 6, it was indeed possible that the address attached to the VRRP interface was totally unrelated to the address attached to the underlying "physical" one, so you could e.g. use private or APIPA addresses (169.254.x.y) to let the two devices talk VRRP to each other and a public address with a shorter prefix than /32 as the virtual one, so you could use dynamic routing protocols to advertise the router where the VRRP was currently in master mode as the gateway to that public subnet.
So also here, it should be possible to use just a single public address as the virtual one and let the devices negotiate which one will use it using private/APIPA addresses.
However, what I find much more problematic about VRRP than synchronizing multiple VRRP interfaces on the same device with one another is that a VRRP device does not become a master if the underlying physical interface stays up at L1 but the L2 path to the rest of the network is broken. Because if that happens, a VRRP interface always becomes a master as it stops receiving VRRP messages from the other VRRP routers, so if it also raises its priority on the other VRRP interface, it starts stealing the traffic and effectively blackholing it.
So once the WAN side VRRP interface becomes a master, it is very important to check that some canary destinations in the internet are indeed reachable through it before raising the LAN side VRRP interface priority to make it become a master.
On the "preferred master" router, the VRRP priority on both interfaces should be higher than on the "preferred backup" one and it should not be adjusted. On the "preferred backup", the fact that the WAN side VRRP becomes a master should trigger a check that internet is reachable through it and only if the result of that check is positive, the priority of the LAN side VRRP can be raised to make it beat the "preferred master" one. The same behavior is necessary also for the event that the LAN side VRRP becomes a master, but of course you have to add some additional checks to prevent a self-locking effect, i.e. each VRRP interface on the "preferred backup" side may only raise the priority of the other one if its own priority has not been raised yet.
VRRP2
RTR 1 = 192.168.1.40
RTR 2 = 192.168.1.50
VRRP Address = x.x.60.100 (WAN address for my internet connection)
In this setup the VRRP packets travel the .40 and .50 IPs to check keepalive. If the primary routers goes offiline it triggers the failover and the backup router would take over all routing NAT and forwarding duties, but until there is a failover the backup is in a shutdown/blocking state? I just want to make sure I don't accidentally create a loop or issue within even my test environment right now as I have some other stuff I am working on there as well.
As a secondary question would the setup I am trying to implement with using a second CHR as a back for the WAN connection be better accomplished using a script of some kind? I am completely unfamiliar with using scripts on Mikrotik, but it is the only other thing I could think of as a way to have a failover solution.
I appreciate the help everyone is giving me as well. Thank you.
Re: VRRP with single WAN and Single LAN Address
99.999 percent over my head, but I thought that when one leased a server for CHR, part of the deal was redundancy so that if the server failed, the CHR would automatically be migrated to another server etc....... ??
ex....
https://www.vultr.com/company/sla/
https://docs.vultr.com/high-availabilit ... ip-and-bgp
https://docs.vultr.com/benefits-of-impl ... plications
ex....
https://www.vultr.com/company/sla/
https://docs.vultr.com/high-availabilit ... ip-and-bgp
https://docs.vultr.com/benefits-of-impl ... plications
Re: VRRP with single WAN and Single LAN Address
To be precise: each of the two routers periodically sends a multicast VRRP "advertisement" from its individual 192.168.1.x address. The one that receives such advertisements whose priority field bears a higher value than its own goes to backup mode, which makes the VRRP interface appear to be down to its routing stack, so it does not use the x.x.60.100 address in this state. The router that doesn't receive these advertisements at all, or receives them with a lower value in the priority field than its own configured one, runs in master mode so its routing stack can see the VRRP interface as up.So on the WAN interface we would use a private IP address to each router.
VRRP2
RTR 1 = 192.168.1.40
RTR 2 = 192.168.1.50
VRRP Address = x.x.60.100 (WAN address for my internet connection)
In this setup the VRRP packets travel the .40 and .50 IPs to check keepalive. If the primary routers goes offiline it triggers the failover and the backup router would take over all routing NAT and forwarding duties, but until there is a failover the backup is in a shutdown/blocking state?
Some scripting is needed in any case if you want to let the master/backup state of the LAN side track the one of the WAN side and vice versa. But the bigger challenge here is that given the way it works, the VRRP switches to master mode also if it completely loses contact with the rest of the network via the underlying "physical" interface but that interface itself stays up. So if the WAN side VRRP on the"normally backup" machine becomes a master in this way and propagates that change to the LAN side, it will break things rather than saving them. So you need to "somehow" check that you actually can reach something in the internet via the WAN before you propagate the change to the LAN side. And the least complicated way to accomplish this is to use another public address for this test, assigned to the "physical" interface as an additional one (or, to be completely on a safe side, to another VRRP interface attached to the same physical one, to avoid that the machine decides to use that address as a source of the VRRP advertisements it sends). But it should be enough to do this check on the "normally backup" machine, so you only spend one more public address on this, not two.As a secondary question would the setup I am trying to implement with using a second CHR as a back for the WAN connection be better accomplished using a script of some kind? I am completely unfamiliar with using scripts on Mikrotik, but it is the only other thing I could think of as a way to have a failover solution.
Re: VRRP with single WAN and Single LAN Address
Indeed, if you run a machine in a 3rd party data center, this is exactly what you expect; however, the OP calls "datacenter" a bunch of servers in his own premises and it is his responsibility to provide redundancy. So an automatic migration of VMs from a failed host to another one may not be an option.I thought that when one leased a server for CHR, part of the deal was redundancy so that if the server failed, the CHR would automatically be migrated to another server etc....... ??
Re: VRRP with single WAN and Single LAN Address
I host the CHRs myself. In the final setup I will have one CHR at 1 location the other CHR at my other site. The two sites are interconnected and can share connectivity and yes I can easily spin up the CHR at either site in the event of a failure, but that can take some time to do depending on the type of failure. If I can get something going via VRRP or some kind of script that is able to failover automatically that is what I am looking for. This is for a video service my company offers and downtime is very much not a good thing.99.999 percent over my head, but I thought that when one leased a server for CHR, part of the deal was redundancy so that if the server failed, the CHR would automatically be migrated to another server etc....... ??
ex....
https://www.vultr.com/company/sla/
https://docs.vultr.com/high-availabilit ... ip-and-bgp
https://docs.vultr.com/benefits-of-impl ... plications
Re: VRRP with single WAN and Single LAN Address
Correct. Each of my datacenters as I use the term is a Cisco UCS deployment with 4 hosts in each of two locations. Each location can be used as a backup for the other in the event of a failure of one site. The four hosts at each site are in a cluster and managed by vCenter. In the event of a host failure at a site vCenter should automatically migrate the VMs on that host to one of the other three. But in the event of a failure of the entire UCS and having to failover to the other site I have to manually kick off a failover for my protected VMs. Hope that clarifies some of my physical setup.Indeed, if you run a machine in a 3rd party data center, this is exactly what you expect; however, the OP calls "datacenter" a bunch of servers in his own premises and it is his responsibility to provide redundancy. So an automatic migration of VMs from a failed host to another one may not be an option.I thought that when one leased a server for CHR, part of the deal was redundancy so that if the server failed, the CHR would automatically be migrated to another server etc....... ??
Re: VRRP with single WAN and Single LAN Address
d1lazarus,
i have read some of your previous posts, probably you have a wrong solution design, the wrong application for your streaming service requirements.
this...
if you were thinking about using that vrrp virtual ip for your streaming service access - then nope, that won't work if those 2 Datacenter in different subnet.
what you need is an layer 7 load balancer on each Datacenter - so need at least 2 public ips (don't need to be in the same subnet). have a read on haproxy. it can have vrrp as well - but you don't need that part. it will directly make your streaming service highly available on both sides. point those haproxy to your streaming servers (private ips, response will be routed back to the load balancer).
i have read some of your previous posts, probably you have a wrong solution design, the wrong application for your streaming service requirements.
this...
it probably won't work in real world. if those separate datacenters used different subnets - because the floating virtual ip for vrrp needs to be in the same subnet for the heartbeat.So on the WAN interface we would use a private IP address to each router.
VRRP2
RTR 1 = 192.168.1.40
RTR 2 = 192.168.1.50
VRRP Address = x.x.60.100 (WAN address for my internet connection)
if you were thinking about using that vrrp virtual ip for your streaming service access - then nope, that won't work if those 2 Datacenter in different subnet.
what you need is an layer 7 load balancer on each Datacenter - so need at least 2 public ips (don't need to be in the same subnet). have a read on haproxy. it can have vrrp as well - but you don't need that part. it will directly make your streaming service highly available on both sides. point those haproxy to your streaming servers (private ips, response will be routed back to the load balancer).