Multiple public IPs, different internal zones

GeneralMarmite · Wed Apr 24, 2024 9:44 pm

I've edited this because my initial write-up wasn't particularly clear. And I want to acknowledge that this topic seems to come up fairly frequently, and yet here I am writing another post. I took a look at this post, which seems to be the definitive VLAN guide, but what I can't seem to get done right in my setup is the layer 3 routing between zones. For example, their "Switch with a separate router (RoaS)" has all the layer 3 routing elsewhere. In my case, I have the RB4011 as the only smart device in the network. Everything plugged into it is either an edge device or a dumb switch. So I'm struggling to combine the examples in that post.

I'm using an RB4011iGS+ running RouterOS 7.14.3.

Router Diagram.png

One other thing that complicates my situation is that my ISP (Verizon Business FIOS) gives me 5 public IPs. I've represented them as 70.70.70.12-16 with default gateway 70.70.70.1. The connection comes in on an ordinary copper CAT5 plugged into ether1. It doesn't use PPPoE or any kind of authentication. Just plug it in and it works. It's not a /29 delegation or anything. It's just 5 allocated IPs out of a /24.

So all the examples have a single IP on a single egress port. I have 5 IPs on a single egress port and I need to associate them differently. (e.g., each VLAN or bridge has a specific set of public IP(s) for egress) All the examples have vlan tagging on switches, and then a separate router using those vlan tags to route. I have it all in one device. Finally, in my config, I'm using bridges without VLAN tagging. I could use VLAN tagging if that made sense.

In a reply to this post, I have a redacted copy of my config. What I'm trying to achieve is this:

                     Public                Private
          Public IP    Port    Private IP     Port
--------------------------------------------------
Zone A    70.70.70.12    22    172.30.0.3       22
                       8123    172.30.0.3     8123
Zone B    70.70.70.13    22    172.30.2.5       22
                         25    172.30.2.30      25
                         53    172.30.2.40      53
                         80    172.30.2.10      80
                        443    172.30.2.10     443
                        587    172.30.2.30     587
                        993    172.30.2.30     993
                       2222    172.30.2.55    2222
Zone C    70.70.70.14    22    172.30.4.4       22
                         25    172.30.4.5       25
                         53    172.30.4.9       53
                        587    172.30.4.5      587
                        993    172.30.4.5      993
                       2222    172.30.4.27    2222
Zone C    70.70.70.15    80    172.30.4.10      80
                        443    172.30.4.10     443
Zone C    70.70.70.16    80    172.30.4.12      80
                        443    172.30.4.12     443

I do plan to have some cross-zone traffic. As a general rule, cross-zone traffic should work like public traffic. That is, if 172.30.0.3 port 22 is available to the public, then it's available to the 172.30.2.0/24 and 172.30.4.0/24 networks also. All other cross-zone traffic should be denied. If I have to use public IPs and hairpin NAT because that is simplest, that's fine.

What's the right approach? Do I need to turn on VLANs? Route marks? How do I get the right srcnat and routing to match the design?

Thanks for all the help.

anav · Wed Apr 24, 2024 10:09 pm

So far all seems double but the problem I have is with your last zone.
How do you propose to assign some users to 14, some to 15 and some to 16?
We could do it with address lists I suppose but why the grouping.

Why not zone3 to 14, zone4 to 15, and zone5 to 16 for example.
OR
Alternatively Share (load balance all zone 3 user to the three WANIPs )

GeneralMarmite · Wed Apr 24, 2024 10:15 pm

How do you propose to assign some users to 14, some to 15 and some to 16?

It's not "users" that I'm assigning. It's systems. Individual internal IP addresses. I might, for whatever reason, assign 172.30.4.5 to be src-natted to 70.70.70.14 and I might assign 172.30.4.6 to src-nat to 70.70.70.15. Between IP forwarding, dst-nat, and src-nat rules, I think I can make that happen. I have done this easily in the past, when I had multiple public IPs, but i had a single zone. All public IPs and all internal hosts were on a single bridge, single DHCP pool, etc.

anav · Wed Apr 24, 2024 10:19 pm

Now you are making less sense.
Suggest
a. full config
/export file=anynameyouwish ( minus router serial number, any real public WANIP information, keys etc.)

b. requirements without solutions
i. identify all user(s)/devices(s) groups of users devices
ii. identify all traffic they require.

Include admin of course, and from that a config can be discerned.
Its still not clear what you are doing at zone 3 for example, with better defined requirements things get clearer faster.

GeneralMarmite · Thu Apr 25, 2024 1:30 am

I appreciate your effort to study it and understand and help.

The three zones are basically ZoneA: ordinary users doing wifi and ordinary user things. Just a couple publicly-exposed inbound services. Zone B: a group of systems providing typical internet services (mail, DNS, SSH, git). Zone C: a slightly more complex group of systems. 2 IPs used for load balancing web apps, and the third IP for the core services of mail, DNS, git, etc. Note that none of those things are built out yet. (You can see the one port 8123 in Zone A, but that's it). I'm still in the planning stages, so you won't see my attempts at trying to do port forwarding and dst-nat and stuff, because I don't have the basics working yet.

Screenshot 2024-04-24 at 18.28.12.png

I attached the full, messy, redacted export. The only thing I took out were static DHCP leases and serial numbers and stuff.

GeneralMarmite · Thu May 02, 2024 4:34 am

Now you are making less sense.

I both edited the original post to be clearer, and I've uploaded my full config. Hopefully this makes more sense now.

LdB · Thu May 02, 2024 6:24 am

You are completely overthinking it your provider gave you 5 /32 gateways and you have no idea if they come from a/24 that is a complete misunderstanding

The VLANs are just isolation and not relevant to the NATs

The source NAT is dead simple

/ip firewall nat
add action=src-nat chain=srcnat src-address=172.30.0.0/24 to-addresses=70.70.70.12
add action=src-nat chain=srcnat src-address=172.30.2.0/24 to-addresses=70.70.70.13
add action=src-nat chain=srcnat src-address=172.30.4.10/32 to-addresses=70.70.70.15
add action=src-nat chain=srcnat src-address=172.30.4.12/32 to-addresses=70.70.70.16
add action=src-nat chain=srcnat src-address=172.30.4.0/24 to-addresses=70.70.70.14

All you need to know is same as routing on a NAT a /32 is higher precedence than a /24
Any router that didn't carry subnet precedence would take the NATs in order
So the above will work whichever way it goes

/network precedence is a standard thing on routers

GeneralMarmite · Sat May 04, 2024 1:15 am

You are completely overthinking it your provider gave you 5 /32 gateways and you have no idea if they come from a/24 that is a complete misunderstanding

Thanks for the help. "Overthinking it" is almost certainly part of my problem. The instructions from the ISP say to use the address 70.70.70.12, use netmask of /24, and gateway of .1. You're right that I don't know if the allocation to the ISP is a /24, but that's the netmask they told me to use.

I changed the IP addresses for the router to be like this. So that should treat the 5 outbound IPs as /32s instead of /24s.

/ip address add address=70.70.70.12/32 interface=01ether-WAN network=70.70.70.0

.

The VLANs are just isolation and not relevant to the NATs

That's true. One problem I am having is an inability to route across the bridges. So I have internet -> LAN going fine, via port forwarding and dstnat on all 3 segments. I can't seem to get cross-segment routing working. I can't figure out how to get a random IP on one segment, say, 172.30.0.55 to connect to, say, 172.30.4.4 on port 22. I want to permit it, because 172.30.4.4 is the bastion for that segment. But I haven't figured out what IP input/forward/NAT/routing rules will make that possible.

.

All you need to know is same as routing on a NAT a /32 is higher precedence than a /24
Any router that didn't carry subnet precedence would take the NATs in order
So the above will work whichever way it goes

/network precedence is a standard thing on routers

This was really helpful. That was something I didn't know. Thank you!

LdB · Sat May 04, 2024 6:13 pm

They will have also given you a gateway for the 5 IPs which is where you send all outbound traffic to internet
It might be a /32 or an actual network (/31 /30 /29 etc) often called a transit link where they will give you there end and your end IP.

For a /32
The network IP is the /32 address they gave you as the gateway xxx.xxx.xxx.xxx

/ip address 
add address=70.70.70.12/32 interface=01ether-WAN network=xxx.xxx.xxx.xxx
add address=70.70.70.13/32 interface=01ether-WAN network=xxx.xxx.xxx.xxx
add address=70.70.70.14/32 interface=01ether-WAN network=xxx.xxx.xxx.xxx
add address=70.70.70.15/32 interface=01ether-WAN network=xxx.xxx.xxx.xxx
add address=70.70.70.16/32 interface=01ether-WAN network=xxx.xxx.xxx.xxx

For an actual network (transit link)
The /32 address the network IP is the same IP as the actual IP

/ip address 
add address=70.70.70.12/32 interface=01ether-WAN network=70.70.70.12
add address=70.70.70.13/32 interface=01ether-WAN network=70.70.70.13
add address=70.70.70.14/32 interface=01ether-WAN network=70.70.70.14
add address=70.70.70.15/32 interface=01ether-WAN network=70.70.70.15
add address=70.70.70.16/32 interface=01ether-WAN network=70.70.70.16

Now you anchor your end of the transit IP in an interface then add a static route for 0.0.0.0/0 to there IP

tdw · Sat May 04, 2024 8:56 pm

No. The OP states the provider supplies five IPs with a /24 netmask, these should just be added to the WAN ethernet interface with a single default route to the provided gateway.

All you need to know is same as routing on a NAT a /32 is higher precedence than a /24

No. You are conflating two things - more precise routes have precedence, firewall rules are processed in order.

In RouterOS v7 pref-src in routes only applies to traffic originated from the router, for traffic forwarded through the router you have to use NAT rules. This is a change from the behaviour in v6.

The routing rules other than the default are not required, src-nat rules for the outbound traffic from the various address ranges and dst-nat rules for the inbound traffic matching the appropriate public IP and port(s) should be sufficent. Usually with multiple WAN connections you have to use mangle rules to return inbound traffic to the correct gateway, as the gateway is the same in this case it is not necessary.

By default all subnets can communicate with each other, they are only prevented from doing so by having different routing tables which don't reference the target address and/or firewall drop rules.

LdB · Sun May 05, 2024 2:56 pm

Seriously NO and really NO !!!!

Go back and read the ISP gave him 5 /32 IPs he introduced the /24 and I pointed out he doesn't know that which he agreed.

He has clarified and as I expected they GAVE HIM A GATEWAY as a /32 as well

One other thing that complicates my situation is that my ISP (Verizon Business FIOS) gives me 5 public IPs. I've represented them as 70.70.70.12-16 with default gateway 70.70.70.1

No you can't assume it's a /24 he has to treat it as a /32 gateway unless they give him the network mask that is critical !!!!!!!!!!!
I doubt Horizon are going to burn 254 IPs for the 5 he wants or merge him with other customers

If they were going to use transit network you would have expected a /29 for 5 IPs that is sort of the tell we are likely got /32s all round

As IPv4 has got tighter many of us ISPs are using /32 not transit networks for transit so we don't burn IPs.
If there is a /32 gateway there may actually be a customer link on 70.70.70.255 which you want to broadcast on with your /24 and the ISP router will likely block you.

The rest of the problem with your answer is obvious if that gateway is a /32.

Multiple public IPs, different internal zones

Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones

Re: Multiple public IPs, different internal zones