Hello folks,

I have setup VLANs on my Mikrotik router, clients get addresses and can see each other, but from VLANs I cannot reach internet. I am obviously missing something, but my attempts with creating additional bridges etc. did not lead to success. Hence, I am hoping you can help me troubleshoot.

Setup:
Mikrotik Router acts as gateway. eht1 is GW interface with 10.201.1.1
Normal clients get addresses on 10.201.1.0/24 network (here all works fine)

eth2 & eth3 are bonded and go to my XenServer (10.201.1.5)
On XenServer I run several Virtual Machines. Goal is to have 3 servers as follows:
1) WEB -> in "DMZ" -> VLAN 100 (10.201.11.0/24 network)
2) FileStorage -> in VLAN 100 & in VLAN 200 (10.201.12.0/24 network; server VLAN only, no outside access)
3) Domain Controller -> in 10.201.1.0/24 network and in VLAN 200

Perhaps I do not need VLAN 200, as it could be in the 10.201.1.0 network, but that I can solve later.
I do not want (cannot) setup a regular DMZ by adding additional router, therefore this VLAN DMZ solution.

What I currently have is VLANs working and assigning IPs. Clients seeing each other on VLANs and correctly cannot see other clients on other VLAN, but no access to internet from VLANs.
How I achieved that (for clarity, I will describe only VLAN 100, as the other one is the same):
1) Added new VLAN interface under the "bond" interface called DMZ with VLAN set to 100
2) Under IP->Addresses assigned Address to this new interface 10.201.11.1/24 and network 10.201.11.0
3) Created a pool "dhcp_dmz" (...11.2-....11.254) and DHCP server assigned to DMZ interface

So far this works. Clients pinging each other. I know that the assigned gateway (10.201.11.1; which is the new DMZ interface) is not connected to internet. I somehow need to connect it with the working gateway (eth1 10.201.1.1) and perhaps setup routing. But I am stuck.

If you need a printout of some commands, please let me know.
Your help is greatly appreciated.

Cheers,
B.

First off, do you have a NAT setup permitting your LAN IPs to reach your WAN?


The NAT should be using your WAN interface/IP when it translates an internal LAN IP to go outside of your network. Which port is your WAN connected to?

Hello,

thanks for the reply. Yes, I have the NAT setup and it's working fine for several months now. That's also a reason I was not touching the "original" 10.201.1.0/24 network.

The WAN interface is on eth1-gateway. I'll try to describe interfaces:
eth1-gateway -> wan interface, on the LAN side the address is 10.201.1.1
eth2 -> LAN interface connected to iLO port of HP server
eth3 -> LAN connected to regular HP interface #1
eth4 -> LAN connected to regular HP interface #2
eth3+eth4 are bonded into single bond (interface) called HP ProLiant Bond (access to XenServer is on address 10.201.1.5)
eth5-eth8 -> several LAN clients and hubs/swiches are connected -> all are part of 10.201.1.0/24 network

(Edit: Perhaps worth to mention is that eth2-eth8 are bridged together with eth2 being master and eth3-eth8 slaves)
Once again, all NATing works fine for the above network.

Now, I have created 2 VLAN interfaces "under" the HP ProLiant Bond interface. Interface DMZ and interface LocalFS. Let's focus on DMZ only, as the other one is the same approach.

DMZ interface is assigned address 10.201.11.1 and VLAN id of 100. On this interface I have setup DHCP with appropriate pool, which assigns addresses in 10.201.11.2-254 range with gateway 10.201.11.1

On XenServer I have created also 2 new interfaces for the above mentioned VLANs. I can assign these interfaces then to virtual machines running on XenServer. This works, as my machines do get IPs from the correct DHCP server (if I specify regular Bond interface, then I get 10.201.1.0/24 address, if I choose VLAN interface, I get 10.201.11.0/24 address). Clients on XenServer on the same VLAN can see and ping each other. They can't ping accross VLANs (good, I wanted this), but they can't access 10.201.1.1 either.

Thanks in advance for your suggestions.

Cheers,
B.

OK, here is an update. I have a small progress.
My VLAN clients now can reach the internet. The problem was that when client (VM) had more than 1 interface, I was assigning multiple gateways. And since my NAT is not VLAN based, it is routing any subnet I throw at it.

So that works.

Last piece missing and I am unable so far to troubleshoot it is the interconnection between clients. Friend told me, that I should be able to ping clients between both networks (10.201.1.0/24 and 10.201.11.0/24) and if I want to limit access then I should use firewall rules based on VLAN.

After running few tests, I was able to conclude:
Clients connected to Mikrotik router -> 10.201.1.0/24 -> can ping each other and can ping also 10.201.11.1 (as this is the virtual DMZ interface on Mikrotik).
Clients on XenServer (Virtual machines) -> can ping each other regardless of which network they belong to. So 1.0/24 can ping 11.0/24 and vice-versa.
The only problem is when client connected to Mikrotik tries to ping a VM client in VLAN on XenServer. Same goes if I do it the other way. But clients on XenServer can ping the DMZ virtual interface (10.201.11.1) on Mikrotik.

Well, if you have any tips / ideas, I will be thankful.

Cheers,
B.

Hello,

The WAN interface is on eth1-gateway. I'll try to describe interfaces:
eth1-gateway -> wan interface, on the LAN side the address is 10.201.1.1
eth2 -> LAN interface connected to iLO port of HP server
eth3 -> LAN connected to regular HP interface #1
eth4 -> LAN connected to regular HP interface #2
eth3+eth4 are bonded into single bond (interface) called HP ProLiant Bond (access to XenServer is on address 10.201.1.5)
eth5-eth8 -> several LAN clients and hubs/swiches are connected -> all are part of 10.201.1.0/24 network

(Edit: Perhaps worth to mention is that eth2-eth8 are bridged together with eth2 being master and eth3-eth8 slaves)
Once again, all NATing works fine for the above network.

Now, I have created 2 VLAN interfaces "under" the HP ProLiant Bond interface. Interface DMZ and interface LocalFS. Let's focus on DMZ only, as the other one is the same approach.

I assume you use the switch functionality on the RB (eth2-eth8 are bridged together with eth2 being master and eth3-eth8 slaves). If this is the case, then you have to set up the LAN subnet 10.201.1.0/24 on eth2 and HP ProLiant Bond - which is of course not working.

An approach would be to bridge eth2 + Bond and assign the 10.201.1.1 on the bridge interface? I'm not sure if eth3/4 still can be slaves.

Dirk

Thanks Dirk for the suggestion.
Maybe I am using wrong terminology - yes, I am using the switch functionality.

After examining the router based on your input, the situation is as follows:
eth2 and Bond are already bridged together on "bridge-local"
eth3 & eth4 -> they do not have Master port set (and can't be assigned, as it gives error, that they are already part of the Bond interface).

How do I assign address to bridge interface? I can't find that option in GUI.
If I assign the 10.201.1.1 to the bridge, what do I assign to eth1-gateway interface?

Thanks in advance.
Cheers,
B.

How do I assign address to bridge interface? I can't find that option in GUI.
If I assign the 10.201.1.1 to the bridge, what do I assign to eth1-gateway interface?

eth1-gateway should be whatever the WAN IP address is - this interface is not bridged or switched, and no device other than the router should be attached to the Internet feed, right?

So - 10.201.1.1/24 should be on the bridge-local interface.
When you do that, anything connected to ether2, ether5-8, and the bond interface UNTAGGED (no vlan) will be able to communicate through the router at layer 2 - broadcasts, etc will all just be forwarded like a switch. (and will be hardware switched if the devices on e2, and e5-8 communicate directly with each other)

Okay - and the question becomes - do you want to take one of the trunk VLANs and drop it out on a physical port, or do you just want those to exist only inside the virtual environment and the Mikrotik?
If VLANs don't need to come out into the real world, then just put the VLAN interfaces on the bonding interface as you've done, put whatever IP address is needed on the VLAN interfaces, and there will be layer3 (routing) connectivity between those networks and the 10.201.1.x LAN network.

If you need to drop VLAN 50 (for example) out on a physical interface as an untagged port - then you'll need to create another bridge - e.g. bridge50, remove one of the physical ports from the switch, e.g. ether8, and then add the VLAN50 interface and ether8 as ports on bridge50. Finally, you would move the IP address of the Mikrotik off of the VLAN interface and onto the bridge interface.

Note that whenever you move these IP addresses onto a bridge interface, you may need to go modify your firewall rules. If a rule says out-interface=ether2 then change that to out-interface=bridge-lan (or whatever the LAN bridge's name is). If you move an IP from vlan50 onto bridge50 - then you'll need to modify all rules that reference the interface. References by IP won't need to be updated.

Thanks ZeroByte,

I will have to digest your answer and go slowly line by line experimenting.
In the meanwhile I was watching traffic with torch and maybe found where the problem is, but don't know what is the fix

I am pinging from VM 10.201.11.6 (in XenServer) to a regular PC connected to Mikrotik 10.201.1.151
Now, the route should be: VM -> (switch in XenServer-can't torch that) -> DMZ virtual interface -> Bond interface -> bridge-local -> one of the ethx ports...

What I see is a icmp packet with VLAN tag 100 going all the way to bridge local. I did not investigate further, because immediately noticed that the 10.201.1.151 client is trying to reply to 10.201.11.6
However, that reply makes it only to the bridge-local interface and disappears (does not go to Bond interface, does not go to DMZ interface [logically, first must pass Bond], and neither gets routed out via eth1).
I guess the problem is that the reply never gets the VLAN tag 100, and I think somewhere it should. But I have no idea how to do that.

If that sheds some light on the issue, please let me know what needs to be fixed.
In the meanwhile I will go trying to figure out ZeroByte's answer

Thanks!
Cheers,
B.

OK ZeroByte,

went setp-by-step through your instructions and:
eth1 -> has wan ip assigned (as you suggest)
local-bridge -> has the 10.201.1.1/24 as you expected
(DMZ virtual port has 10.201.11.1/24 assigned if it matters)

And answer to your question is: I need the VLAN to communicate between virtual environment and Mikrotik. No need to "drop it on a physical port".

(My ultimate solution which I am trying to achieve is to have DMZ via VLAN and I will use filters than so that access from DMZ VLAN to regular LAN is very strictly limited and all external access [web, ftp, etc.] is directed at server which is in DMZ VLAN.)

Now, according to your instructions, I should have that setup working. And it sort of does, except the access from LAN back to VLAN. Please read my post above.

Thanks for your help so far.

Cheers,
B.

... I should have that setup working. And it sort of does, except the access from LAN back to VLAN. Please read my post above.

This sounds like a firewall rule to me - or perhaps a mangle rule that's setting a packet into a routing table or something like that. I'm assuming that you haven't tried anything that fancy yet.

If you're seeing the pings and replies at the 1.151 and the destinations of the replies are to 11.6, then you probably have a filter rule that comes before any "allow established/related connections" rule.

You can make a test by putting two temporary rules right at the top of the forward chain of the filter rules:
accept src-address=192.168.11.6
accept dst-address=192.168.11.6

You should see counters increment on both rules as pings are successfully forwarded.

Instead of using torch, try doing a packet capture on the bonding interface, and save the results to a file. After the packet capture is run, download the file and open it in wireshark. look for icmp packets and see whether you see both the request and the reply, or just the request and then a reply on the wrong VLAN, or a bunch of ARP requests for 11.6 but no ICMP ping replies...

Well,

what do you know... Mea culpa.
I hate coincidences. The 10.201.1.151 host turns out to be hooked up to another switch (ZyXel router in a switch mode) which aparently discard the VLAN info. Since I work remotely on that network, I couldn't have known. I used that host because it was always on and I was sure it responds to pings...

But after trying everything else out, I did a flood ping on the 10.201.1.0 network and found few more devices, all of them responding to ping requests coming from 10.201.11.0 VLAN.

So, maybe the setup was correct from the beginning

Anyway, thanks a lot for your help. Now I need to go and figure out how to setup firewall to allow only few very specific connections from VLAN to LAN, but that's a different topic.

Thanks again.
Cheers,
B.