Hello
it seems to me that I read most of the posts related to ntp and vrf but failed to achieve the goal: to indicate another interface for NTP not in the main table but in another vrf table . I tried to use Magle but no success. Normal NAT turns ip but does not change the output interface.
I've run out of ideas. Could someone help me?
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
Hi,
you should define a static default-route in the main table. E.g. your ISP gives you gateway=192.168.178.1/24 and the WAN interface is ether1, lying in VRF Green
then the route is
/ip route add dst-address=0.0.0.0/0 gateway=192.168.178.1/24%ether1
Greetings
UPDATE: a generic solution (workaround) is provided in post #20
you should define a static default-route in the main table. E.g. your ISP gives you gateway=192.168.178.1/24 and the WAN interface is ether1, lying in VRF Green
then the route is
/ip route add dst-address=0.0.0.0/0 gateway=192.168.178.1/24%ether1
Greetings
UPDATE: a generic solution (workaround) is provided in post #20
Last edited by heribertos on Tue Feb 26, 2019 10:22 am, edited 1 time in total.
Re: NTP in VRF
@heribertos:
thank you for answering, but you have not understood my problem.
More specifically:
1.I have one VRF mgmt, one of networks in VRF mgmt have an NTP server
2. The rest of the network is separated from mgmt is in main table.
I know that when i use NTP client , service search only in main table and try to match dst address to network in that table. I must get NTP time from VRF mgmt .
thank you for answering, but you have not understood my problem.
More specifically:
1.I have one VRF mgmt, one of networks in VRF mgmt have an NTP server
2. The rest of the network is separated from mgmt is in main table.
I know that when i use NTP client , service search only in main table and try to match dst address to network in that table. I must get NTP time from VRF mgmt .
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
Hi,
thanks for clarification, but I believe the solution concept remains unchanged. Instead of default-route you add the route to the NTP Server. The special Gateway syntax
gateway=w.x.y.z%ether1
says, that the route is to be resolved at ether1 which is in VRF management, in your case. In case of p2p you just specify the interface ether1.
I am using this solution for NTP via a VRF and it works. If it works not for you then I have definitly not understood the problem.
Observe the percentage character % in front of the interface name: ip-address%interface-name
Greetings
thanks for clarification, but I believe the solution concept remains unchanged. Instead of default-route you add the route to the NTP Server. The special Gateway syntax
gateway=w.x.y.z%ether1
says, that the route is to be resolved at ether1 which is in VRF management, in your case. In case of p2p you just specify the interface ether1.
I am using this solution for NTP via a VRF and it works. If it works not for you then I have definitly not understood the problem.
Observe the percentage character % in front of the interface name: ip-address%interface-name
Greetings
Re: NTP in VRF
You solution works but its not isolate networks in VRF mgmt. Because its main goal i cant let packect etc. for leaking trought from main table.
Cisco have beter solution - you simlpy indicate ntp source interface and that should be for separation - not putting route in main table.
Cisco have beter solution - you simlpy indicate ntp source interface and that should be for separation - not putting route in main table.
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
OK, I understood now your intention.
My idea in this case is: do not use the route in main table I suggested but a mangle rule in OUTPUT-chain like
Chain output
match dst-address=NTP-Server
action routing-mark=mgmt
The respone from NTP-server goes "automatically" even in VRF magmt into the INPUT-chain. This should also work and suites better your needs.
My idea in this case is: do not use the route in main table I suggested but a mangle rule in OUTPUT-chain like
Chain output
match dst-address=NTP-Server
action routing-mark=mgmt
The respone from NTP-server goes "automatically" even in VRF magmt into the INPUT-chain. This should also work and suites better your needs.
Re: NTP in VRF
@heribertos
You have absolutely right ! But it doesnt work i try this on begining of unfair fight with this . In result of setup mangle - output chain is that no packet was send via physical output interface ( checked by wireshark on pc ) . I gues that if there is no route (or rule ) in main table connection wont be initiated
You have absolutely right ! But it doesnt work i try this on begining of unfair fight with this . In result of setup mangle - output chain is that no packet was send via physical output interface ( checked by wireshark on pc ) . I gues that if there is no route (or rule ) in main table connection wont be initiated
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
I just tested the idea in my environment and it is working, but you have to make sure that your NTP-Server has a route back to the destination, which ROS uses as Source Address.
ROS uses any IP from an interface on your Router, and this might be from another VRF or Main, not from VRF Mgmt.
I checked this in Firewall/Connections port 123. I saw that the src-address used by ROS has no route on the router where my NTP is. I added the route and it works.
Your observation is true that there must be at least one default-route in main. Even if it is not used by (in this case) NTP Client. This is really strange. OK, if required it can be blocked in Firewall.
Greetings
ROS uses any IP from an interface on your Router, and this might be from another VRF or Main, not from VRF Mgmt.
I checked this in Firewall/Connections port 123. I saw that the src-address used by ROS has no route on the router where my NTP is. I added the route and it works.
Your observation is true that there must be at least one default-route in main. Even if it is not used by (in this case) NTP Client. This is really strange. OK, if required it can be blocked in Firewall.
Greetings
Re: NTP in VRF
Code: Select all
# feb/25/2019 08:41:11 by RouterOS 6.41.2
/interface vlan
add interface=ether1 name=vlan92 vlan-id=92
add interface=ether1 name=vlan453 vlan-id=453
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=X.Y.Z.8/24 interface=vlan92 network=X.Y.Z.0
add address=X.Y.Z.146/30 interface=vlan453 network=X.Y.Z.144
add address=X.Y.Z.7/24 network=X.Y.Z.0
/ip firewall filter
add action=accept chain=input src-address=X.Y.0.0/16
add action=accept chain=input src-address=X.Y.Z.1
add action=drop chain=input
add action=accept chain=input disabled=yes
/ip firewall mangle
add action=mark-routing chain=output dst-address=X.Y.Z.1 log=yes \
new-routing-mark=mgmt passthrough=yes
/ip route
add distance=1 dst-address=X.Y.0.0/16 gateway=X.Y.Z.1 pref-src=X.Y.Z.8 \
routing-mark=mgmt
add distance=1 gateway=X.Y.Z.145
/ip route rule
add action=lookup-only-in-table src-address=X.Y.Z.8/32 table=mgmt
add action=lookup-only-in-table disabled=yes dst-address=X.Y.Z.1/32 table=\
mgmt
add action=lookup-only-in-table interface=vlan92 table=mgmt
add action=unreachable routing-mark=main
add action=unreachable routing-mark=mgmt
add action=lookup-only-in-table dst-address=X.Y.Z.11/32 table=mgmt
/ip route vrf
add interfaces=vlan92 routing-mark=mgmt
/radius
add address=X.Y.Z.11 secret=PASSWORD service=login
/system identity
set name=sw-torun-test
/system ntp client
set enabled=yes primary-ntp=X.Y.Z.1
/user aaa
set default-group=full interim-update=30s use-radius=yes
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
Hi,
I tried to understand your network topology. What I would like to comment is:
If x.y.z. in the ip-addresses means always the same then you are working with overlapping ip-addresses in your VLANs.
I know that VRFs are advertised as capable of handling overlapping ip-addresses, but this is only true as long as no routing decision has to be taken.
The rule is therfore that hosts from overlapping ip-addresses shall not communicate with eachother nor shall they communicate with shared services.
This is somehow logical, because the VRF is associated based on ingress interface whereas routing decision is concerned with output interface, at the end of the day (and with gateway reachable there). Identical dst-addresses but with different output interfaces cannot be handled. Routing Distinguishers are not evaluated in Routing Tables.
ROS is a shared service, as MikroTik ROS services are not VRF aware. If ip-addresses on vlan453 must stay as they are my only idea is to define several smaler subnets on vlan 92 instead of 1 big /24.
E.g. with /28 and may be some /30 subnets you can cover the /24 space exept a gap for beeing used on vlan 453. The hosts may stay as they are configured with /24. Of course, hosts with ip-address from the gap will no longer exist in vlan 92.
Other comments:
/ip address third line, there is no interface specified.
/ip filter some redundant rule. first rule covers second rule, then you drop, then accept (ok is disabled)
is fast track disabled? should be (at least in the beginning)
/ip route rules
some are redundant to mangle, some are conflicting w.r.t. overlapping ip-addresses. disable all for test purposes, Later you can enable the "unreachable" rules.
Is the default-route with gateway x.y.z.145 resolved correctly in the routing table?
I tried to understand your network topology. What I would like to comment is:
If x.y.z. in the ip-addresses means always the same then you are working with overlapping ip-addresses in your VLANs.
I know that VRFs are advertised as capable of handling overlapping ip-addresses, but this is only true as long as no routing decision has to be taken.
The rule is therfore that hosts from overlapping ip-addresses shall not communicate with eachother nor shall they communicate with shared services.
This is somehow logical, because the VRF is associated based on ingress interface whereas routing decision is concerned with output interface, at the end of the day (and with gateway reachable there). Identical dst-addresses but with different output interfaces cannot be handled. Routing Distinguishers are not evaluated in Routing Tables.
ROS is a shared service, as MikroTik ROS services are not VRF aware. If ip-addresses on vlan453 must stay as they are my only idea is to define several smaler subnets on vlan 92 instead of 1 big /24.
E.g. with /28 and may be some /30 subnets you can cover the /24 space exept a gap for beeing used on vlan 453. The hosts may stay as they are configured with /24. Of course, hosts with ip-address from the gap will no longer exist in vlan 92.
Other comments:
/ip address third line, there is no interface specified.
/ip filter some redundant rule. first rule covers second rule, then you drop, then accept (ok is disabled)
is fast track disabled? should be (at least in the beginning)
/ip route rules
some are redundant to mangle, some are conflicting w.r.t. overlapping ip-addresses. disable all for test purposes, Later you can enable the "unreachable" rules.
Is the default-route with gateway x.y.z.145 resolved correctly in the routing table?
Re: NTP in VRF
Ok make some clarification ( i know that 999 doesnt exist in ip add ) in config :
Ip rule overlaping with magle chain it is my way to achieve the goal, not necessarily correct and desirable if i want completely separate in VRF mgmt - buy why magle doesnt work ?
Code: Select all
# feb/25/2019 08:41:11 by RouterOS 6.41.2
/interface vlan
add interface=ether1 name=vlan92 vlan-id=92
add interface=ether1 name=vlan453 vlan-id=453
/ip address
add address=10.999.1.8/24 interface=vlan92 network=10.999.1.0
add address=77.999.999.146/30 interface=vlan453 network=77.999.999.144
/ip firewall filter
add action=accept chain=input src-address=172.999.0.0/16
add action=accept chain=input src-address=10.999.1.1
add action=drop chain=input
/ip firewall mangle
add action=mark-routing chain=output dst-address=10.999.1.1 log=yes \
new-routing-mark=mgmt passthrough=yes
/ip route
add distance=1 dst-address=172.999.0.0/16 gateway=10.999.1.1 pref-src=10.999.1.8 \
routing-mark=mgmt
add distance=1 gateway=77.999.999.145
/ip route rule
add action=lookup-only-in-table src-address=10.999.1.8/32 table=mgmt
add action=lookup-only-in-table disabled=yes dst-address=10.999.1.1/32 table=\
mgmt
add action=lookup-only-in-table interface=vlan92 table=mgmt
add action=unreachable routing-mark=main
add action=unreachable routing-mark=mgmt
add action=lookup-only-in-table dst-address=172.999.1.11/32 table=mgmt
/ip route vrf
add interfaces=vlan92 routing-mark=mgmt
/radius
add address=172.999.1.11 secret=mt109318 service=login
/system identity
set name=sw-torun-test
/system ntp client
set enabled=yes primary-ntp=10.999.1.1
/user aaa
set default-group=full interim-update=30s use-radius=yes
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
Do you mean the posted configuration is working?
I would disable all routing rules (better delete them) and then check:
Are packets counted in mangle rule? -> it is used
which ip-address is NTP-Client using? Observe port 123 in firewall/connections. it is visible for a few seconds.
If this works, what about the route back from NTP Server?
What happens, if you disable VRF mgmt? Is NTP working?
I would disable all routing rules (better delete them) and then check:
Are packets counted in mangle rule? -> it is used
which ip-address is NTP-Client using? Observe port 123 in firewall/connections. it is visible for a few seconds.
If this works, what about the route back from NTP Server?
What happens, if you disable VRF mgmt? Is NTP working?
Re: NTP in VRF
Ok ,
When VRF, default gw is active i delete all rules , mangle have some hits like this :
and dont work - But this is what i do whant avoid .
When i disable vrf , mangle have hits like this :
and works
When VRF, default gw is active i delete all rules , mangle have some hits like this :
Code: Select all
output : in : (unknown 0 ) out:vlan453, proto UDP, 77.999.999.146:123->10.999.1.1:123, len 76When i disable vrf , mangle have hits like this :
Code: Select all
output : in : (unknown 0 ) out:vlan92, proto UDP,10.999.1.8:123->10.999.1.1:123, len 76-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
Hi, what means in your ip-addresses 999.
You can use only 0 ... 255 in your addresses.
Is that realy configured or are you hiding something?
You can use only 0 ... 255 in your addresses.
Is that realy configured or are you hiding something?
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
It goes into the wrong VLAN. There is no physical interface with ip-address? Only VLAN?
Try instead of mangle routing rule just for test: src-address=77.999.999.146 action=lookup routing-mark=mgmt
and/or specify in mangle as match property src-address=77.999.999.146
Try instead of mangle routing rule just for test: src-address=77.999.999.146 action=lookup routing-mark=mgmt
and/or specify in mangle as match property src-address=77.999.999.146
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
I could imagine that the ip-address is taken by ROS from the VRF or main, where the default-route is resolved.
For test purposes, change default-route to
/ip route dst-address=0.0.0.0/0 gateway=VLAN92
or
/ip route dst-address=0.0.0.0/0 gateway=10.999.1.8%VLAN92
(with character % in front of VLAN92)
In VRF mgmt a default-route pointing back could work, but that is for later
For test purposes, change default-route to
/ip route dst-address=0.0.0.0/0 gateway=VLAN92
or
/ip route dst-address=0.0.0.0/0 gateway=10.999.1.8%VLAN92
(with character % in front of VLAN92)
In VRF mgmt a default-route pointing back could work, but that is for later
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
deleted
Last edited by heribertos on Tue Feb 26, 2019 8:44 am, edited 1 time in total.
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
My last ideas:
I played with the default route in table main, and it is influencing the src-address NTP client uses:
- loopback interface in VRF gateway=loopback-interface -> IP of loopback is used! -> is working
- ether1 is in my config a port of a bridge and bridge is in VRF, ether1 does not carry any ip-address. gateway=ether1 -> ip of bridge in VRF is used -> is working
In all cases the mangle rule was essential: with rule -> working & without rule -> not working
So for you: define loopback interface (bridge without ports), set unique IP-address as you want, add loopback interface to VRF.
Set default-route in main with gateway=loopback-interface (not the ip-address, then route is unreachable!!!)
Dont forget back route at NTP Server to ip-address of loopback-interface.
I tried also valid route in table main which can resolve the destination and it worked:
So my conclusion is from this:
When ROS forms the ip-packet it needs to know via which interface the packet will leave the router in order to select the right ip-address.
This is done in the routing table main, where must exist a valid route to the desired destination. Default route is not required.
After Routing Decision (see Manual Packet Flow) the ip-packet travels through the output-chain, is subject to mangle rules and so on and finaly Routing Rules ar applied. Mangle Rule and Routing Rule may change the by ROS assumed interface, but a packet destined for ROS may arrive at any interface, so it is no problem.
Without a valid route in table main nothing is working
I played with the default route in table main, and it is influencing the src-address NTP client uses:
- loopback interface in VRF gateway=loopback-interface -> IP of loopback is used! -> is working
- ether1 is in my config a port of a bridge and bridge is in VRF, ether1 does not carry any ip-address. gateway=ether1 -> ip of bridge in VRF is used -> is working
In all cases the mangle rule was essential: with rule -> working & without rule -> not working
So for you: define loopback interface (bridge without ports), set unique IP-address as you want, add loopback interface to VRF.
Set default-route in main with gateway=loopback-interface (not the ip-address, then route is unreachable!!!)
Dont forget back route at NTP Server to ip-address of loopback-interface.
I tried also valid route in table main which can resolve the destination and it worked:
So my conclusion is from this:
When ROS forms the ip-packet it needs to know via which interface the packet will leave the router in order to select the right ip-address.
This is done in the routing table main, where must exist a valid route to the desired destination. Default route is not required.
After Routing Decision (see Manual Packet Flow) the ip-packet travels through the output-chain, is subject to mangle rules and so on and finaly Routing Rules ar applied. Mangle Rule and Routing Rule may change the by ROS assumed interface, but a packet destined for ROS may arrive at any interface, so it is no problem.
Without a valid route in table main nothing is working
Last edited by heribertos on Tue Feb 26, 2019 8:48 am, edited 1 time in total.
Re: NTP in VRF
thanks for time spend on my issue. We have same conclusion about issue that without route in table or proper rule separation in VRF doent work at all.
Why in ROS we can not chosse output interface for internal services - that isnt profesional looking way to do what you what to do .
Eh once again thanks , i owe'u beer ,
Why in ROS we can not chosse output interface for internal services - that isnt profesional looking way to do what you what to do .
Eh once again thanks , i owe'u beer ,
-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
It was me a pleasure. I gained also deeper insight into this subject.
I would like to share my conclusions:
The decision chain for traffic originated at ROS is:
Routing Decision → Mangle (Output Chain) → Filter → Routing-Adjustment → Routing Rule
The Routing Decision requieres an active route which matches the destination (in our example the NTP Server). A default route is not required. From the resolved route ROS knows the egress interface and takes the ip-address defined on that interface (pref-source) as src-address. This procedure is required in order to accept or discard the reply packets.
The mangle rule may redirect traffic to an VRF, although Routing Decision has been already taken, and this is implemented in Routing-Adjustment.
Finally, Routing Rules may override everything. If e.g. traffic shall leave router at VRF1 and mangle rule redirects to VRF2, but Routing Rule redirects to VRF1, than it is working.
From this conclusions I derived a generic concept to control the ROS traffic. I defined a fake VRF, called FAKE, which contains just 1 loopback-interface with name loFAKE and without any routes inside. In the main table I placed a default-route with gateway=loFAKE (not the IP-Address on loFAKE). So, ROS always takes IP-Address on loopback-interface loFAKE as src-address. Mangle rules then redirect to the desired VRF. Routing Rules are used for other purposes like redirecting Winbox Traffic based on a IP-Address of an interface inside VRF, which ROS will use as response src-address. I implemented this concept last night and it works fine, at least for me. It may of course be modified to special needs.
It is a pitty that ROS does not offer VRF aware ROS services. But as we have now understood the issue we can build suitable workarounds.
Remark: As we use the VRF FAKE for resolving VRF related routes, we can use without any risk specific routes in the main table
/ip route
add dst-address=0.0.0.0/0 gateway=VLAN453
add dst-address=IP-of-NTP-Server gateway=loFAKE
add dst-address=192.168.23.0/24 gateway=10.0.0.1 comment=remote subnet at sstp-tunnel
VPN-client adds
dst-address=10.0.0.1 gateway=sstp1-in
The one and only client who can gain access to NTP-Server by these routes is the NTP-Client, because VRF FAKE is empty and we use mangle in outout-chain for redirecting traffic. This is in contrast to Routing Rules, where a client with spoofed src-address could gain access.
Greetings
I would like to share my conclusions:
The decision chain for traffic originated at ROS is:
Routing Decision → Mangle (Output Chain) → Filter → Routing-Adjustment → Routing Rule
The Routing Decision requieres an active route which matches the destination (in our example the NTP Server). A default route is not required. From the resolved route ROS knows the egress interface and takes the ip-address defined on that interface (pref-source) as src-address. This procedure is required in order to accept or discard the reply packets.
The mangle rule may redirect traffic to an VRF, although Routing Decision has been already taken, and this is implemented in Routing-Adjustment.
Finally, Routing Rules may override everything. If e.g. traffic shall leave router at VRF1 and mangle rule redirects to VRF2, but Routing Rule redirects to VRF1, than it is working.
From this conclusions I derived a generic concept to control the ROS traffic. I defined a fake VRF, called FAKE, which contains just 1 loopback-interface with name loFAKE and without any routes inside. In the main table I placed a default-route with gateway=loFAKE (not the IP-Address on loFAKE). So, ROS always takes IP-Address on loopback-interface loFAKE as src-address. Mangle rules then redirect to the desired VRF. Routing Rules are used for other purposes like redirecting Winbox Traffic based on a IP-Address of an interface inside VRF, which ROS will use as response src-address. I implemented this concept last night and it works fine, at least for me. It may of course be modified to special needs.
It is a pitty that ROS does not offer VRF aware ROS services. But as we have now understood the issue we can build suitable workarounds.
Remark: As we use the VRF FAKE for resolving VRF related routes, we can use without any risk specific routes in the main table
/ip route
add dst-address=0.0.0.0/0 gateway=VLAN453
add dst-address=IP-of-NTP-Server gateway=loFAKE
add dst-address=192.168.23.0/24 gateway=10.0.0.1 comment=remote subnet at sstp-tunnel
VPN-client adds
dst-address=10.0.0.1 gateway=sstp1-in
The one and only client who can gain access to NTP-Server by these routes is the NTP-Client, because VRF FAKE is empty and we use mangle in outout-chain for redirecting traffic. This is in contrast to Routing Rules, where a client with spoofed src-address could gain access.
Greetings
Code: Select all
# generic template for controling ROS Services
#
# define loopback-interface "loFAKE" and set ip-address
#
/ip bridge
/interface bridge
add fast-forward=no name=loFAKE
/ip address
add address=10.11.11.10 interface=loFAKE network=10.11.11.10
#
# define VRF "FAKE"
#
/ip route vrf
add routing-mark=FAKE interface=loFAKE
#
# define default-route in table main using FAKE
#
/ip route
add dst-address=0.0.0.0/0 gateway=loFAKE
#
# define all mangle rules to control ROS Services
#
/ip firewall mangle
add chain=output src-address=10.11.11.10 dst-address=192.168.150.1 action=mark-routing new-routing-mark=mydesiredVRF
#
# set at destination a route to 10.11.11.10 to any interface of ROSRe: NTP in VRF
In my opinion i must say that your solution is more complicated but have same level of security of leakage as :
Code: Select all
/ip route rule
add action=lookup-only-in-table dst-address=IP-of-NTP-Server table=mydesiredVRF-
-
heribertos
just joined
- Posts: 19
- Joined:
Re: NTP in VRF
I dont agree about the same security levels.
In the beginning, when I proposed a route to your NTP-Server in the main table you refused that solution for security reasons.
I am confused, because this is exactly what you are doing now, at the and of the day.
Routing Rules are applied to any routed traffic, and not only to output-chain traffic Any client who is able to send to your router an ip-packet with destination NTP-Server via an interface (routed in main table, but this is not mandatory) gets the packet routed to your NTP-Server. Whereas in my solution ip-packets only originated by ROS are routed to the NTP-Server. This extends and maintains the traffic separation we expect from VRFs to the ROS.
In the beginning, when I proposed a route to your NTP-Server in the main table you refused that solution for security reasons.
I am confused, because this is exactly what you are doing now, at the and of the day.
Routing Rules are applied to any routed traffic, and not only to output-chain traffic Any client who is able to send to your router an ip-packet with destination NTP-Server via an interface (routed in main table, but this is not mandatory) gets the packet routed to your NTP-Server. Whereas in my solution ip-packets only originated by ROS are routed to the NTP-Server. This extends and maintains the traffic separation we expect from VRFs to the ROS.