Hello!

I tried to realize "ip unnumbered" with ROS 6.36 and DHCP relay enabled. All was fine while I tested with only one VLAN. When I created 3 VLANs the troubles begin.
My configuration:

[boris@MikroTik] /routing filter> /interface print detail where name=Loopback1
Flags: D - dynamic, X - disabled, R - running, S - slave
0 R name="Loopback1" type="bridge" mtu=auto actual-mtu=1500 l2mtu=65535
mac-address=00:00:00:00:00:00 fast-path=yes
last-link-down-time=sep/10/2016 10:57:01
last-link-up-time=sep/12/2016 20:44:00 link-downs=4

10 address=89.223.20.254/24 network=89.223.20.0 interface=Loopback1
actual-interface=Loopback1
15 address=89.223.20.254/32 network=89.223.20.254 interface=VL1.46
actual-interface=VL1.46

16 address=89.223.20.254/32 network=89.223.20.254 interface=VL1.64
actual-interface=VL1.64

17 address=89.223.20.254/32 network=89.223.20.254 interface=VL1.69
actual-interface=VL1.69

boris@MikroTik] /routing filter> /ip dhcp-relay print detail
Flags: X - disabled, I - invalid
0 name="DRLY-VL1.46" interface=VL1.46 dhcp-server=31.44.12.9 delay-threshold=none
local-address=0.0.0.0 add-relay-info=no

1 name="DRLY-VL1.83" interface=VL1.83 dhcp-server=31.44.12.9 delay-threshold=none
local-address=0.0.0.0 add-relay-info=no

2 name="DRLY-VL1.64" interface=VL1.64 dhcp-server=31.44.12.9 delay-threshold=none
local-address=0.0.0.0 add-relay-info=no

So, the client is trying to get address on VL1.64. I see the requests are coming on this interface, but responses are going out of VL1.46!!! So, what is wrong with my configuration or my understanding of ROS?

Regards,
Boris

You cannot assign the same IP on multiple interfaces, when you do so, you're creating a "Directly connected" route on ROS routing table, that's why ROS sends the reply out the first interface (first ocurrence of the IP).

What are you trying to achieve?

I try to repeat Cisco "ip unnumbered" feature. Where there is one supernet interface with ip address, and other interfaces without ip address. But routing is still possible,

Detailed description:

One interface (named Loopback) holds /24 supernet. Other interfaces (misc vlans) stay without ip address, with only DHCP relay enabled on it. When there is DHCP Discover request, it must be relayed to our DHCP server. There is a script on our DHCP server, when the address is assigned to client (after the filnal ACK) it connectes to Mikrotik via API and installs the route: /ip route add dst-address=client_ip/32 pref-src=Loopback_IP gateway=client_vlan

So, in this scheme I can't get dhcp relay work properly. So, I understand that same ips on interfaces is wrong, so I removed /32 ips from vlans and only supernet is active. There is no problem when I use static ip on client's PC + /ip route on Mikrotik. But DHCP does not work.

Regards,
Boris

Try this:

- Create a bridge.
- Add the VLAN interfaces as ports of it, horizon=X (same numeric value) if you want them to be isolated from each other.
- Assign the ip on top of that bridge
- Setup DHCP Relay on top of that bridge.

I still don't get why the need of setting it up the way you want, nor why are you using a public IP for that, the setup you're insisting on using doesn't make sense to me.

To advise on the best practice way of dealing with your situation, knowing the scenario with more detail, and what are the problem(s) you want to solve or features you want to implement, is the most important information to help.

Now my VLANs are children of Bonding LACP interface and I need they do. May I use this scheme with bonding or may I encapsulate bridge as child of bonding?

Why I need this scheme: we are an ISP with a small number of IPv4 addresses. For some reasons we can't use NAT for our users. For security reasons our clients must be L2 isolated. So, the only way to do this in Cisco world is ip unnumbered + vlan per user scheme. But yes, this is Cisco way. As we never used Mikrotik before we doing things in Cisco way. Of course, may be we are wrong and there is another way with Mikrotik. Please share if there is another way.

Regards,
Boris

From a management point of view, IMHO, that's a mess... too much work , too many places to manage, for little or no advantage.

Common practice (not only in Mikrotik world) is deploying and using OSPF/MPLS on POPs routers. That way you can bring isolated L2 segments tunnelled inside L3 to an access concentrator, where PPPoE is usually deployed.

This way you'll have a central point of management and could disconnect a user with a couple clics (to do changes while running DHCP you'll depend on client); if the same router running PPPoE server is the PE router you're done with routes to customers; if using an specific router for PPPoE, you can either set static routes between the PE and PPPoE router, or run OSPF between them (you'll prefer OSPF between all routers as this makes deploying MPLS a breeze)

Kevin Myers gave a presentation on US MUM regarding MPLS on WISP scenarios that may be of your interest (Presentation PDF)

@pukkita, tried your receipt. Unfortunatelly, after adding vlans to bridge (no matter what horizon value used) traffic stop in this vlans. Removing them from bridge returns normal behaviour.

1 R name="Loopback1" mtu=auto actual-mtu=1500 l2mtu=1576 arp=enabled
arp-timeout=auto mac-address=E4:8D:8C:3C:33:D8 protocol-mode=none
priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s
forward-delay=15s transmit-hold-count=6 ageing-time=5m

[boris@MikroTik] /routing filter> /interface bridge port print detail
Flags: X - disabled, I - inactive, D - dynamic
0 interface=VL1.37 bridge=Loopback1 priority=0x80 path-cost=10 edge=auto
point-to-point=auto external-fdb=auto horizon=5 auto-isolate=no

1 interface=VL1.39 bridge=Loopback1 priority=0x80 path-cost=10 edge=auto
point-to-point=auto external-fdb=auto horizon=5 auto-isolate=no

No traffic after this configuration.

Regards,
Boris

From a view of ISP - there are many disadvantages to use PPPoE or any other form of additional encapsulation. Yes, I agree, it's very simple to admin and hard to solve problems, especcialy if they are on user side and if they are on user's router. And sometimes users use their own tunnels which have troubles when encapsulated in pppoe or same. And again - PPP is not so easy as DHCP, where the user just plug and play. PPP needs some interaction with configuration on PC / router.

Regards,
Boris

Using the same horizon value will prevent traffic between VLANs, isolating them from each other. If this is not what you want, do not specify the horizon value.

From a view of ISP - there are many disadvantages to use PPPoE or any other form of additional encapsulation. Yes, I agree, it's very simple to admin and hard to solve problems, especcialy if they are on user side and if they are on user's router. And sometimes users use their own tunnels which have troubles when encapsulated in pppoe or same. And again - PPP is not so easy as DHCP, where the user just plug and play. PPP needs some interaction with configuration on PC / router.

Regards,
Boris

On WISPs the pppoe-client is usually run on the CPE (router mode) while the domestic "router" can be left in L2 mode as a simple Wired/wireless "switch". This way you keep control, and users keep simplicity.

Nothing prevents you running DHCP on the MPLS "hub" however if you so desire.

An additional MPLS advantage: performance.

And yes, @pukkita, thanks for presentation. I know what MPLS is, I know how to interact with it. But it can solve only one of my problem - client isolation. And this is a little evil in comparision of small IPv4 space which MPLS can't solve. Am I still wrong?

Regards,
Boris

To conserve address space the best tool is network design strategy, starting from not using any public addresses on the internal network.

In fact even private addresses are usually assigned by (W)ISPs on NOCs for transit, using a single /32 IP address on top of a loopback interface (empty bridge) just for the PE router: the goal is reserving all of them for the customers. And a whole (W)ISP can be designed like that: just one public IP address for the PE router, all the rest of public addresses for customers.

IPv6 is another possible option; and also not all customers need nor want public addresses.

Maybe this presentation by Brian Horn (PDF) can help you figuring out how.

We don't use public IPs for equipment. We don't use unnecessary subnetting. And to not to do unnecessary subnetting there is Cisco like "IP unnumbered". We can implement it with static and can't with DHCP. I want understand why - this is just a bug and it will be solved, or this is ROS6 limitation and it will be solved with ROS7, or it never be implemented in ROS and we should look more precisely on another technologies or even equipment (Linux + ppp-accel) for example.

Regards, Boris

As for IPv6 - Mikrotik MPLS isn't ready for it. Also no features like NAT64.

Regards, Boris

We don't use public IPs for equipment.

10 address=89.223.20.254/24 network=89.223.20.0 interface=Loopback1
actual-interface=Loopback1

Unless you purposedly changed it for the post, that's a public IP AFAIK...

No bugs that I am aware of... but noticed you set a /24 as the netmask. Try the bridge with none, or /32 netmask (ip address == network address).

That's how you can achieve an equivalent behaviour under ROS, by using empty bridges with /32 ip on top of them so that they aren't directly connected to any interface (along with proper routing), or by assigning the interfaces to a bridge and the ip on top of the bridge.

@pukkita, I'm wondering, what ips you're using on pppoe loopbacks? Private ip for loopback and public for users? Yes, this is possible, but violates internet routing policy. I mean - we not use public addresses for unnecessary reasons, for example for management reasons.

As I wrote before - assigning vlans to bridge stops any traffic there. I showed my configuration.

Misunderstood about empty bridges. Please explain.

Regards,
Boris

pukkita? Would You please explain your idea about bridges?

Regards,
Boris

so you add /24 address to the bridge, and cannot ping a client in some vlan added to that bridge?..

Hello!

I got working configuration with bridge and vlan. The only problem is that users are bound to bridge, not the unnumbered vlan, so Cisco like idea isn't work.

Regards, Boris

So, to continue...

The Cisco like unnumbered mechanics works in case:
1) empty bridge for supernet
2) unnumbered vlan
3) dhcp relay on unnumbered vlan with local-address=bridge_ip
4) ip route client's with gateway to vlan

Unfortunately this works only for one vlan, because local-address must be unique. And if not to use local-address, dhcp relay does not work for this van. Dear Mikrotik, why this restriction? Current dhcp servers use many ways to identify request, as option 82 for example. Would you please to remove it?
Forum gurus, is there a way to submit a bug/request to Mikrotik?

Regards, Boris

The only problem is that users are bound to bridge, not the unnumbered vlan, so Cisco like idea isn't work.

what's the idea? why don't you like the bridge?

4) ip route client's with gateway to vlan

if your IPs are static, you do then you don't need additional routes

3) dhcp relay on unnumbered vlan with local-address=bridge_ip

Unfortunately this works only for one vlan, because local-address must be unique. And if not to use local-address, dhcp relay does not work for this van.

can't you just use different addresses for relays? even like customer's addresses, for example. or just from some provate subnet (10.10.20.46 for VL1.46, 10.10.20.64 for VL1.64)

Forum gurus, is there a way to submit a bug/request to Mikrotik?

sure, support@mikrotik.com

Hello!

The idea is simple - to secure the network and the users and to save the ISP address space.
1) How to save - there is one big supernet for all users instead of small subnets for everyone
2) How to secure - users are L2 isolated (different vlans) and also there is no traffic to user until direct interface route is not present. Cisco installs this route automaticaly, based on snooping dhcp traffic. But there is no problem, I can do it by API.

And, answering to your questions, Chupaka:
1) IPs are not static, DHCP is used
2) I can't use subnets as (read my first post) I have a small IPv4 space, I can't use NAT.

Regards,
Boris

So my idea is simple too:

1) Create DHCP Relays with unique local-addresses from any private subnet
2) On DHCP lease, add necessary IP address to the vlan via API

You just need to check whether step 1 will work

Chupaka, thanks, your idea is good and have to work. But different relays will do our billing configuration too and unnecessary complicated

Regards,
Boris