Hello all, I have yet to install the RouterOS, but I've heard good things.

I am working at an established Wireless and Fiber-optic ISP, operated by the local consumer owned electric utility. Pretty sure we weren't set up the best and we're experiencing some growing pains on the wireless side. So, I'm looking for advice on how to better our network.

We have our own fiber-optic network backbone though our service area, and our wireless gear is almost exclusively Trango Broadband.Our core router is a Riverstone at our office here. We have a spare, but not configured for VRRP. From the Riverstone in central POP here, we run to three fiber switches in the three geographic areas we serve and branch out from there. Some of our AP sites backhaul on fiber, some over wireless. Each of the AP sites have a managed switch.

I am not by any means an expert on TCP/IP and Routing, but I understand we're running nearly 1000 wireless subscribers on a switched network and that isn't ideal. In addition to that, we are using statically assigned private NAT addresses which comes with it's own headaches. Any time there is a NAT issue with a customer, we have to statically assign a public address. We have no DHCP, no RADIUS, no PPPoE, etc, etc.

I think it's time to rework the backend of our wireless network, and I'm open to ideas. Where should I start?

This is a hard one, it really depends more on how open your company is to change.

Typically, when there is fibre backhaul we would run vlan's or VPLS from each AP or Distribution switch back to a central PPPoE concentrator. If your employers are open to Mikrotik then the RB1000 will easily perform this task. Otherwise you will be up for a Juniper MX or E series, or a Cisco 72xx/ASR. I would highly recommend whatever you do, that you build redundancy in to your solution.

You could also run a concentrator for each geographic region, this distributes network load more evenly. This is typically not done when there is a fibre backhaul however as speed/cost of transport are not such a concern.

I hope this is helpful.


Regards,




Andrew

Oh, so we wouldn't need an RB at each remote AP site if we can set up the VLANs right (had that in my head for some reason). Might involve upgrading some of our older hardened switches, but that can be done. And a RB1000U hardly costs anything.

We are a very small company, I think I could make it happen.

Suppose the real trick is the migration path for about 1000 statically assigned routers, maybe carving off one AP at a time.

To pitch this to the management, what are the most obvious pro's to moving to a PPPoE routed solution?

Suggestions on the best way to set this up? If I load the ISO image on an old workstation with a few NICs, should I be able to mock this up right?

What are people doing with NAT and/or dynamic IP assignment to make it less of a headache?

Typically, when there is fibre backhaul we would run vlan's or VPLS from each AP or Distribution switch back to a central PPPoE concentrator.

Andrew,
I'm in the same situation as RedOktober.
We now have about 500 customers... about 10 AP (Motorola) and we use Mikrotik for backhauls and core router.
Our network is bridged but every customer CPE has NAT+DHCP enabled on customer site.
That's working generally fine except for few cases in which double natting could be an issue (gaming online, P2P, VPN).
My idea would be to move toward a PPPoE solution as you suggested.
First question: Why do I need to have a different VLAN for every AP?
I already limit broadcast by using NAT. Implementing PPPoE will allow me to assign a public IP to every customer. Should I still have any broadcast issue?
Second: could be a unique PPPoE concentrator enough to serve 500-1000 customers?
Thanks
Massimo

If you have 1000 customers on one switched network, you will have problems.

With that setup, I would move wireless sites to separated routed networks one at a time.

You could have a MT router at the core, with vlans dedicated to each site. Each site's switch would untag the vlans.

I don't do DHCP for customers, as a single customer plugging in a router backwards could screw up a site. Furthermore we lock nonpaying customers via a script which sends a firewall command (for blocking IPs) to the appropriate MT router (The customer's gateway address).

I have done a little PPPOE, but don't care for it. Some customer routers don't like to do PPPOE with the MTs for some reason, and some internet sites have path-mtu issues with PPPOE.

Just to be clear, get away from DHCP!!!! PPPoE is definately the way to go.

What jp has said is what I was referring to. Having 1000 clients in a single broadcast domain, even with PPPoE is asking for trouble. You are best to either run concentrators at the edge of the fibre network, with the fibre acting as a routed backbone. Or bridge the AP's on to the fibre and terminate these into individual vlan's leading in to your PPPoE concentrator.

It is much easier to run a central concentrator from a QOS perspective, as you will be able to share bandwidth more effectively as the concentrator will know how much upstream it has in total and be able to divide this more evenly between clients.


I hope this provides a little more insight for you.



Regards,





Andrew

If you have 1000 customers on one switched network, you will have problems

Could you be more clear?
What if all the customers are natted? (so they cannot generate any broadcast traffic)

I don't do DHCP for customers, as a single customer plugging in a router backwards could screw up a site

Again, my diagram is the following:
INTERNET --- CORE ROUTER (NAT) --- (BRIDGED WIRELESS NETWORK) --- CPE (DHCP+NAT) --- CUSTOMER LAN

How can a single customer affect my broadcast domain if they are natted?

Thanks
Massimo

You are correct, they cannot generate broadcast traffic when behind the CPE, but if they plugged a machine in front of it they can easily cause you significant problems. Imagine if they plug your access port in to a switch with no spanning tree and create a broadcast storm, or if they maliciously inject broadcast traffic on to your network.

If you do want to keep away from PPPoE to reduce complexity, and continue using DHCP I would take JP's suggestions and at least route the network. You could easily perform this with Layer3 switches, or with Mikrotik RouterBoards. This will stop any chance of a broadcast storm, and if you take the chance to implement MPLS at the same time you can offer your customers a "Virtual Wire" type service using VPLS circuits to join their sites accross your network.

You are correct, they cannot generate broadcast traffic when behind the CPE, but if they plugged a machine in front of it they can easily cause you significant problems. Imagine if they plug your access port in to a switch with no spanning tree and create a broadcast storm, or if they maliciously inject broadcast traffic on to your network.

Thanks for your reply nz_monkey,

1) they actually cannot plug a machine in front of the CPE.
I forgot to explain that I supply preconfigured CPE with WPA2 encryption. The user cannot change the CPE configuration.
The CPE has DHCP+NAT enabled on ethernet side, so I can't see any way to inject any broadcast in my wireless network, do you agree?

2) I agree about "routing is better" but: how can I solve the "public ip" issue?
I mean: Let say I have a /24 public ip subnet. I have to share it among my users on different towers (AP).
Just forget about how to assign them (statically or automatically by DHCP or whatever).
How can I subnet my network without wasting my public IP range?

You are correct, they cannot generate broadcast traffic when behind the CPE, but if they plugged a machine in front of it they can easily cause you significant problems. Imagine if they plug your access port in to a switch with no spanning tree and create a broadcast storm, or if they maliciously inject broadcast traffic on to your network.

Thanks for your reply nz_monkey,

1) they actually cannot plug a machine in front of the CPE.
I forgot to explain that I supply preconfigured CPE with WPA2 encryption. The user cannot change the CPE configuration.
The CPE has DHCP+NAT enabled on ethernet side, so I can't see any way to inject any broadcast in my wireless network, do you agree?

2) I agree about "routing is better" but: how can I solve the "public ip" issue?
I mean: Let say I have a /24 public ip subnet. I have to share it among my users on different towers (AP).
Just forget about how to assign them (statically or automatically by DHCP or whatever).
How can I subnet my network without wasting my public IP range?

1. Unless you have hotglued the ethernet cables in place, they could perhaps while reorganizing their desk, plug the wan and lan in reversed and become a DHCP server on your network, giving out 192.168.x.x instead of your DHCP servers necessary IPs. They might also not like the configuration and put their own CPE in, perhaps so they can do port forwarding or use special applications. We had trouble back in 2004 where linksys's were repsonding improperly to broadcast traffic and causing huge packet spikes when they all responded in unison. Linksys was able to patch it at our urging. Then, gigafast routers started to do the same thing a year later and they were not able to fix it. 90% of consumer routers are completely full of junky firmware that's assembled by people who know enough to be dangerous; it's good enough to minimize RMAs, but no better than necessary.

2. You'll have to learn subnetting, then how to do static routes. A /24 can be subdivided into 8 /27's. Enough for a core and 7 sites. If you properly divide your IPs for your site's needs, and use them all up, you're not wasting them. In fact, it might be good to use them all up and get more while you can. It's going to be getting harder and harder in the near future to get IPs. If you keep it as a /24 and aren't efficiently using them , you'll get denied when you need more.

We do have DHCP running on our network in most places to give out private range non natted IPs, but customers don't depend on it. We do it so we can plug a laptop into the radio and have internal network access for limited testing/troubleshooting. If a customer calls and says they can get to our webpage and nowhere else, we know their linksys has lost it's settings and defaulted to dhcp.

Agree with jp 100%

You can also route your /27 public ranges over RFC1918 ranges, this is quite a common practise to reduce wasting IP's.

1. Unless you have hotglued the ethernet cables in place, they could perhaps while reorganizing their desk, plug the wan and lan in reversed and become a DHCP server on your network, giving out 192.168.x.x instead of your DHCP servers necessary IPs.

Sorry for not being so clear
We (as WISP) supply preconfigured CPE to our customers. CPE remain our property. They are password locked.
Customer cannot change the configuration in any way.
CPE only have ONE ethernet port and integrated antenna.
Customer CPE have NAT + DHCP enabled on ethernet side (internally - NOT VIA EXTERNAL ROUTERS).
So, how could a customer switch LAN/WAN? (just one ethernet coming out from the CPE) How could a customer break the NAT?
Reverse engineering the firmware? Breaking WPA2?

They might also not like the configuration and put their own CPE in

Not possible... (unless they know the WPA2 key and others parameters)

2. You'll have to learn subnetting, then how to do static routes. A /24 can be subdivided into 8 /27's. Enough for a core and 7 sites.

Ok, I know how to subnet a /24 into 8 /27 but I also know that
a /24 give me 253 available IP for customers,
while 8 /27 give me 232 available IP,

But, since the ideal case should be one /30 class for every customer (point-to-point), a /24 results in serving only 64 customers.
Which is the suggested strategy? One class for AP?

Thanks anyway for your time.
Massimo

Hopefully you are starting to answer your own question, the arguments you supply for IP allocation are the same arguments that are in favor of implementing PPPoE. With PPPoE you are able to allocate /32 addresses to clients with an RFC1918 gateway address meaning you can effectively use every single address in a /24 range.

I'm still trying to wrap my head around all this and figure out where to start.

What prerequisites do I need to address?

First I'm trying to set up FreeRADIUS with MySQL on a linux server (right now on Ubuntu LTS in VMWare), though I will also have a fair physical Windows 2k3 Server coming available soon (several years old, dual 2.66GHz Xeon, 4GB RAM, U320 10k SCSI). Would the Mikrotik User Manager eliminate the need for a RADIUS server? We should have one for network security anyway shouldn't we?

I have a RouterOS evaluation installed on an old (real old) P4 with a couple NICs. We will need a Level 6 controller license eventually, but I'm still trying to learn this before I commit to buying an RB1000U. I'm thinking of picking up an RB450[G] ($99/$129) device with a Level 4 to get started with vs. getting an x86 license for this old clunker PC.

Anything else I'd need other than a RADIUS [and/or User Manager] Server and a Mikrotik router?

I was thinking I could parallel the Mikrotik PPPoE solution onto the VLANs of our current NAT as we transition the individual subscribers, and afterwards break off the APs to their own VLANs. Make sense?

Keep in mind that the 1000 is not the only options. We make the PowerRouter 732 and 2200 series units as well. These we can add Fiber interfaces in, as well as we offer fiber conversion products as well. We do quite a bit with the fiber on the 2282 plus it gets you 10 GigE Copper interfaces as well. Contact me off list at support@linktechs.net if you need more information.

I haven't worked on this in a few weeks before this weekend, forget where exactly I left off. At this point I have an RB1000U talking OSPF with our core router. I have a class C of public addresses allocated to us from our upstream provider that I'm going to dedicate to the Mikrotik solution right away. I don't think I have OSPF completely configured, perhaps the core router isn't advertising it's default route or somesuch, as I am not able to get out with the public addresses (but if I create that ip interface on the core router, those addresses work fine).

What I'm planning on doing, is creating a NAT pool with the mikrotik and using those as the default pool for PPPoE, with the ability to manually assign a public address when needed.

We're about to roll out some new wireless gear (Wavion beam-forming WiFi), and I intend to start using this PPPoE backend with that. The Wavion gear is on it's way, so finishing the OSPF/NAT/PPPoE configuration is a priority now.

[...]

We're about to roll out some new wireless gear (Wavion beam-forming WiFi), and I intend to start using this PPPoE backend with that. The Wavion gear is on it's way, so finishing the OSPF/NAT/PPPoE configuration is a priority now.

I have not heard of wavion before so I googled. It seems quite pricey but they promise good NLOS performance in addition to LOS. Can you share your experience?

It's really not all that pricey considering we've been using all Trango gear up to this point. Ubiquiti CPEs are recommended for best performance with the Wavion, and those are cheeeeap, like $50-$80. Right now we're paying $288-$460 for our 5.8 and 900 Trango gear that don't have half the throughput.

Supposedly the beam-forming technology is a real game-changer for outdoor wifi. We avoided wifi when we started six years ago after some bad experiences, hopefully this stuff works as promised...

Five-plus year later update. Almost everything is awesome. Thanks to all who contributed suggestions, our current design uses a lot of what was discussed here.

Our WISP network is fully routed wish OSPF. We have RouterBoards at all our wireless sites, still use some RB450Gs at some smaller sites, larger sites are running CCRs. Almost all our sites are fiber connected. Our wireless network is almost entirely Ubiquiti AirMAX now, starting to test out their AC gear.

We had PowerRouters from linktechs.net loaded with SFP cards in our three regional POPs for a while, but they were GARBAGE. Horribly unreliable, dead ports, etc. Thankfully the CCRs came out with high enough fiber port density and saved us from those piles. Would still love to see a CCR with 4 SFP+ and 24 SFP. The CCRs have not missed a beat.

We run PPPoE at the edge with a Routerboard/CCR at the tower site, with OSPF tying all our routers together over fiber and redundant wireless links. For the conversion, we ran a central PPPoE concentrator on the switched network until we could convert everyone on a tower site from static assignment to PPPoE, then we broke that site off on it's own router/PPPoE server.

Our call volume is less now with 1700 customers than it was when I started here with 300 customers. Lots of growing pains, many iterations of router configurations, complete wireless gear replacement to get away from legacy gear, but nowadays things are pretty smooth. Always chasing congestion at the last-mile because everyone wants to stream HD everywhere, but that's what it is to be an ISP...

Happy you got things sorted out. This might come a bit late but I saw things in this thread that gave me serious shivers.

I have a bit of a service provider background and while I've moved on to a less stressful job I've still been helping a rural FTTH-network. Hard to keep away from the fun

Anyway, the security in Internet Protocol is not built into subnets, but between them. All kinds of nasty can happen inside subnets, like ARP-spoofing, rogue DHCP and other similar things. There are ways of keeping customers that share a broadcast domain separated but I'm strongly of the opinion that it's a bad and expensive idea. Business customers are usually given their own public subnet and they receive the added security, why should consumers be treated as second class citizens? Also, maintaining "large" layer 2 networks is painful.

This particular FTTH-provider was given some but not enough IPv4-addresses, which lead to a Carrier Grade Nat setup with 1:1 NAT at the core for customers who wish to have their own public IP. Needless to say, the whole network is numbered with RFC1918 -addresses. What really is special about this network is that everything is fully routed all the way, albeit statically. Every customer is placed on it's own L2 segment and L3 -subnet and this subnet is routed at distribution or in some cases even at the access level if the equipment is capable. The customer subnets have a DHCP service that enables automatic configuration of the customer owned (not enforced by law but the communications ministry in this country makes it quite clear) CPE. True plug and play, IP over Ethernet.

The beautiful part is that whatever kind of storm the customer creates, it will in a worst case scenario make it's way through an access device to a distribution device and in best case it will only affect the access-port and CPU of the access device. But, customers will never be able to spoof any traffic because each customer is isolated from the others by one or more L3-routers. Some L2-filtering is required to protect the Provider Edge from overload but at minimum possible level. Also, the customers are allowed to communicate over the shortest route with each other (also according to recommendation from the communications ministry). As the customer subnets are very static and will only be present on one access port, It's very easy to pinpoint the source of possible attacks. The provider network works just like the Internet, but at a much smaller scale.

IPv6 on the other hand is right around the corner. I've recommended that the FTTH-network sweet talk their upstream providers into sponsoring a Provider Independent block of addresses and then a suitable amount of CCR1016:s at distribution layer. Once again, statically assigned Prefix Delegation and separate L2 and L3 segments for each customer.

I understand that WISP:s have the burden of shared broadcast domains on the last mile but I still cannot see a need for PPPoE. This, I believe, is where CAPsMAN comes in and saves the day. To my understanding it should be possible to tunnel each WiFi-client to a controller and treat the traffic as needed at the controller. This should enable separating customers sufficiently.

Time to end my ramblings: Remember kids, shared broadcast domains are bad, mmmkay?