MikroTik

Getting lots of requests for this from customers.

Really? Isn't it just a fancy sounding putting of cloud word everywhere? Nothing against, just asking what is it above a vpn?

A VPN only functions over a single connection. SD WAN is more robust. You can utilize multiple connections for instance and create a secure tunnel that sends packets over whichever has the least congestion at the moment. So for instance, you're a bank and your traffic to the banking server over a VPN at the main site becomes congested so transactions slow down. With SD WAN you can bring in a second carrier (since the first would just still be congested) and maintain that secure tunnel with packets going over the fastest route. I suppose it would be more like secure OSPF.

SD-WAN can balance, prioritice (duplicate packets for guaranteed delivery with lowest possible latency) on multiple encrypted paths and such on applications/steams level.

Yes I would love MT to do this but it is currently not possible with the design in current hardware or CPU power and software.
Build the next generation of boxes with lots of cpu power and extra 100cores tilera tiles with a coupe of monster fpga's and the lock yourself up for two years and program..... Yes then we have it..... or not....

If you just want sd-wan on ip level, perhaps with some queing mt has your back covered with all the tunnel interfaces and ml-ppp and queues.
What you do where and why is another talk of the rest of your life.

In my ears SD-wan is another shit bussword. There are fancy Products with real "application", "flow" and "network intent" based policys that can do fancy stuff.
What they Don't do is teach you where it is a good thing to do and where is not.
A customer just asking to do sd-wan is in no shape of knowing what problem they trying to solve. The best of breed SD-WAN product cant do jack shit with crap lines.

SD-WAN is not a Silver bullet to solve a problem you don't know you have.


And then Again:

I'm All for it! It has its use cases but it is not a New technique it is normal ip router with a combination of many techniques.

Really? Isn't it just a fancy sounding putting of cloud word everywhere? Nothing against, just asking what is it above a vpn?

Yeah I don't get this either.

SDR I get - software defined radio. And it makes sense too. Very powerful and useful.

Sofware Defined WAN ? Well of course, it's always software defined already :-/ Definitely a buzzword that just confuses IMO. but maybe I'm missing something?

A true WAN isn't really "software designed". When you have multiple buildings, you could use ethernet if all the buildings are within about 300 ft. Then you have a WAN. Internet need not be part of it at all. If you go wider you could use dedicated leased fiber or dark fiber. It's physical connectivity between locations. You can do this with P2P wireless as well. Again, it's a complete physical WAN. Not "software desiged" although software does have a role.

VPN in a construct where an internet connection can be setup at multiple locations and then the devices at the endpoints use software withing the endpoint hardware to create secure encrypted links between the two over public internet to protect the data. This way for example, credit card transactions and such can't be sniffed between a bank branch in a remote location without the ability to get a dedicated WAN connection to the main server. It's encrypted and only the equipment on the other end can decrypt it.

But because of the way that it's designed, it can only traverse a single internet connection from static IP to static IP. It's still subject to the ups and downs of traffic on its public internet connection. And often when traffic gets heavy, the VPN may time-out and it takes a few minutes for that VPN connection to reset itself and during this time, a credit card customer may be waiting in line holding up transactions for the people behind them.
SDWAN simply takes it a step further. It basically allows what amounts to a single VPN to traverse multiple networks based on whichever has the lowest latency, jitter, and fastest connection at the time. It can take a single transaction and send packets over multiple connections simultaneously or over whichever connection is fastest between the two endpoints. So you could have two or three different carriers with internet connections coming in connected to a single device and greatly improve the reliability of the traffic between the locations while still maintaining the security of a true WAN or VPN as one carrier's traffic may be faster than another's at any given moment. It helps elimiate the possibility of dropped VPNs. Basically it's automatic load balancing with the security of a VPN or WAN. Secure OSPF. Call it what you want, but the bottom line is that when it comes to multi-location companies that must maintain secure connections that can't afford a true dedicated WAN to the rest of their network, or have sites that are off-net from their WAN provider, this is a solution that can increase reliability and throughput for them while maintaining the secure connection that they require to remain in compliance with whichever industry compliance standards they need to meet such as HiPAA, Sarbanes-Oxley, CPI, etc.

As for processing power, I don't know why it would take that much more to build it into the CCRs. They already do OSPF, BGP, EOIP, PPtP, etc. They can load balance and they can secure connections. A load-balanced secure multiple internet connection bonding shouldn't require much more processing power as they handle these other things quite well.

You can read more about it here. https://en.wikipedia.org/wiki/SD-WAN

Much of the typical SDWAN features can be implemented if you like to geek out to a high degree. Actually I think it would take considerable development from Mikrotik to have a simplified approach like Pepwave.
At one medical facility I have 3 different WAN connections and three tunnels back to my POP. If is weighted to prefer the Fiber and second is Comcast Coax and third is AT&T LTE. You can unplug the Fiber while on an active VoIP call and the call doesn't even drop.

I do think that SDWAN will be a requirement by most business customers soon. And whoever has the most reliable system for the lowest price will get the market share.

Right. SD WAN is going to be asked for and needed more and more. That was my point.

Like you said, you already have three tunnels. But from what I could get from your explanation, they are tiered as basically failover. If one drops, the next picks up.
But imagine having your three automatically routing packets over whichever is fastest at that particular moment and only having one VPN basically to deal with instead of 3.
That's the difference.

Right now if connection A gets sloppy, intermittently dropping packets on your VPNs, with throughput falling down to minimal levels but not long enough to cause it to switch to connection B because a bad spot on the fiber because a car ran into a pole, it's still going to use the fiber because it hasn't dropped connectivity. But VoIP calls will go to crap. Meanwhile you have two other connections there that are totally being wasted while you could have the combined throughput of all three.

It seems to me with all the other features, the architechture is already there. So it should be a matter of a firmware update. It shouldn't take a whole lot of development. If it did, these other companies wouldn't already be rolling it out.

I think what's happening instead is that the devs for Mikrotik aren't really aware of how fast SD WAN is taking off and many of the users of the equipment aren't aware of the need or desire for it so they aren't asking. And with that disconnect, the devs aren't even working on it which is why I decided to bring it up here. Hopefully by this conversation it will trigger someone there to start looking into it.

As it is, I have several banks wanting the functionality and I'm stuck, unable to do it. Of course there are plenty of 3rd party options.

Here's an "SD WAN for dummies" free ebook I downloaded. http://bit.ly/2DxmbKB

And some other articles

https://www.sdxcentral.com/sd-wan/defin ... hitecture/
https://www.sdxcentral.com/sdn/network- ... l-network/
https://www.sdxcentral.com/reports/2017 ... y-results/
http://www.velocloud.com/sd-wan/sd-wan-traditional-wan
https://www.networkworld.com/article/30 ... e-day.html
https://info.talari.com/SEM-Gartner-Com ... 2XEALw_wcB
https://www.riverbed.com/forms/gartner- ... vDEALw_wcB
https://www.networkcomputing.com/cloud- ... /212008131
http://viewer.zmags.com/publication/2ff5f0ed

I agree with you. However I doubt you will see it anytime soon. I use OSPF with BFD on each tunnel and it switches very fast. But your right that its not balancing loads etc.
That particular client uses a static IP from my POP that floats between the tunnels. They have an onsite PBX and it allows them to float between providers without having to change the src nat and of course the incoming VoIP calls routed to their IP doesn't have to change with every switch.

https://www.cisco.com/en/US/technologie ... 44005.html

I agree with you. However I doubt you will see it anytime soon. I use OSPF with BFD on each tunnel and it switches very fast. But your right that its not balancing loads etc.
That particular client uses a static IP from my POP that floats between the tunnels. They have an onsite PBX and it allows them to float between providers without having to change the src nat and of course the incoming VoIP calls routed to their IP doesn't have to change with every switch.

https://www.cisco.com/en/US/technologie ... 44005.html

So you're using Cisco equipment? Or Mikrotik for this? Can you tell me more about how you set this up? Assuming it doesn't take up too much of your time.
Thanks!

I'm using Mikrotik. The Cisco doc just explains it really well.

In my situation I configured the HA or SDWAN mikrotik router and then connected their existing Mikrotik router to it. I just assumed the gateway position and was in between the ISP connections and their router.

HA - SDWAN Router
Ether1 ISP1
Ether2 ISP2
Ether3 ISP3

EoIP tunnel 1 from POP to Customer HA Router over ISP 1
EOIP tunnel 2 from POP to Customer HA Router over ISP 2
EOIP tunnel 3 from POP to Customer HA Router over ISP 3

OSPF w/ BFD enabled between each end of EoIP tunnel 1 (POP=10.20.0.1/30 Customer=10.20.0.2/30)
OSPF w/ BFD enabled between each end of EoIP tunnel 2 (POP=10.20.0.5/30 Customer=10.20.0.6/30)
OSPF w/ BFD enabled between each end of EoIP tunnel 3 (POP=10.20.0.9/30 Customer=10.20.0.10/30)

I routed the whole /29 to the HA router from my POP. So the first IP address of the /29 was placed on Ether5 and it acts as the gateway for the customer router.
The default route of the HA SDWAN router is the POP. (OSPF default route it receives from POP)

0.0.0.0/0 over 10.20.0.1 if OSPF & BFD select primary ISP
0.0.0.0/0 over 10.20.0.5 if OSPF & BFD select secondary ISP
0.0.0.0/0 over 10.20.0.9 if OSPF & BFD select tertiary ISP

Since the default route is through the tunnel mangle rules and routing marks are required to force the traffic for the EoIP tunnels out the appropriate ISP connection.

When everything is dialed in you can be on an active VoIP call and unplug the primary Internet connection and you only have maybe 1 or 2 seconds of audio loss but the call stays connected and audio resumes just fine. Several times the parties on the phone never noticed the switch over.

Anyway a very rough outline of the framework. Took an insane number of failed attempts before I landed on this one and it has been working great.

Once in a great while the mangle rules would not be followed and I would have to manually clear connections to get them to do what they should. I have a script to do that occasionally now and that issue never came up again.

Its overall not horribly complex I have some configs that are 1700 lines so this one seemed middle of the road for complexity. The OSPF and BFD take the most time to get to their happy place.

awesome. thanks for the tip. I'm looking at some software defined options for this right now. Seems like my only option to do this.

A VPN only functions over a single connection.

this is not true. any overlay vpn can make use of _every_ available uplink path if required. but believe me, no one on earth want to do per packet load balancing over multiple independent connections, as OOO packet delivery is a pain in the back. Mikrotik has only per packet load balancing and a quite obscure way to actually utilise this (multiple gw values in a single route entry).

You can utilize multiple connections for instance and create a secure tunnel that sends packets over whichever has the least congestion at the moment. So for instance, you're a bank and your traffic to the banking server over a VPN at the main site becomes congested so transactions slow down. With SD WAN you can bring in a second carrier (since the first would just still be congested) and maintain that secure tunnel with packets going over the fastest route. I suppose it would be more like secure OSPF.

i love how cisco (or replace this with any "big" brand name) marketing folks glue buzzwords to "destination address based packet forwarding tweaked as we need it". the whole SD-WAN movement is against the "classic" virtual pipe providers, so to replace L2 and L3 MPLS VPNs, and give a slight possibility to the customer to steer how stuff is being delivered over the remote connections. there ain't nothing new in it - and to be honest the whole L3 VPN story was found out how to fight "physical" pipes to make the service more affordable and more versatile. now you as customer can just buy any random internet connection and run your OTT VPNs on top of it. like we did back in 1999 with IPSec.
of course you have a lot more tweaks to influence how your packets/flows will be forwarded - but those are more or less convenient frontends to policy routing.
but if you can milk the whole world again - yes the ones you already sold MPLS VPNs 10-15 years ago, and IPSec prior to this, and leased lines even earlier.

and since you aren't in control of the infrastructure, you do some sort of poor man's traffic engineering - oh wait, your can just use RSVP to do the heavy weight lifting.
but inherently it's just connecting the customers sites w/o any dedicated infra. you do your best at the CPE (say, prioritise certain flows over others, do traffic shaping/bw guarantees for some other flows) and then let the good old internet transfer your encapsulated security payload to the far end. whatever happens in-between, you have no influence on that.

so customers can reduce their telecommunication out-payments to carriers and buy bigger capacity internet pipes (even some asymmetric residential service lines like VDSL or GPON) instead. as for redundancy, it seems the Internet itself is far more robust/redundant than any telco carrier on the face of earth. yet you don't need to negotiate nothing with your carrier.

glue on a point and click interface on top of it - the almighty SDN controller - which hides the complexity and there you have it. and when it comes down to troubleshooting, you just "re-provision" and hope this fixes things.

regarding the packet multiplication phenomena:
it is possible to duplicate the same packet and send it over multiple paths (tunnels) then deliver the first one arriving and drop all the others. however the performance part of this is the big burden. routers are fast, cause they push packets as simple as possible. call it CEF or FastTrack, it means roughly the same: pass the packet along the cached path AFAP. if you want to maintain states of flows, it's also possible - this is something that firewalls do right now. but maintaining states for individual packets, which is knowing that a certain packet has been received or not, is torture to them. usually routers don't care whether a packet is an OoP, they just forward it. To maintain packet states you'd need OoP detection and maybe even some sort of packet reordering buffer for each freaking flow, so you can be sure a packet sent over 3 different tunnels will arrive just once.
if you leave dealing with it to the TCP stack of the end, you end up having terrible TCP performance.
there are tools out there, like multi-path TCP or UDP based reliable transmission methods (as Google's QUIC) that can do stuff on app level, that are next to impossible at network level.

in my opinion the "SD WAN" story is a brief stop in the journey towards a more application heavy secure networking - and sooner or later most enterprises will realise that trusting application level encryption with personal internet banking can secure enough for their everyday business, and forget about the "secure" pipes. trends are anyway that you want to avoid cleartext (read: unsecured) communications anyway, regardless if they transmitted over a secure medium or not.

the wikipedia article cites stuff as VoIP or videoconferencing to be painful over traditional WAN. welcome to the future, where you can get all this stuff as a service, and it's about trust anyway. whatever can go on now, you can even get flat tariff for all your in-enterprise communications even from the big ol' PSTN operator which makes enterprise VoIP more costly - or just use a paid OTT communication provider to make sure your stuff remains private. we use webex for company and inter-company meetings w/o any dedicated infra... since _years_. i doubt SD WAN will change this. as it seems now it targets to fix some issues that already became non-existing.

and the SOHO/SMB segment already uses "services" that run in "the cloud" - and they don't need no call-home, nor interconnecting distant offices - their stuff is already is on the 'net for some time so they can access it from anywhere.

Wow...there's a lot of FUD in here. SD-WAN can be many different things, but at its core, SD-WAN is a way to abstract the physical transport away from the service layer of the network. Classically, when building a network you must answer the question: "What kind of topology do I want to use to build this network?" and then you are forced to live with the consequences/limitations of the network topology chosen. By breaking the physical transport layer of the network away from the logical layout of the network, you can have the best of both worlds.

My company is currently using Viptela's SD-WAN tech and actually, it is really, really good! Most of my sites have either MPLS + Internet -or- dual Internet circuits via disparate providers. The Viptela router hides the underlying topology and presents my sites with a single overlay network. Every few seconds the routers will make a decision about the quality of one circuit vs the other and then pick the circuit which is in the best shape at the time, based on the application. My VoIP traffic gets routed over whichever circuit at the time hast the lowest end-to-end packet reorder/jitter/loss. Also, if I have a temporary need to increase bandwidth at a specific site, I can throw another WAN circuit at the site and the viptela router will start load balancing traffic across that circuit. Viptela's hardware is based on a quad-core Octceon board, which does not have much more CPU power than many of the Mikrotik devices.

With respect to SD-WAN's place in history, all I can say is that while the name "Software Defined" might be a bit misleading, but the truth is that the ideas within current SD-WAN deployments fix many of the problems facing traditional WAN deployments....and that is not a passing fad. Provisioning time is faster, transport routing changes are less impactful to branch offices, and site outages due to WAN circuit failure are almost non-existent. There are many scenarios where a SD-WAN deployment makes no sense and is massive overkill, but for a medium to large business setting where uptime and scalability are important, SD-WAN serves a very useful purpose.

i think what keeps being overlooked here is that unlike MPLS or an ELAN solution, SD-WAN has packets traversing two separate independent networks. At least it should be. It kind of defeats the purpose to carry it over an internet T1 and DSL if they both go back to the same switch and network.

Assuming you configure it right such as with a wireless P2P to my tower which is riding a dark fiber network to a Zayo internet egress that goes up to Indianapolis (I'm in Evansville IN) and a Coax that rides Spectrum to Nashville, then you have a single VLAN that's going to route very well and provide solid call quality on a VoIP system while having redundancy inherently built in. Someone running a car into a telephone pole isn't going to take them down as can happen with a single MPLS or ELAN connection. Meanwhile when both connections are working, they're getting the throughput of both combined.

Of course, if the equipment fails.........I guess we need a second SD WAN device running in tandem, both on UPS units to really do this right.

Like anything else, it has its place but it has to be planned properly if you truly want to benefit from it. What I see is a lot of people throwing it out there without understanding it which makes it seem like "the latest fashion" but there are benefits if it's planned properly.

The great concept of SD-WAN is that you let the software decide and build tunnels/routes. You just define a routing domain or logical topology and deicide what members that will be included, then the nodes establishes the routes/tunneling, whether it's Ethernet, leased MPLS or 3G/LTE.

Mikrotik can config to run multiwan and zero packet loss when switch line.
Hire consultants they will do it for you

I'm using Mikrotik. The Cisco doc just explains it really well.

In my situation I configured the HA or SDWAN mikrotik router and then connected their existing Mikrotik router to it. I just assumed the gateway position and was in between the ISP connections and their router.

HA - SDWAN Router
Ether1 ISP1
Ether2 ISP2
Ether3 ISP3

EoIP tunnel 1 from POP to Customer HA Router over ISP 1
EOIP tunnel 2 from POP to Customer HA Router over ISP 2
EOIP tunnel 3 from POP to Customer HA Router over ISP 3

OSPF w/ BFD enabled between each end of EoIP tunnel 1 (POP=10.20.0.1/30 Customer=10.20.0.2/30)
OSPF w/ BFD enabled between each end of EoIP tunnel 2 (POP=10.20.0.5/30 Customer=10.20.0.6/30)
OSPF w/ BFD enabled between each end of EoIP tunnel 3 (POP=10.20.0.9/30 Customer=10.20.0.10/30)

I routed the whole /29 to the HA router from my POP. So the first IP address of the /29 was placed on Ether5 and it acts as the gateway for the customer router.
The default route of the HA SDWAN router is the POP. (OSPF default route it receives from POP)

0.0.0.0/0 over 10.20.0.1 if OSPF & BFD select primary ISP
0.0.0.0/0 over 10.20.0.5 if OSPF & BFD select secondary ISP
0.0.0.0/0 over 10.20.0.9 if OSPF & BFD select tertiary ISP

Since the default route is through the tunnel mangle rules and routing marks are required to force the traffic for the EoIP tunnels out the appropriate ISP connection.

When everything is dialed in you can be on an active VoIP call and unplug the primary Internet connection and you only have maybe 1 or 2 seconds of audio loss but the call stays connected and audio resumes just fine. Several times the parties on the phone never noticed the switch over.

Anyway a very rough outline of the framework. Took an insane number of failed attempts before I landed on this one and it has been working great.

Once in a great while the mangle rules would not be followed and I would have to manually clear connections to get them to do what they should. I have a script to do that occasionally now and that issue never came up again.

Its overall not horribly complex I have some configs that are 1700 lines so this one seemed middle of the road for complexity. The OSPF and BFD take the most time to get to their happy place.

I always wonder what is the underlying technology that build those peplink Speedfusion, this should give me some light.....

SD-WAN Standard by MEF https://www.mef.net/resources/technical ... leid=file1

SD-WAN boils down to managing multiple WANs with respect to application types. There is of course various levels of sophistication from, say a Viptela can doing a minute granularity switching, to all the way of a packet-level switching granularity of Mushroom Networks' Broadband Bonding. The sophistication of the overlay tunnels is important but it is also important how easy and automated the setup and management of the whole system is. So, SD-WAN really has two main aspects, namely, the "multi-WAN overlay tunnels" and the "centralized dashboard for managing SD-WAN endpoints".

Mikrotik could easily add some easy SD-WAN type functionality if they added Zerotier VPN. There is already a feature request open: viewtopic.php?t=98659

Check out more at zerotier.com. It's a cool solution. It has the ability to run independently on client devices, but you can also use it on some routers/firewalls already (and Linux boxes such as Raspberry Pi) to to layer-2 bridging or routing over it. You could have a Zerotier backbone and run whatever routing protocol you'd like, or extend your local LAN to remote machines via the layer-2 bridging with working broadcasts, etc... If Mikrotik devs aren't willing to redo everything to incorporate a Mikrotik SDWAN solution, then a Zerotier VPN option would be great for those who don't already have a Peplink or other SDWAN appliance option. And the Zerotier service is free for up to 100 clients over unlimited networks (and there are upgraded plans available, or you can host your own service). It could really be a no-brainer for small shops and big shops alike. Get one (or several) cheap Internet connections for redundancy (no need for leased lines or MPLS links) for WAN redundancy and connect all the offices with Zerotier. Could help Mikrotik to not lose so much market share to SDWAN router providers.

You already have MPLS button. Why dont add SD-WAN one?