I finally have some more time fiddling around with RouterOS on a CCR1009.

Goal is to get multi-WAN to work. From what I read is that PCC is the way to go but the only example in the manual is a bit thin for me: http://wiki.mikrotik.com/wiki/Manual:PCC as the explanation is very dense, likely expecting much more knowledge than I have now.

Regard me as a relative RouterOS noob.

I have multiple upstreams with different configurations and megabit speeds:

- ISP1 provides fiber with 50 down 50 up hands 1 static DHCP non-public IP address (as for now there is a NAT router in-between that does PPPoE, VoIP, ISDN/VoIP-bonding, etc)
- ISP2 provides fiber with 50 down 50 up hands out 6 fixed public IP addresses (same /29 subnet; ascending order) with a fixed gateway (they do PPPoE, I might later get rid of this one)
- ISP3 provides ADSL 10 down 2 up and hands 1 static DHCP non-public IP address (there is a NAT router in-between that does PPoE and NAT port forwarding). I will get rid of ISP3 within a year as speed is too slow compared to cost.
- ISP4 provides Cable 100 down 20 up hands out a non-static DHCP public IP address. I will get rid if ISP4 within about 6 months as speeds and latency are varying too much and not knowing when the dynamic IP changes is bad (it hardly does but when it does: havoc)
- LAN with non-public IP /24 subnet behind NAT with many port-forwards from the various ISP* provided addresses/ports to a bunch of locally running VMs/devices (think http(s), ssh, DNS, SMTP(s), etc)
- Wanting L2TP/IPSEC on all public IP-addresses

ISP1 and ISP2 have very low latency. ISP3 has big latency. ISP4 varies a lot.

I want to start with the above configuration step by step:

1. Start with ISP2: it is already connected so I can test outgoing traffic from the router: that works; incoming doesn't work well
2. Get port-forwarding to work
3. Add ISP1
4. Add ISP3 and ISP4
5. Migrate ISP1 PPPoE to the CCR1009
6. Migrate ISP2 PPPoE to the CCR1009 as well
7. Slowly ditch ISP4 and ISP3 (there is some stuff on the outside with hardcoded IP addresses that disallows me doing this sooner).
8. Maybe add UPnP to the mix

Currently I have a TP-Link ER-5120 router that handles everything, but it's a piece of crap for many reasons (DNS and DHCP are unreliable; it cannot handle incoming L2TP/IPSEC, PPTP or OpenVPN connections, more at https://wiert.me/2016/03/18/tp-link-er5120-limitations/) of which the latest is that many secure connections are unreliable (likely because they do multi-WAN all wrong) which for instance makes electronic banking and using the Google suite of services a pain.

A few questions up-front:

- for ISP1 - how to configure this on the CCR1009 so I won't harm the upcoming multi-WAN steps
- for ISP2 - how to configure multiple public-IP addresses still going out the same gateway upon incoming traffic.
- for all - how to use the above mix of external addresses with PCC multi-wan setup

I've spread out the above over multiple phases for two reasons:

- I want to learn not only how a certain phase should be solved but also why
- They seem like the right chronological order to use (but if the order is wrong: feel free to explain why)

Any help is welcome, especially explanations with references on *why* a certain solution works.

Thanks for any help,

--jeroen