PCC Load Balancing but to specify what ISP would reflect as source IP

mpuria · Mon Nov 07, 2022 5:23 am

Greetings to all here

We currently have multiple uplinks provided by different ISPs but for some reason, what we wanted to do is to specify WAN1 port's IP address to reflect when doing speedtest or when searching "What is my IP" but the rest of traffic would be routed to other WANs which have a higher bandwidth capacity.
(only our WAN1 has a dedicated public static IP with our ASN, other WANs have dynamic public IPs which is a headache for us) .
(sorry for my possible wrong chosen set of words)

WAN1=50MBPS Bandwidth
WAN2=300MBPS Bandwidth
WAN3=300MBPS Bandwidth
WAN4= 1GBPS Bandwidth
WAN5=1GBPS Bandwidth

with that said, we are currently using PCC Load Balancing to distribute the bandwidth usage, our only remaining problem is to somewhat be able to make/reflect the WAN1 public IP when conducting find my IP and Speedtest.net

This is the current configuration on a CCR1009:

/interface ethernet
set [ find default-name=ether1 ] name=ether1-50MBPS
set [ find default-name=ether2 ] name=ether2-300MBPS
set [ find default-name=ether3 ] name=ether3-300MBPS
set [ find default-name=ether4 ] name=ether4-1GBPS
set [ find default-name=ether5 ] name=ether5-1GBPS
set [ find default-name=ether6 ] name=ether6-LAN-DHCP
set [ find default-name=ether7 ] name=ether7-LAN-DHCP
/interface list
add name=LOCAL
add name=WAN
/routing table
add fib name=to_WAN1
add fib name=to_WAN2
add fib name=to_WAN3
add fib name=to_WAN4
add fib name=to_WAN5
/interface list member
add interface=DHCP-bridge list=LOCAL
add interface=ether1-50MBPS list=WAN
add interface=ether2-300MBPS list=WAN
add interface=ether3-300MBPS list=WAN
add interface=ether4-1GBPS list=WAN
add interface=ether5-1GBPS list=WAN
add interface=ether6-LAN-DHCP list=LOCAL
add interface=ether7-LAN-DHCP list=LOCAL
/ip firewall mangle
add action=mark-connection chain=input comment="INPUT MARK CONNECTION" \
    in-interface=ether1-50MBPS new-connection-mark=WAN1_conn passthrough=yes
add action=mark-connection chain=input in-interface=ether2-300MBPS \
    new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=input in-interface=ether3-300MBPS \
    new-connection-mark=WAN3_conn passthrough=yes
add action=mark-connection chain=input in-interface=ether4-1GBPS \
    new-connection-mark=WAN4_conn passthrough=yes
add action=mark-connection chain=input in-interface=ether5-1GBPS \
    new-connection-mark=WAN5_conn passthrough=yes
add action=mark-routing chain=output comment="OUTPUT MARK ROUTING" \
    connection-mark=WAN1_conn new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn \
    new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN3_conn \
    new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN4_conn \
    new-routing-mark=to_WAN4 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN5_conn \
    new-routing-mark=to_WAN5 passthrough=yes
add action=accept chain=prerouting in-interface-list=WAN
add action=mark-connection chain=prerouting comment=\
    "PRE ROUTING MARK CONNECTION ISP" dst-address-type=!local \
    in-interface-list=LOCAL new-connection-mark=WAN1_conn passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:8/0
add action=mark-connection chain=prerouting dst-address-type=!local \
    in-interface-list=LOCAL new-connection-mark=WAN1_conn passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:8/1
add action=mark-connection chain=prerouting dst-address-type=!local \
    in-interface-list=LOCAL new-connection-mark=WAN2_conn passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:8/2
add action=mark-connection chain=prerouting dst-address-type=!local \
    in-interface-list=LOCAL new-connection-mark=WAN3_conn passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:8/3
add action=mark-connection chain=prerouting dst-address-type=!local \
    in-interface-list=LOCAL new-connection-mark=WAN4_conn passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:8/4
add action=mark-connection chain=prerouting dst-address-type=!local \
    in-interface-list=LOCAL new-connection-mark=WAN4_conn passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:8/5
add action=mark-connection chain=prerouting dst-address-type=!local \
    in-interface-list=LOCAL new-connection-mark=WAN5_conn passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:8/6
add action=mark-connection chain=prerouting connection-rate=0-4294967295 \
    dst-address-type=!local in-interface-list=LOCAL new-connection-mark=\
    WAN5_conn passthrough=yes per-connection-classifier=\
    both-addresses-and-ports:8/7
add action=mark-routing chain=prerouting comment=\
    "PREROUTING MARK CONNECTION DHCP BRIDGE" connection-mark=WAN1_conn \
    new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn \
    new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN4_conn \
    new-routing-mark=to_WAN4 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3_conn \
    new-routing-mark=to_WAN3 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN5_conn \
    new-routing-mark=to_WAN5 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment=WAN1-----50MBPS out-interface=\
    ether1-50MBPS
add action=masquerade chain=srcnat comment=WAN2-----300MBPS out-interface=\
    ether2-300MBPS
add action=masquerade chain=srcnat comment=WAN3-----300MBPS out-interface=\
    ether3-300MBPS
add action=masquerade chain=srcnat comment=WAN4-----1GBPS out-interface=\
    ether4-1GBPS
add action=masquerade chain=srcnat comment=WAN5----1GBPS out-interface=\
    ether5-1GBPS
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.10.1 \
    pref-src="" routing-table=to_WAN1 scope=255 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.100.1 \
    pref-src="" routing-table=to_WAN2 scope=255 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=192.168.141.1 \
    pref-src="" routing-table=to_WAN3 scope=255 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.10.1 \
    pref-src="" routing-table=main scope=255 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.100.1 \
    pref-src="" routing-table=main scope=255 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.141.1 \
    pref-src="" routing-table=main scope=255 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.109.1 \
    pref-src="" routing-table=to_WAN4 scope=255 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.109.1 \
    pref-src="" routing-table=main scope=255 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ether5-1GBPS \
    pref-src="" routing-table=to_WAN5 scope=255 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ether5-1GBPS \
    pref-src="" routing-table=main scope=255 suppress-hw-offload=no \
    target-scope=10

sindy · Mon Nov 07, 2022 11:06 am

While most similar requests ask for prioritization of speedtest traffic to be able to cheat on the customers, the issue is the same - you can never handle all the services that inform the user about the public IP from which the request has arrived, because there's not only speedtest.net and whatismyip.com, and each domain name cna resolve to many IP numbers and this can change over time. Leaving aside that some IP addresses host multiple domains, so by sending traffic towards these IP addresses via WAN1, you would load that uplink much more than you expect.

Instead, I would suggest to spawn two CHRs in two unrelated datacenters, assign your public addresses to these CHRs, and create tunnels from your router via the individual uplinks to the CHRs. No need to encrypt the tunnels as the traffic proceeds further to open internet anyway. Of course the CHRs must be able to deal with twice the aggregate bandwidth of the uplinks.

anav · Mon Nov 07, 2022 1:33 pm

Your request, on the face of it, makes NO SENSE.

First, WAN1 has a known STATIC wanip and thus will always be the same regardless of how many times you ask whats my IP.
Second, you know the throughput, its 50Mbps

To reach your stated aim, simply hard code one PC on your LAN network only to WAN1 and you can play whats my IP and speed tests with that pc 24/7.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

However, I dont really think what you have stated is actually the requirement.
What questions are you trying to answer that you didnt articulate well?
Does this have anything to do with user requirements or a just for fun exercise?

mpuria · Sun Nov 13, 2022 6:03 pm

Thanks Sindy and Anav for the replies.

To clarify one thing: we don't aim to cheat customers for speedtest or find my IP websites.

Yes it has to do with user requirements and as well as to consider bandwidth management of our network.

What I wanted to achieve is to somewhat reflect the public IP of WAN1 without putting any significant strain on the link capacity of WAN1, and instead load balance the remaining traffic on the remaining WANs available.
Another way to put it is to make use of the public IP assigned to WAN1(I know that bandwidth would be utilized, but as much as possible to be lowest) but using the other WANs for heavy bandwidth loads.

Why I came to this idea you may ask?
Well, while observing the PCC load balancing method and conducting several speedtests(since our uplink providers are from different ISPs, we get different AS Names popping up), I noticed the IP address reflected on the speedtest page is different from the bandwidth that is being utilized while running the test. Also every time I refresh the page, the IP address that would show is randomly selected from the WAN public IP that we get.

sindy · Sun Nov 13, 2022 6:31 pm

To clarify one thing: we don't aim to cheat customers for speedtest or find my IP websites.

I've only mentioned that because the required behaviour was the same, but it was clear that the goal was not cheating, as you wanted to force speedtest to the slowest WAN.

What I wanted to achieve is to somewhat reflect the public IP of WAN1 without putting any significant strain on the link capacity of WAN1, and instead load balance the remaining traffic on the remaining WANs available.

But that would have no positive effect for the users, and would only confuse them.

Another way to put it is to make use of the public IP assigned to WAN1(I know that bandwidth would be utilized, but as much as possible to be lowest) but using the other WANs for heavy bandwidth loads.

The thing is that you don't know in advance which TCP session turns out to occupy a lot of bandwidth, and you cannot switch the WAN in the middle of a session.

while observing the PCC load balancing method and conducting several speedtests(since our uplink providers are from different ISPs, we get different AS Names popping up), I noticed the IP address reflected on the speedtest page is different from the bandwidth that is being utilized while running the test. Also every time I refresh the page, the IP address that would show is randomly selected from the WAN public IP that we get.

Not only this - it may also cause problems when a user connects to a website that uses multiple TCP sessions forming up a single "application session", checks that all the TCP sessions are initiated from the same public address, and rejects the application session if it doesn't.

So without engaging tunnels as suggested in my previous post, the only way to avoid all this is to set the per-connection-classifier to src-address, so that each customer would always use the same WAN. So the IP address shown in speedtest will be the one of the WAN through which the actual test sessions will be established.

anav · Sun Nov 13, 2022 8:27 pm

Which in the end does not better guarantee fair/equitable usage by users or divy up the load amongst disparate WAN throughputs any better.
In other words a fruitless exercise and where you should be concentrating your talents is on the PCC itself.

2650 in total:
WAN1=50MBPS Bandwidth 2%
WAN2=300MBPS Bandwidth 11%
WAN3=300MBPS Bandwidth 11%
WAN4= 1GBPS Bandwidth 38%
WAN5=1GBPS Bandwidth 38%

Therefore one should be looking at a logic that states, choose wan4 /wan5 19/50 times, choose wan2 / wan 3 approx 5/50 times (pick one of these for 6 times) and wan1 1/50 times
Clearly not so easy using PCC.

Perhaps this points to a better way??
https://mum.mikrotik.com/presentations/US12/tomas.pdf

sindy · Sun Nov 13, 2022 9:23 pm

If you want to make sure that the same client always uses the same WAN, per-connection-classifier is the only way that does not involve matching on src-address(-list) and allows to distribute the sessions in a pre-defined ratio. So I would be lazy and do the following to send 2 % of connections to WAN 1, 9.8% to WAN2 and WAN 3 each, and 39.2 % to WAN 4 and WAN 5 each:

/ip firewall mangle
add chain=prerouting                         per-connection-classifier=src-address:50/0 action=mark-connection new-connection-mark=via-WAN-1
add chain=prerouting connection-mark=no-mark per-connection-classifier=src-address:2/0  action=jump jump-target=pr-W24
add chain=prerouting connection-mark=no-mark per-connection-classifier=src-address:5/0  action=mark-connection new-connection-mark=via-WAN-3
add chain=prerouting connection-mark=no-mark                                            action=mark-connection new-connection-mark=via-WAN-5
add chain=pr-W24     connection-mark=no-mark per-connection-classifier=src-address:5/0  action=mark-connection new-connection-mark=via-WAN-2
add chain=pr-W24     connection-mark=no-mark                                            action=mark-connection new-connection-mark=via-WAN-4

I can justify the laziness by the fact that the real distribution of the actual bandwidth (not just connections) would only match the intended 1:6:6:20:20 ratio if there was a really high number of customers with identical traffic patterns. If that collides with your sense of perfection, you can replace each rule with per-connection-classifier=src-address:5/0 by three otherwise same rules with 13/0, 13/1, and 13/2.

PCC Load Balancing but to specify what ISP would reflect as source IP

PCC Load Balancing but to specify what ISP would reflect as source IP

Re: PCC Load Balancing but to specify what ISP would reflect as source IP

Re: PCC Load Balancing but to specify what ISP would reflect as source IP

Re: PCC Load Balancing but to specify what ISP would reflect as source IP

Re: PCC Load Balancing but to specify what ISP would reflect as source IP

Re: PCC Load Balancing but to specify what ISP would reflect as source IP

Re: PCC Load Balancing but to specify what ISP would reflect as source IP