Strange problems with PCC LB, packet mark problem?

xtornado · Fri Mar 15, 2019 10:49 am

Hello

Can you please check my configuration and tell me what is wrog? it will make me crazy

Configuration:

/system routerboard pr
       routerboard: yes
             model: CCR1009-8G-1S-1S+
     firmware-type: tilegx
  factory-firmware: 3.22
  current-firmware: 6.44
  upgrade-firmware: 6.44

I have a setup PCC LB setup over 4 VDSL lines, all local LAN's use the LB rules.. and i have this problems:

1. Sometimes connections from local LAN 192.168.5.0/24 (SIP phones), cannot reach the LAN subnet 192.168.2.0/24 (address 192.168.2.21 - asterisk) , and i have figured out that some times connections is mark as one of PCC LB connection mark pppoe-dsl-1-pcc-con ... and there is no reason to mark local lan to local lan like that :/ (so i remove connection few times until pppoe-dsl-1-pcc-mark is not wissible on that session in /ip firewall connections.

2. 10.17.1.0/24 is my OSPF range - in connections i see that simmilar problem, but in these case OSPF address marks multicast 224.0.0.5 as one of connections for loadbalance pcc ... (see picture)

3. And if i VPN to router (PPtP) get address 10.0.1.8, local addrs 10.0.1.1 (simple pptp config) , cannot reach nothing except 10.0.1.1 and all subnets gateway , ex 192.168.2.1 but not 192.168.2.21 witch is in the same subnet, until i set this:

/ip firewall nat
add action=masquerade chain=srcnat disabled=yes dst-address-list=local-addr src-address=10.0.1.0/24

router config:

/ip firewall address-list

add address=192.168.0.0/16 list=local-addr
add address=10.17.1.0/24 list=local-addr
add address=10.0.1.0/24 list=local-addr
add address=10.199.199.0/24 list=local-addr

add address=192.168.110.0/24 list=users_network
add address=192.168.111.0/24 list=users_network
add address=192.168.112.0/24 list=users_network
add address=192.168.113.0/24 list=users_network
add address=192.168.114.0/24 list=users_network
add address=192.168.210.0/23 list=users_network
add address=192.168.0.0/16 comment=" - load balancing addresses" list=\
    access_internet_pcc
add address=10.0.0.0/8 comment=" - load balancing addresses" disabled=\
    yes list=access_internet_pcc
add address=10.0.0.0/8 comment=" - private addresses" disabled=yes list=\
    private_addresses
add address=172.16.0.0/12 comment=" - private addresses" disabled=yes \
    list=private_addresses
add address=192.168.0.0/16 comment=" - private addresses" list=\
    private_addresses
add address=10.17.1.0/24 list=private_addresses
add address=10.17.1.0/24 comment=" - load balancing addresses" list=\
    access_internet_pcc

ip firewall mangle:

/ip firewall mangle

add action=mark-routing chain=prerouting comment=\
    " - mark routing - direct access (dsl-1)" dst-address-list=\
    !private_addresses dst-address-type=!local new-routing-mark=\
    pppoe-dsl-1-traffic passthrough=no src-address-list=\
    access_internet_pppoe-dsl-1
add action=mark-routing chain=prerouting comment=\
    " - mark routing - direct access (dsl-2)" dst-address-list=\
    !private_addresses dst-address-type=!local new-routing-mark=\
    pppoe-dsl-2-traffic passthrough=no src-address-list=\
    access_internet_pppoe-dsl-2
add action=mark-connection chain=input comment=\
    " - mark connection - IN dsl-1 -> OUT dsl-1" in-interface=\
    pppoe-dsl-1 new-connection-mark=pppoe-dsl-1-conn passthrough=yes
add action=mark-routing chain=output comment=\
    " - mark routing - IN dsl-1 -> OUT dsl-1" connection-mark=\
    pppoe-dsl-1-conn new-routing-mark=pppoe-dsl-1-traffic passthrough=no
add action=mark-connection chain=input comment=\
    " - mark connection - IN dsl-2 -> OUT dsl-2" in-interface=\
    pppoe-dsl-2 new-connection-mark=pppoe-dsl-2-conn passthrough=yes
add action=mark-routing chain=output comment=\
    " - mark routing - IN dsl-2 -> OUT dsl-2" connection-mark=\
    pppoe-dsl-2-conn new-routing-mark=pppoe-dsl-2-traffic passthrough=no
add action=mark-connection chain=input comment=\
    " - mark connection - IN dsl-3 -> OUT dsl-3" in-interface=\
    pppoe-dsl-3 new-connection-mark=pppoe-dsl-3-conn passthrough=yes
add action=mark-routing chain=output comment=\
    " - mark routing - IN dsl-3 -> OUT dsl-3" connection-mark=\
    pppoe-dsl-3-conn new-routing-mark=pppoe-dsl-3-traffic passthrough=no
add action=mark-connection chain=input comment=\
    " - mark connection - IN dsl-4 -> OUT dsl-4" in-interface=\
    pppoe-dsl-4 new-connection-mark=pppoe-dsl-4-conn passthrough=yes
add action=mark-routing chain=output comment=\
    " - mark routing - IN dsl-4 -> OUT dsl-4" connection-mark=\
    pppoe-dsl-4-conn new-routing-mark=pppoe-dsl-4-traffic passthrough=no
add action=mark-connection chain=forward comment=" - mark connection for \
    new conn (packet forward) - IN dsl-1 -> OUT dsl-1" connection-state=new \
    in-interface=pppoe-dsl-1 new-connection-mark=pppoe-dsl-1-pfw passthrough=\
    no
add action=mark-routing chain=prerouting comment=" - mark routing for new\
    \_conn (packet forward) - IN dsl-1 -> OUT dsl-1" connection-mark=\
    pppoe-dsl-1-pfw new-routing-mark=pppoe-dsl-1-traffic passthrough=no \
    src-address-list=port_forwarding_addresses
add action=mark-connection chain=forward comment=" - mark connection for \
    new conn (packet forward) - IN dsl-2 -> OUT dsl-2" connection-state=new \
    in-interface=pppoe-dsl-2 new-connection-mark=pppoe-dsl-2-pfw passthrough=\
    no
add action=mark-routing chain=prerouting comment=" - mark routing for new\
    \_conn (packet forward) - IN dsl-2 -> OUT dsl-2" connection-mark=\
    pppoe-dsl-2-pfw new-routing-mark=pppoe-dsl-2-traffic passthrough=no \
    src-address-list=port_forwarding_addresses
add action=mark-connection chain=forward comment=" - mark connection for \
    new conn (packet forward) - IN dsl-3 -> OUT dsl-3" connection-state=new \
    in-interface=pppoe-dsl-3 new-connection-mark=pppoe-dsl-3-pfw passthrough=\
    no
add action=mark-routing chain=prerouting comment=" - mark routing for new\
    \_conn (packet forward) - IN dsl-3 -> OUT dsl-3" connection-mark=\
    pppoe-dsl-3-pfw new-routing-mark=pppoe-dsl-3-traffic passthrough=no \
    src-address-list=port_forwarding_addresses
add action=mark-connection chain=forward comment=" - mark connection for \
    new conn (packet forward) - IN dsl-4 -> OUT dsl-4" connection-state=new \
    in-interface=pppoe-dsl-4 new-connection-mark=pppoe-dsl-4-pfw passthrough=\
    no
add action=mark-routing chain=prerouting comment=" - mark routing for new\
    \_conn (packet forward) - IN dsl-4 -> OUT dsl-4" connection-mark=\
    pppoe-dsl-4-pfw new-routing-mark=pppoe-dsl-4-traffic passthrough=no \
    src-address-list=port_forwarding_addresses
add action=accept chain=prerouting comment=" - accept (dsl-1)" \
    in-interface=pppoe-dsl-1
add action=accept chain=prerouting comment=" - accept (dsl-2)" \
    in-interface=pppoe-dsl-2
add action=accept chain=prerouting comment=" - accept (dsl-3)" \
    in-interface=pppoe-dsl-3
add action=accept chain=prerouting comment=" - accept (dsl-4)" \
    in-interface=pppoe-dsl-4
add action=mark-connection chain=prerouting comment=" - pcc 4/0 (dsl-1)" \
    dst-address-list=!private_addresses dst-address-type=!local \
    new-connection-mark=pppoe-dsl-1-pcc-conn passthrough=yes \
    per-connection-classifier=both-addresses:4/0 src-address-list=\
    access_internet_pcc
add action=mark-connection chain=prerouting comment=" - pcc 4/1 (dsl-2)" \
    dst-address-list=!private_addresses dst-address-type=!local \
    new-connection-mark=pppoe-dsl-2-pcc-conn passthrough=yes \
    per-connection-classifier=both-addresses:4/1 src-address-list=\
    access_internet_pcc
add action=mark-connection chain=prerouting comment=" - pcc 4/2 (dsl-3)" \
    dst-address-list=!private_addresses dst-address-type=!local \
    new-connection-mark=pppoe-dsl-3-pcc-conn passthrough=yes \
    per-connection-classifier=both-addresses:4/2 src-address-list=\
    access_internet_pcc
add action=mark-connection chain=prerouting comment=" - pcc 4/3 (dsl-4)" \
    dst-address-list=!private_addresses dst-address-type=!local \
    new-connection-mark=pppoe-dsl-4-pcc-conn passthrough=yes \
    per-connection-classifier=both-addresses:4/3 src-address-list=\
    access_internet_pcc
add action=mark-routing chain=prerouting comment=\
    " - mark routing for pcc (dsl-1)" connection-mark=\
    pppoe-dsl-1-pcc-conn new-routing-mark=pppoe-dsl-1-pcc passthrough=yes
add action=mark-routing chain=prerouting comment=\
    " - mark routing for pcc (dsl-2)" connection-mark=\
    pppoe-dsl-2-pcc-conn new-routing-mark=pppoe-dsl-2-pcc passthrough=yes
add action=mark-routing chain=prerouting comment=\
    " - mark routing for pcc (dsl-3)" connection-mark=\
    pppoe-dsl-3-pcc-conn new-routing-mark=pppoe-dsl-3-pcc passthrough=yes
add action=mark-routing chain=prerouting comment=\
    " - mark routing for pcc (dsl-4)" connection-mark=\
    pppoe-dsl-4-pcc-conn new-routing-mark=pppoe-dsl-4-pcc passthrough=yes

ip route

/ip route
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-1) - distance 1" distance=1 gateway=\
    pppoe-dsl-1 routing-mark=pppoe-dsl-1-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-2) - distance 2" distance=2 gateway=\
    pppoe-dsl-2 routing-mark=pppoe-dsl-1-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-3) - distance 3" distance=3 gateway=\
    pppoe-dsl-3 routing-mark=pppoe-dsl-1-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-4) - distance 4" distance=4 gateway=\
    pppoe-dsl-4 routing-mark=pppoe-dsl-1-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-2) - distance 1" distance=1 gateway=\
    pppoe-dsl-2 routing-mark=pppoe-dsl-2-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-1) - distance 2" distance=2 gateway=\
    pppoe-dsl-1 routing-mark=pppoe-dsl-2-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-3) - distance 3" distance=3 gateway=\
    pppoe-dsl-3 routing-mark=pppoe-dsl-2-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-4) - distance 4" distance=4 gateway=\
    pppoe-dsl-4 routing-mark=pppoe-dsl-2-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-3) - distance 1" distance=1 gateway=\
    pppoe-dsl-3 routing-mark=pppoe-dsl-3-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-1) - distance 2" distance=2 gateway=\
    pppoe-dsl-1 routing-mark=pppoe-dsl-3-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-2) - distance 3" distance=3 gateway=\
    pppoe-dsl-2 routing-mark=pppoe-dsl-3-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-4) - distance 4" distance=4 gateway=\
    pppoe-dsl-4 routing-mark=pppoe-dsl-3-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-4) - distance 1" distance=1 gateway=\
    pppoe-dsl-4 routing-mark=pppoe-dsl-4-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-1) - distance 2" distance=2 gateway=\
    pppoe-dsl-1 routing-mark=pppoe-dsl-4-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-2) - distance 3" distance=3 gateway=\
    pppoe-dsl-2 routing-mark=pppoe-dsl-4-traffic
add check-gateway=arp comment=\
    " - traffic route (pppoe-dsl-3) - distance 4" distance=4 gateway=\
    pppoe-dsl-3 routing-mark=pppoe-dsl-4-traffic
add check-gateway=arp comment=" - PCC route (pppoe-dsl-1) - distance 1" \
    distance=1 gateway=pppoe-dsl-1 routing-mark=pppoe-dsl-1-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-2) - distance 2" \
    distance=2 gateway=pppoe-dsl-2 routing-mark=pppoe-dsl-1-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-3) - distance 3" \
    distance=3 gateway=pppoe-dsl-3 routing-mark=pppoe-dsl-1-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-4) - distance 4" \
    distance=4 gateway=pppoe-dsl-4 routing-mark=pppoe-dsl-1-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-2) - distance 1" \
    distance=1 gateway=pppoe-dsl-2 routing-mark=pppoe-dsl-2-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-1) - distance 2" \
    distance=2 gateway=pppoe-dsl-1 routing-mark=pppoe-dsl-2-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-3) - distance 3" \
    distance=3 gateway=pppoe-dsl-3 routing-mark=pppoe-dsl-2-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-4) - distance 4" \
    distance=4 gateway=pppoe-dsl-4 routing-mark=pppoe-dsl-2-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-3) - distance 1" \
    distance=1 gateway=pppoe-dsl-3 routing-mark=pppoe-dsl-3-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-1) - distance 2" \
    distance=2 gateway=pppoe-dsl-1 routing-mark=pppoe-dsl-3-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-2) - distance 3" \
    distance=3 gateway=pppoe-dsl-2 routing-mark=pppoe-dsl-3-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-4) - distance 4" \
    distance=4 gateway=pppoe-dsl-4 routing-mark=pppoe-dsl-3-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-4) - distance 1" \
    distance=1 gateway=pppoe-dsl-4 routing-mark=pppoe-dsl-4-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-1) - distance 2" \
    distance=2 gateway=pppoe-dsl-1 routing-mark=pppoe-dsl-4-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-2) - distance 3" \
    distance=3 gateway=pppoe-dsl-2 routing-mark=pppoe-dsl-4-pcc
add check-gateway=arp comment=" - PCC route (pppoe-dsl-3) - distance 4" \
    distance=4 gateway=pppoe-dsl-3 routing-mark=pppoe-dsl-4-pcc
add check-gateway=ping comment=\
    " - default route (pppoe-dsl-3) - distance 1" distance=1 gateway=\
    pppoe-dsl-3
add check-gateway=arp comment=\
    " - default route (pppoe-dsl-2) - distance 1" distance=1 gateway=\
    pppoe-dsl-2
add check-gateway=arp comment=\
    " - default route (pppoe-dsl-1) - distance 1" distance=1 gateway=\
    pppoe-dsl-1
add check-gateway=ping comment=\
    " - default route (pppoe-dsl-4) - distance 1" distance=1 gateway=\
    pppoe-dsl-4
add disabled=yes distance=110 dst-address=10.17.1.19/32 gateway=10.17.1.78 \
    scope=20
add disabled=yes distance=110 dst-address=10.17.1.19/32 gateway=10.17.1.78 \
    scope=20
add distance=1 dst-address=10.17.1.90/32 gateway=10.17.1.34
add distance=1 dst-address=10.17.1.90/32 gateway=10.17.1.34

ip firewall nat

/ip firewall nat

add action=masquerade chain=srcnat comment="masquarade dsl1" out-interface=\
    pppoe-dsl-1
add action=masquerade chain=srcnat comment="masquarade dsl2" out-interface=\
    pppoe-dsl-2
add action=masquerade chain=srcnat comment="masquarade dsl3" out-interface=\
    pppoe-dsl-3
add action=masquerade chain=srcnat comment="masquarade dsl4" out-interface=\
    pppoe-dsl-4

Sob · Sat Mar 16, 2019 2:41 pm

1) Quick fix can be:

/ip route rule
add action=lookup-only-in-table dst-address=192.168.0.0/16 table=main
<same for other local subnets>

It doesn't prevent wrong connection marks and route marks based on them from being set, but will make sure that they won't affect traffic to local subnets.

anav · Sat Mar 16, 2019 3:02 pm

I think you may be onto something LOL.
What I see is that the OP has erroneously put distance rules in PCC route rules that use mark-routing!
(Yes on the default rules WITHOUT route-mark (main table) one can do that..........)
Thus I suspect that by doing so he is defeating PCC?????

My understanding is that:
a. the route rules for each WAN with mark-route require ping gateway but do not require distance. Thusly, if one WAN is not available, the router will distribute the load amongst the remaining WANs while checking to see if the unavailable WAN has come back up. The PCC algorithm decides which WAN traffic will go out on!!!!!
b. the route rules on the main table without mark-route do not require ping gateway but probably should get a distance difference if nothing else an order for the router to know where to send packets for any data not using the route rules with mark-route.

xtornado · Sun Mar 17, 2019 8:09 am

1) Quick fix can be:
Code: Select all
/ip route rule
add action=lookup-only-in-table dst-address=192.168.0.0/16 table=main
<same for other local subnets>
It doesn't prevent wrong connection marks and route marks based on them from being set, but will make sure that they won't affect traffic to local subnets.

Hello

Hmmm. I add this last night..
This rule seems to have solved the problems.
I do not know why I had to add this rule when all LAN subnets are in the main routing table...

Ip filter rule is usually used if I do a PBR between multiple vrf's?

Sob · Sun Mar 17, 2019 4:23 pm

Routes to all LAN subnets are in the main routing table, true. But if packet to LAN happens to get different routing mark, it means that route will be looked up in different routing table, not in main. And if that routing table doesn't have specific routes to LAN subnets, packet will be sent to internet. This routing rule overrides it and forces given destinations to be looked up only in main routing table. If you find how the unwanted marking happens and fix that, you won't need this rule.

xtornado · Sun Mar 17, 2019 4:32 pm

All working fine now after rule.. routing works fine and i have no more strange connection marks
Thx

anav · Sun Mar 17, 2019 4:48 pm

Routes to all LAN subnets are in the main routing table, true. But if packet to LAN happens to get different routing mark, it means that route will be looked up in different routing table, not in main. And if that routing table doesn't have specific routes to LAN subnets, packet will be sent to internet. This routing rule overrides it and forces given destinations to be looked up only in main routing table. If you find how the unwanted marking happens and fix that, you won't need this rule.

@Sob
I don't know what game you are playing but you are not following the rules LOL.
The accepted PCC method of doing what you trick phucked above is the following:

/ ip firewall mangle
add chain=prerouting dst-address=10.111.0.0/24 action=accept in-interface=LAN
add chain=prerouting dst-address=10.112.0.0/24 action=accept in-interface=LAN

/ ip route
add dst-address=0.0.0.0/0 gateway=10.111.0.1 routing-mark=to_ISP1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.112.0.1 routing-mark=to_ISP2 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.111.0.1 distance=1 check-gateway=ping
add dst-address=0.0.0.0/0 gateway=10.112.0.1 distance=2 check-gateway=ping

Now I will be honest here, I dont know why the heck we care a rats ass about traffic on the main table.
There is no effing traffic on the main table. We are mangling ALL LAN Traffic through PCC rules out the various WAN for load balancing.
Where is this magical mysterious OTHER traffic that needs the main table.
Very frustrating here as there seems to be no logic.

Sob · Sun Mar 17, 2019 4:56 pm

Then read the original post again. Some connections get marked, even though they should not. The config is posted, so go for it and find some typo that causes it. Or maybe there isn't one and it's something else. Or take take the easy way and use the nice little routing rule.

Oh and the other traffic is to LAN.

anav · Sun Mar 17, 2019 5:03 pm

If I was a Vulcan, the first person I would mind meld with is Sob, to steal err transfer all his MT knowledge!!!!
Then I too could be smugly be content to answer complex questions with riddles and half answers.

xtornado · Sun Mar 17, 2019 6:34 pm

I change from arp check to ping as requested

Sob · Sun Mar 17, 2019 9:08 pm

@anav: You can choose between short (and hopefully helpful) hints and nothing. I couldn't answer all your questions even if I cloned myself. Although, with enough copies, maybe. But the technology is not there yet.

Strange problems with PCC LB, packet mark problem? [SOLVED]

Strange problems with PCC LB, packet mark problem?

Re: Strange problems with PCC LB, packet mark problem? [SOLVED]

Re: Strange problems with PCC LB, packet mark problem?

Re: Strange problems with PCC LB, packet mark problem?

Re: Strange problems with PCC LB, packet mark problem?

Re: Strange problems with PCC LB, packet mark problem?

Re: Strange problems with PCC LB, packet mark problem?

Re: Strange problems with PCC LB, packet mark problem?

Re: Strange problems with PCC LB, packet mark problem?

Re: Strange problems with PCC LB, packet mark problem?

Re: Strange problems with PCC LB, packet mark problem?

Who is online