Routing VLAN to specific WAN using Policy Routing

liviu2004 · Fri May 31, 2024 9:43 am

I need help with the following setup. Home user.

Attached network diagram. The traffic flow requirements are as follows:
1. VLAN 10, 20, 30 and 60 access to internet
2. VLAN no intercommunication with the exceptions:
a) VLAN 10 access to VLAN 40 for IP Camera and NAS
b) VLAN 30 access to VLAN 10 for printer services
3. VLAN 10, 20, 30 routed to WAN LTE19, which is on VLAN 50
4. VLAN 60 routed to WAN VDSL2 modem (KPN)
5. VLAN 10 needs local access to Mikrotik router, switch and access points.
6. VLAN 10 needs access to VLAN 50 for management purposes of the LTE19 modem.
7. No fail-over routes needed.
Added 02 June 2024: 8. Queues: VLAN 20 Internet of Things to get little bandwidth (500k/500k sufficient?). VLAN 10 Internal_LAN and VLAN 30 Work_Devices to be able to share the LTE WAN bandwith dynamically equal, which is I think PCQ method.

I moved away from IP Firewall Mangle routes and went to Policy Routing. But I struggle.

Issue is on item 3, at least as far I tested, VLAN 10 is not routed to WAN LTE19 but to WAN VDSL2, unless I disable the WAN VDSL2 modem (KPN). I am obviously missing something in my setup.

I have another issue with scripts, but I want to tackle one issue at the time.

# 2024-05-31 08:27:44 by RouterOS 7.15
# software id = LRF1-VRV8
#
# model = RB5009UPr+S+
# serial number = 
/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge1 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] mtu=1508 poe-out=off
set [ find default-name=ether2 ] poe-out=off
set [ find default-name=ether3 ] poe-out=off
set [ find default-name=ether5 ] poe-out=off
set [ find default-name=ether6 ] poe-out=off
set [ find default-name=ether7 ] poe-out=off
set [ find default-name=ether8 ] poe-out=off
set [ find default-name=sfp-sfpplus1 ] disabled=yes
/interface vlan
add interface=bridge1 name=IPTV vlan-id=60
add interface=bridge1 name=IP_camera_nas vlan-id=40
add interface=bridge1 name=Internal_LAN vlan-id=10
add interface=bridge1 name=Internet_of_Things vlan-id=20
add interface=bridge1 name=LTE19 vlan-id=50
add interface=bridge1 name=Work_Devices vlan-id=30
add interface=ether1 mtu=1508 name=vlan6 vlan-id=6
/interface list
add name=WAN
add name=VLAN
/ip pool
add name=Internal_LAN ranges=192.168.1.100-192.168.1.200
add name=Internet_of_Things ranges=10.0.20.100-10.0.20.200
add name=Work_Devices ranges=10.0.30.100-10.0.30.200
add name=IP_camera_nas ranges=10.0.40.100-10.0.40.200
add name=IPTV ranges=10.0.60.100-10.0.60.200
/ip dhcp-server
add address-pool=Internal_LAN interface=Internal_LAN lease-time=1d name=\
    Internal_LAN
add address-pool=Internet_of_Things interface=Internet_of_Things lease-time=\
    1d name=Internet_of_Things
add address-pool=Work_Devices interface=Work_Devices lease-time=1d name=\
    Work_Devices
add address-pool=IP_camera_nas interface=IP_camera_nas lease-time=1d name=\
    dhcp1
add address-pool=IPTV interface=IPTV lease-time=1d name=dhcp2
/ppp profile
set *FFFFFFFE on-up=pppoe_gateway only-one=yes use-upnp=no
/interface pppoe-client
add add-default-route=yes disabled=no interface=vlan6 keepalive-timeout=30 \
    max-mru=1500 max-mtu=1500 name=pppoe-out1 password=strongpassword profile=\
    default-encryption use-peer-dns=yes user=user
/queue type
add kind=pcq name=pcq-download pcq-classifier=dst-address
add kind=pcq name=pcq-upload pcq-classifier=src-address
/queue simple
add max-limit=130M/25M name=queue1 queue=pcq-download/pcq-upload target=LTE19
add max-limit=70M/30M name=queue2 queue=pcq-download/pcq-upload target=\
    pppoe-out1
/routing table
add disabled=no fib name=to_KPN
add disabled=no fib name=to_LTE19
/interface bridge port
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether4
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether5 pvid=10
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether6 pvid=10
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether7 pvid=10
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=sfp-sfpplus1 pvid=10
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether2 pvid=60
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether8 pvid=10
/ipv6 settings
set accept-router-advertisements=yes
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,ether3,ether4 untagged=\
    ether2,ether5,ether6,ether7,sfp-sfpplus1,ether8 vlan-ids=\
    10,20,30,40,50,60
/interface list member
add interface=Internal_LAN list=VLAN
add interface=Internet_of_Things list=VLAN
add interface=Work_Devices list=VLAN
add interface=IP_camera_nas list=VLAN
add interface=pppoe-out1 list=WAN
add interface=LTE19 list=WAN
add interface=ether1 list=WAN
/ip address
add address=10.0.10.2/24 interface=ether1 network=10.0.10.0
add address=192.168.1.1/24 interface=Internal_LAN network=192.168.1.0
add address=10.0.20.1/24 interface=Internet_of_Things network=10.0.20.0
add address=10.0.30.1/24 interface=Work_Devices network=10.0.30.0
add address=10.0.40.1/24 interface=IP_camera_nas network=10.0.40.0
add address=192.168.8.2/24 interface=LTE19 network=192.168.8.0
add address=10.0.60.1/24 interface=IPTV network=10.0.60.0
/ip dhcp-client
add interface=LTE19
/ip dhcp-server network
add address=10.0.20.0/24 gateway=10.0.20.1
add address=10.0.30.0/24 gateway=10.0.30.1
add address=10.0.40.0/24 gateway=10.0.40.1
add address=10.0.60.0/24 gateway=10.0.60.1
add address=192.168.1.0/24 gateway=192.168.1.1
/ip dns
set allow-remote-requests=yes
/ip firewall address-list
add address=192.168.1.0/24 list=connected_subnets
add address=10.0.20.0/24 list=connected_subnets
add address=10.0.30.0/24 list=connected_subnets
add address=10.0.40.0/24 list=connected_subnets
add address=192.168.8.0/24 list=connected_subnets
add address=10.0.10.0/24 list=connected_subnets
add address=10.0.60.0/24 list=commected_subnets
/ip firewall filter
add action=accept chain=input comment="Allow established, related, untracked" \
    connection-state=established,related,untracked
add action=reject chain=input comment="drop dns resolver" dst-port=53 \
    in-interface-list=WAN protocol=udp reject-with=icmp-network-unreachable
add action=reject chain=input comment="drop dns resolver" dst-port=53 \
    in-interface-list=WAN protocol=tcp reject-with=icmp-network-unreachable
add action=accept chain=input comment="accept icmp" protocol=icmp
add action=drop chain=input comment="drop all not coming from VLAN" \
    in-interface-list=!VLAN
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=fasttrack-connection chain=forward comment=fasttrack \
    connection-mark=no-mark hw-offload=yes
add action=accept chain=forward comment=\
    "accept established, related, untracked" connection-state=\
    established,related,untracked
add action=accept chain=forward comment=\
    "Allow access to IP camera from Internal LAN" dst-address=10.0.40.64 \
    in-interface=Internal_LAN out-interface=IP_camera_nas
add action=accept chain=forward comment=\
    "Allow access to NAS surveillance from Internal LAN" dst-address=\
    10.0.40.182 in-interface=Internal_LAN out-interface=IP_camera_nas
add action=accept chain=forward comment=\
    "allow access to printer from Work Devices" dst-address=192.168.1.5 \
    dst-address-list="" in-interface=Work_Devices out-interface=Internal_LAN
add action=drop chain=forward comment=\
    "no outside access to IP_camera_nas VLAN" in-interface=IP_camera_nas \
    out-interface-list=WAN
add action=accept chain=forward comment="VLAN internet access only" \
    connection-state=new in-interface-list=VLAN out-interface-list=WAN
add action=drop chain=forward comment="VLAN no inter communication" \
    in-interface=all-vlan out-interface=all-vlan
add action=drop chain=forward comment="drop all from WAN not DSTNATed" \
    connection-nat-state=!dstnat in-interface-list=WAN
add action=drop chain=forward comment="drop invalid" connection-state=invalid
/ip firewall mangle
add action=change-ttl chain=postrouting new-ttl=set:64 out-interface=LTE19 \
    passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
/ip route
add comment=to_KPN disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    195.190.228.96 routing-table=to_KPN scope=30 suppress-hw-offload=no \
    target-scope=10
add comment=to_LTE19 disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    10.0.0.1 routing-table=to_LTE19 scope=30 suppress-hw-offload=no \
    target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 address
add from-pool=ipv6_pool interface=Internal_LAN
add from-pool=ipv6_pool interface=Work_Devices
add from-pool=ipv6_pool interface=Internet_of_Things
/ipv6 dhcp-client
add interface=pppoe-out1 pool-name=ipv6_pool request=prefix
/ipv6 firewall address-list
add address=fe80::/10 comment="defconf: RFC6890 Linked-Scoped Unicast" list=\
    no_forward_ipv6
add address=ff00::/8 comment="defconf: multicast" list=no_forward_ipv6
add address=::1/128 comment="defconf: RFC6890 lo" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: RFC6890 IPv4 mapped" list=\
    bad_ipv6
add address=2001::/23 comment="defconf: RFC6890" list=bad_ipv6
add address=2001:db8::/32 comment="defconf: RFC6890 documentation" list=\
    bad_ipv6
add address=2001:10::/28 comment="defconf: RFC6890 orchid" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: RFC6890 Discard-only" list=\
    not_global_ipv6
add address=2001::/32 comment="defconf: RFC6890 TEREDO" list=not_global_ipv6
add address=2001:2::/48 comment="defconf: RFC6890 Benchmark" list=\
    not_global_ipv6
add address=fc00::/7 comment="defconf: RFC6890 Unique-Local" list=\
    not_global_ipv6
add address=::/128 comment="defconf: unspecified" list=bad_dst_ipv6
add address=::/128 comment="defconf: unspecified" list=bad_src_ipv6
add address=ff00::/8 comment="defconf: multicast" list=bad_src_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept ICMPv6 after RAW" \
    protocol=icmpv6
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept IPSec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept IPSec ESP" protocol=\
    ipsec-esp
add action=drop chain=input comment="defconf: drop all not coming from VLAN" \
    in-interface-list=!VLAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment="defconf: drop bad forward IPs" \
    src-address-list=no_forward_ipv6
add action=drop chain=forward comment="defconf: drop bad forward IPs" \
    dst-address-list=no_forward_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6 after RAW" \
    protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches IPSec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from VLAN" in-interface-list=\
    !VLAN
/ipv6 nd
set [ find default=yes ] advertise-dns=no disabled=yes hop-limit=64 \
    managed-address-configuration=yes other-configuration=yes ra-preference=\
    low reachable-time=5m
add advertise-dns=no interface=pppoe-out1 ra-lifetime=none ra-preference=low \
    reachable-time=5m
add dns=2a02:a47f:e000::54,2a02:a47f:e000::53 interface=Internal_LAN \
    other-configuration=yes ra-preference=high reachable-time=5m
add dns=2a02:a47f:e000::54,2a02:a47f:e000::53 interface=Work_Devices \
    other-configuration=yes ra-preference=high reachable-time=5m
add dns=2a02:a47f:e000::54,2a02:a47f:e000::53 interface=Internet_of_Things \
    other-configuration=yes ra-preference=high reachable-time=5m
/routing rule
add action=lookup-only-in-table comment="route VLAN10 to Mikrotik router" \
    disabled=no dst-address=192.168.1.0/24 src-address=192.168.1.0/24 table=\
    main
add action=lookup-only-in-table comment="route VLAN30 to VLAN10 printer" \
    disabled=no dst-address=192.168.1.0/24 src-address=10.0.30.0/24 table=\
    main
add action=lookup-only-in-table comment="route VLAN10 to VLAN50 modem" \
    disabled=no dst-address=192.168.8.0/24 src-address=192.168.1.0/24 table=\
    main
add action=lookup-only-in-table comment="route VLAN10 to VLAN40" disabled=no \
    dst-address=10.0.40.0/24 src-address=192.168.1.0/24 table=main
add action=lookup-only-in-table comment="route VLAN10 to ISP LTE19" disabled=\
    no src-address=192.168.1.0/24 table=to_LTE19
add action=lookup-only-in-table comment="route VLAN20 to ISP LTE19" disabled=\
    no src-address=10.0.20.0/24 table=to_LTE19
add action=lookup-only-in-table comment="route VLAN30 to ISP LTE19" disabled=\
    no src-address=10.0.30.0/24 table=to_LTE19
add action=lookup-only-in-table comment="route VLAN60 to ISP KPN" disabled=no \
    src-address=10.0.60.0/24 table=to_KPN
/system clock
set time-zone-name=Europe/Amsterdam
/system clock manual
set dst-delta=+01:00 dst-end="2024-10-27 03:00:00" dst-start=\
    "2024-03-31 02:00:00" time-zone=+01:00
/system identity
set name=Router
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp server
set broadcast=yes enabled=yes manycast=yes multicast=yes
/system ntp client servers
add address=0.nl.pool.ntp.org
add address=1.nl.pool.ntp.org
add address=2.nl.pool.ntp.org
add address=3.nl.pool.ntp.org
/system scheduler
add interval=1m name=schedule1 on-event=lte19_gateway policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2024-05-30 start-time=18:30:00
/system script
add dont-require-permissions=no name=pppoe_gateway owner=liviu policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    local pppoenewgw [/ip address get [find interface=\"pppoe-out1\"] network]\
    ;\r\
    \n:local pppoeroutegw [/ip route get [find comment=\"to_KPN\"] gateway];\r\
    \n:if (\$pppoenewgw != \$pppoeroutegw) do={\r\
    \n      /ip route set [find comment=\"to_KPN\"] gateway=\$pppoenewgw;\r\
    \n      /ip firewall connection {:foreach r in=[find] do={remove \$r}};\r\
    \n}"
add dont-require-permissions=no name=lte19_gateway owner=liviu policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    local newgw [ip dhcp-client get [find interface=\"LTE19\"] gateway];\r\
    \n:local routegw [/ip route get [find comment=\"to_LTE19\"] gateway];\r\
    \nif (\$newgw != \$routegw) do={\r\
    \n     /ip route set [find comment=\"to_LTE19\"] gateway=\$newgw;\r\
    \n     /ip firewall connection {:foreach r in=[find] do={remove \$r}};\r\
    \n}"

liviu2004 · Fri May 31, 2024 1:10 pm

I might of solve it.

Solution (to be confirmed long-term): remove default route from the WAN LTE19 DHCP-Client, remove add default route from the WAN pppoe-out1. And add Routing Rule to be able to access WAN KPN modem from VLAN10

/interface pppoe-client
add disabled=no interface=vlan6 keepalive-timeout=30 max-mru=1500 max-mtu=\
    1500 name=pppoe-out1 password=superpassword profile=default-encryption \
    use-peer-dns=yes user=user

/ip dhcp-client
add add-default-route=no interface=LTE19

/routing rule
add action=lookup-only-in-table comment="route VLAN10 to KPN modem" disabled=\
    no dst-address=10.0.10.0/24 src-address=192.168.1.0/24 table=main

anav · Fri May 31, 2024 1:49 pm

Well let us know for sure, as not going to look at it if solved LOL

liviu2004 · Fri May 31, 2024 9:51 pm

Nope, still issues. After a while, IoT devices started complaining about connectivity issues. Then I enabled Add Default Route to the WAN LTE19 DHCP client and things got restored.

So I am running in circles, if I enable add default routes of WAN, cannot control where traffic flows towards WAN, if I disable the default routes, traffic flows correctly, but the IoT devices have issues.

anav · Fri May 31, 2024 11:09 pm

Heheh, okay will look at it tomorrow, today is booked up or whats left of it.

Amm0 · Sat Jun 01, 2024 1:57 am

So I am running in circles, if I enable add default routes of WAN, cannot control where traffic flows towards WAN, if I disable the default routes, traffic flows correctly, but the IoT devices have issues.

The route always needs to exist in main, if it exists in a routing table. The PBR docs have subtle warning about this:

For a user-created table to be able to resolve the destination, the main routing table should be able to resolve the destination too.

So if you disable the default gateway, you're extra route table is not going to work. What you can do is set a higher distance in DHCP client or LTE APN profile, say default-route-distance=50 (to be safe, although 3 likely work).

Also, you can use that policy route with an action=drop - maybe that help cleanup the rules since you also have firewall rules too & that adds a lot of confusion here when combining the both /ip/firewall/filter and /routing/rules. And same Mikrotik PBR docs also mention**:

It is not recommended to use both methods at the same time or you should know exactly what you are doing. If you really do need to use both mangle and routing rules in the same setup then keep in mind that mangle has higher priority, meaning if the mangle marked traffic can be resolved in the table then route rules will never see this traffic.

** @anav, apparently a notable exception here is wireguard....

liviu2004 · Sat Jun 01, 2024 9:08 am

Thanks for the directions given. I'm avoiding combining Mange and Policy Routing, but I need that change the TTL on LTE.

Following changes now implemented:
- add default routes to both LTE and ppooe (VDSL2) WAN connections so they come into the Route List, and as you mentioned, it was also described in the docs example:

In our example, the main routing table should also have a route to destination x.x.x.x or at least a default route, since the default route is dynamically added by the DHCP for safety reasons it is better to add x.x.x.x also in the main table.

- changed default distance to VDSL2 modem (to_KPN) to 50 for testing purposes.
But I must admit I am very confused about this. Quoting Mikrotik:

Distance is the administrative distance. If you have multiple routes to the same destination network, the route with the lowest distance will be used.

If I specifically indicate that certain source address has to follow a certain route outside, to me, (which I am PLC programmer in my early years) then that is the rule and there is no discussion. I find networking to me more of an art than a precise engineering environment.

Nevertheless, back to testing for 1-2 days and see what happens. In the meantime, I am paying more attention towards your message on policy route with an action=drop.

/interface pppoe-client
add add-default-route=yes disabled=no interface=vlan6 keepalive-timeout=30 \
    max-mru=1500 max-mtu=1500 name=pppoe-out1 password=goodpassword profile=\
    default-encryption use-peer-dns=yes user=user
/ip dhcp-client
add interface=LTE19
/ip route
add comment=to_KPN disabled=no distance=50 dst-address=0.0.0.0/0 gateway=\
    195.190.228.96 routing-table=to_KPN scope=30 suppress-hw-offload=no \
    target-scope=10

liviu2004 · Sat Jun 01, 2024 9:16 am

This is weird. I am doing a speed test from VLAN10 and both WAN routes are used in the same time? This is really messed up somewhere.

Capture.JPG

anav · Sat Jun 01, 2024 4:23 pm

(1) REMOVE frame types from bridge. keep it simple, we add frame types and ingress filtering at /interface bridge ports.
(2) I like order, thus resorted vlans LOL. A cluttered config is a cluttered mind. ;-P
(3) For security added Trusted Interface, assuming the one subnet that is trusted is your internal network vlan10
(4) What is the purpose of queueing your WAN side?? I thought queuing was for the USER side??

(5) Missing IPTV from VLAN interface list !!!
(6) ordered bridge port by ports, funny that.
(7) Doing so, quickly clear that you setup sfp+1 when
a. its disabled in ethernet settings
b. not identified in your diagram as a used router port, so removed!!!
(8) /interface bridge vlans, WRONG in so many ways.
(9) FW rules simplified............
(10) How did you not see (sure RoS would have pointed it out to you) that your list name for firewall address list entry vlan60 IPTV was wrong????
add address=10.0.60.0/24 list=commected _subnets
(11) dont do ipv6 so your on your own for that.
(12) Missing main routes and they need to be differentiated by distance.
(13) No clue as to what you are trying to accomplish in scripts??? If they are for the purposes of routing etc, get rid of them. Will use routing rules much easier.

# model = RB5009UPr+S+
/interface bridge
add name=bridge1 vlan-filtering=yes

/interface vlan
add interface=ether1 mtu=1508 name=vlan6 vlan-id=6 comment="pppoe WAN vlan"
add interface=bridge1 name=Internal_LAN vlan-id=10
add interface=bridge1 name=Internet_of_Things vlan-id=20
add interface=bridge1 name=Work_Devices vlan-id=30
add interface=bridge1 name=IP_camera_nas vlan-id=40
add interface=bridge1 name=LTE19 vlan-id=50 comment="LTE WAN vlan"
add interface=bridge1 name=IPTV vlan-id=60

/interface list
add name=WAN
add name=VLAN
add name=TRUSTED

/interface bridge port
add bridge=bridge1 ingress-filtering=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether2 pvid=60
add bridge=bridge1 ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge1 ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether4
add bridge=bridge1 ingress-filtering=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether5 pvid=10
add bridge=bridge1 ingress-filtering=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether6 pvid=10
add bridge=bridge1 ingress-filtering=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether7 pvid=10
add bridge=bridge1 ingress-filtering=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether8 pvid=10

/interface bridge vlan
add bridge=bridge1 tagged=bridge1,ether3,ether4 untagged=ether5,ether6,ether7,ether8 vlan-ids=10
add bridge=bridge1 tagged=bridge1,ether3,ether4 vlan-ids=20,30,40,50
add bridge=bridge1 tagged=bridge1,ether3,ether4 untagged=ether2 vlan-ids=60

/interface list member
add interface=Internal_LAN list=VLAN
add interface=Internet_of_Things list=VLAN
add interface=Work_Devices list=VLAN
add interface=IP_camera_nas list=VLAN
add interface=IPTV list=VLAN
add interface=pppoe-out1 list=WAN
add interface=LTE19 list=WAN
add interface=Internal_LAN list=TRUSTED

/ip neighbor discovery-settings
set discover-interface-list=TRUSTED

/ip firewall address-list { use static dhcp leases }
add address=192.168.1.X list=Authorized comment="admin pc wired"
add address=192.168.1.Y list=Authorized comment="admin laptop wifi"
add address=10.0.40.64/32 list=Common-Destination
add address=10.0.40.182/32 list=Common-Destination

/ip firewall filter
{default rules to keep}
add action=accept chain=input comment="Allow established, related, untracked" \
connection-state=established,related,untracked
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=accept chain=input comment="accept icmp" protocol=icmp
{admin rules}
add action=accept chain=input in-interface-list=TRUSTED src-address-list=Authorized comment="admin access"
add action=accept chain=input in-interface-list=VLAN dst-port=53,123 protocol=udp comment='users to services'
add action=accept chain=input in-interface-list=VLAN dst-port=53 protocol=tcp comment='users to services'
add action=drop chain=input comment="drop all else" { put this rule in as the very last rule to insert }
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
{default rules to keep}
add action=fasttrack-connection chain=forward comment=fasttrack \
connection-mark=no-mark hw-offload=yes disabled=yes { if you remove queueing you can enable this!! }
add action=accept chain=forward connection-state=established,related,untracked
add action=drop chain=forward comment="drop invalid" connection-state=invalid
{admin rules}
add action=accept chain=forward comment="internet traffic but not IP camera" in-interface-list=VLAN out-interface-list=WAN src-address=!10.0.40.0/24
add action=accept chain=forward comment="Internal LAN to IP Camera" in-interface=Internal_LAN dst-address-list=Common-Destination
add action=accept chain=forward comment="Work Devices to Printer" in-interface=Work_Devices dst-address=192.168.1.5/32
add action=accept chain=forward comment="admin to all vlans" in-interface-list=TRUSTED src-address=Authorized
add action=drop chain=forward comment="drop all else"

/ip route
add comment=ether5-LTE distance=1 dst-address=0.0.0.0/0 gateway=10.0.0.1 routing-table=main
add comment=ether1-KLN distance=2 dst-address=0.0.0.0/0 gateway=195.190.228.96 routing-table=main
add comment="special table LTE" dst-address=0.0.0.0/0 gateway=10.0.0.1 routing-table=to_LTE
add comment="special table KLN" dst-address=0.0.0.0/0 gateway=195.190.228.96 routing-table=to_KLN

/routing rules
add min-prefix=0 action=lookup-only-in-table table=main comment="allows all local traffic first"
add src-address=192.168.1.0/24 action=lookup-only-in-table table=to_LTE comment="vlan10 Internal ONLY to LTE"
add src-address=10.0.20.0/24 action=lookup-only-in-table table=to_LTE comment="vlan20 IoT ONLY to LTE"
add src-address=10.0.30.0/24 action=lookup-only-in-table table=to_LTE comment="vlan30 Work ONLY to LTE"
add src-address=10.0.60.0/24 action=lookup-only-in-table table=to_KLN comment="vlan60 Work ONLY to DSL"

/tool mac-server mac-winbox
set allowed-interface-list=TRUSTED

liviu2004 · Sat Jun 01, 2024 10:04 pm

Thank you very much, I am f...ng impressed. I need some days to digest this.

I need to provide some feedback to some common sense items you mentioned.

(3) For security added Trusted Interface, assuming the one subnet that is trusted is your internal network vlan10

nice one

(4) What is the purpose of queueing your WAN side?? I thought queuing was for the USER side??

Yes, no, I don't know?? It seemed to work fine? Did I discover something good just trying what no one thought about?
But you're having a point, I think I was lazy. It is actually not bad to limit Internet_of_Things to something very slow. But I am unsure how to PCQ Work_Devices and Internal_LAN VLANs. I have to study and test more.

(7) Doing so, quickly clear that you setup sfp+1 when
a. its disabled in ethernet settings
b. not identified in your diagram as a used router port, so removed!!!

For the moment, until the damn fiber comes inside, it is just outside of the door for 1 year now! Well done Open Dutch Fiber, I made an subscription 1 year ago and you don't even answer my phone calls!

(8) /interface bridge vlans, WRONG in so many ways.

Me innocent, I followed a guided tutorial from here on the forum. Seemed to work (except this routing story).

(9) FW rules simplified............

Noted, less rules, faster processing. Makes sense.

(10) How did you not see (sure RoS would have pointed it out to you) that your list name for firewall address list entry vlan60 IPTV was wrong????
add address=10.0.60.0/24 list=commected _subnets

No, ROS is fine, I typed that by hand in the forum, in ROS was ok.

LTE does not know IPV6 yet, so I skip that for the moment.

(12) Missing main routes and they need to be differentiated by distance.

We need to speak here if I got this right. I wasn't missing the main routes, because they were automatically added by the pppoe and DHCP Client WAN sides. And I could not edit them and change distance. So at this moment when I write this, I removed adding of default routes and made them as per your instructions.

(13) No clue as to what you are trying to accomplish in scripts??? If they are for the purposes of routing etc, get rid of them. Will use routing rules much easier.

My LTE gateway changes every 24h, from 100.x to 10.x or viceversa. I needed this script to automatically capture that and update the routes. Last night logs captured that beautifully, so they work. Same for pppoe. Credit to this gent: viewtopic.php?t=139597

As for the config, I added back firewall input rules to drop DNS queries at port 53 udp tcp protocols, I got myself in trouble with my ISP many years ago and I was left without internet for some time.

And small errors corrected to_KPN instead of to_KLN, vlan60 IPTV ONLY to DSL instead of vlan60 Work ONLY to DSL

liviu2004 · Sat Jun 01, 2024 10:34 pm

1st issue encountered, and seen this before: all IoT devices in house lack internet connectivity. Example, Home Zone Heater shows status: Connection issue. Light controller shows no internet connection. Washer and dryer no data shown in phone app. In router, light controller connects and disconnects from dhcp server every minute.

anav · Sun Jun 02, 2024 2:38 am

As usual I work from your latest config, so would need to see it to comment on any new issues. Unless you changed something vlan20 should work same as vlan30 as they are identical in terms of the RB5009 router, which leads me to suspect the problem is down the road like at a switch.

(4) I would disable queueing now and describe in requirement statement what it is you wish to accomplish.
Then a config strategy can be discussed.

(13) I understand LTE requiring a script if its doing something funky, but PPPOE should automatically udpate. ( try it without a script and see what happens )

(14) Firewall rules. Input chain.

Think of the logic instead of an irrational fear due to past events.
WE ONLY ALLOW traffic FROM LAN to dst-port 53
WE THEN DROP ALL other traffic.
Dont need extra rules, they will never be seen because we DROP ALL OTHER TRAFFIC!!!

My bad on wrong wording.......

liviu2004 · Sun Jun 02, 2024 10:16 am

For testing, I connected my phone to Internet_of_Things and I can browse no problem. Funky these IoT devices. I also connected light controller to Internal_LAN and was no improvement.

Point taken on (14) and it makes sense, I think my brain was locked for many many years.

(4) I haven't include it yet. Low priority. I have updated the requirements into the 1st post, item 8.

(3) pppoe_gateway script removed. Not a big issue, this gateway rarely changes, if ever.

I also notice the order of some lines in the script, like /interface vlan, but this is how the export is done by ROS.

I had to add this line to be able to access the vdsl2 modem:
/interface list member
add interface=ether1 list=WAN

Log is riddled with assignments/de-assignments of IoT devices trying to get something.

# 2024-06-02 09:06:45 by RouterOS 7.15
# software id = LRF1-VRV8
#
# model = RB5009UPr+S+
# serial number =
/interface bridge
add name=bridge1 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] mtu=1508 poe-out=off
set [ find default-name=ether2 ] poe-out=off
set [ find default-name=ether3 ] poe-out=off
set [ find default-name=ether4 ] poe-out=forced-on
set [ find default-name=ether5 ] poe-out=off
set [ find default-name=ether6 ] poe-out=off
set [ find default-name=ether7 ] poe-out=off
set [ find default-name=ether8 ] poe-out=off
/interface vlan
add interface=bridge1 name=IPTV vlan-id=60
add interface=bridge1 name=IP_camera_nas vlan-id=40
add interface=bridge1 name=Internal_LAN vlan-id=10
add interface=bridge1 name=Internet_of_Things vlan-id=20
add comment="LTE WAN vlan" interface=bridge1 name=LTE19 vlan-id=50
add interface=bridge1 name=Work_Devices vlan-id=30
add comment="pppoe WAN vlan" interface=ether1 mtu=1508 name=vlan6 vlan-id=6
/interface list
add name=WAN
add name=VLAN
add name=TRUSTED
/ip pool
add name=Internal_LAN ranges=192.168.1.100-192.168.1.200
add name=Internet_of_Things ranges=10.0.20.100-10.0.20.200
add name=Work_Devices ranges=10.0.30.100-10.0.30.200
add name=IP_camera_nas ranges=10.0.40.100-10.0.40.200
add name=IPTV ranges=10.0.60.100-10.0.60.200
/ip dhcp-server
add address-pool=Internal_LAN interface=Internal_LAN lease-time=1d name=\
    Internal_LAN
add address-pool=Internet_of_Things interface=Internet_of_Things lease-time=\
    1d name=Internet_of_Things
add address-pool=Work_Devices interface=Work_Devices lease-time=1d name=\
    Work_Devices
add address-pool=IP_camera_nas interface=IP_camera_nas lease-time=1d name=\
    dhcp1
add address-pool=IPTV interface=IPTV lease-time=1d name=dhcp2
/interface pppoe-client
add disabled=no interface=vlan6 keepalive-timeout=30 max-mru=1500 max-mtu=\
    1500 name=pppoe-out1 password=nicepassword profile=default-encryption \
    use-peer-dns=yes user=user
/routing table
add disabled=no fib name=to_KPN
add disabled=no fib name=to_LTE19
/interface bridge port
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether2 pvid=60
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=ether4
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether5 pvid=10
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether6 pvid=10
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether7 pvid=10
add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged \
    interface=ether8 pvid=10
/ip neighbor discovery-settings
set discover-interface-list=TRUSTED
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,ether3,ether4 untagged=\
    ether5,ether6,ether7,ether8 vlan-ids=10
add bridge=bridge1 tagged=bridge1,ether3,ether4 vlan-ids=20,30,40,50
add bridge=bridge1 tagged=bridge1,ether3,ether4 untagged=ether2 vlan-ids=60
/interface list member
add interface=Internal_LAN list=VLAN
add interface=Internet_of_Things list=VLAN
add interface=Work_Devices list=VLAN
add interface=IP_camera_nas list=VLAN
add interface=IPTV list=VLAN
add interface=LTE19 list=WAN
add interface=Internal_LAN list=TRUSTED
add interface=pppoe-out1 list=WAN
add interface=ether1 list=WAN
/ip address
add address=10.0.10.2/24 interface=ether1 network=10.0.10.0
add address=192.168.1.1/24 interface=Internal_LAN network=192.168.1.0
add address=10.0.20.1/24 interface=Internet_of_Things network=10.0.20.0
add address=10.0.30.1/24 interface=Work_Devices network=10.0.30.0
add address=10.0.40.1/24 interface=IP_camera_nas network=10.0.40.0
add address=192.168.8.2/24 interface=LTE19 network=192.168.8.0
add address=10.0.60.1/24 interface=IPTV network=10.0.60.0
/ip dhcp-client
add add-default-route=no interface=LTE19
/ip dhcp-server lease
add address=192.168.1.200 client-id=1:74:e6:e2:da:9b:c mac-address=\
    74:E6:E2:DA:9B:0C server=Internal_LAN
add address=192.168.1.197 client-id=1:d8:fc:93:e4:35:f9 mac-address=\
    D8:FC:93:E4:35:F9 server=Internal_LAN
/ip dhcp-server network
add address=10.0.20.0/24 gateway=10.0.20.1
add address=10.0.30.0/24 gateway=10.0.30.1
add address=10.0.40.0/24 gateway=10.0.40.1
add address=10.0.60.0/24 gateway=10.0.60.1
add address=192.168.1.0/24 gateway=192.168.1.1
/ip firewall address-list
add address=192.168.1.200 comment="admin pc wired" list=Authorized
add address=192.168.1.197 comment="admin laptop wifi" list=Authorized
add address=10.0.40.64 list=Common-Destination
add address=10.0.40.182 list=Common-Destination
/ip firewall filter
add action=accept chain=input comment="Allow established, related, untracked" \
    connection-state=established,related,untracked
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=accept chain=input comment="accept icmp" protocol=icmp
add action=accept chain=input comment="admin access" in-interface-list=\
    TRUSTED src-address-list=Authorized
add action=drop chain=input comment="drop all else"
add action=fasttrack-connection chain=forward comment=fasttrack \
    connection-mark=no-mark hw-offload=yes
add action=accept chain=forward comment=\
    "Allow established, related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=accept chain=forward comment="internet traffic but not IP camera" \
    in-interface-list=VLAN out-interface-list=WAN src-address=!10.0.40.0/24
add action=accept chain=forward comment="Internal LAN to IP Camera" \
    dst-address-list=Common-Destination in-interface=Internal_LAN
add action=accept chain=forward comment="Work Devices to Printer" \
    dst-address=192.168.1.5 in-interface=Work_Devices
add action=drop chain=forward comment="drop all else"
/ip firewall mangle
add action=change-ttl chain=postrouting new-ttl=set:64 out-interface=LTE19 \
    passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
/ip route
add comment=ether5-LTE distance=1 dst-address=0.0.0.0/0 gateway=10.0.0.1 \
    routing-table=main
add comment=ether1-KPN disabled=no distance=2 dst-address=0.0.0.0/0 gateway=\
    195.190.228.96 routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add comment="special table LTE" dst-address=0.0.0.0/0 gateway=10.0.0.1 \
    routing-table=to_LTE19
add comment="special table KPN" disabled=no distance=1 dst-address=0.0.0.0/0 \
    gateway=195.190.228.96 routing-table=to_KPN scope=30 suppress-hw-offload=\
    no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/routing rule
add action=lookup-only-in-table comment="allows all local traffic first" \
    min-prefix=0 table=main
add action=lookup-only-in-table comment="vlan10 Internal ONLY to LTE" \
    src-address=192.168.1.0/24 table=to_LTE19
add action=lookup-only-in-table comment="vlan20 IoT ONLY to LTE" src-address=\
    10.0.20.0/24 table=to_LTE19
add action=lookup-only-in-table comment="vlan30 Work ONLY to LTE" \
    src-address=10.0.30.0/24 table=to_LTE19
add action=lookup-only-in-table comment="vlan60 IPTV ONLY to DSL" disabled=no \
    src-address=10.0.60.0/24 table=to_KPN
/system clock
set time-zone-name=Europe/Amsterdam
/system clock manual
set dst-delta=+01:00 dst-end="2024-10-27 03:00:00" dst-start=\
    "2024-03-31 02:00:00" time-zone=+01:00
/system identity
set name=Router
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp server
set broadcast=yes enabled=yes manycast=yes multicast=yes
/system ntp client servers
add address=0.nl.pool.ntp.org
add address=1.nl.pool.ntp.org
add address=2.nl.pool.ntp.org
add address=3.nl.pool.ntp.org
/system scheduler
add interval=1m name=schedule1 on-event=lte19_gateway policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2024-06-01 start-time=21:30:00
/system script
add dont-require-permissions=no name=lte19_gateway owner=liviu policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
    local newgw [ip dhcp-client get [find interface=\"LTE19\"] gateway];\r\
    \n:local routegw [/ip route get [find comment=\"ether5-LTE\"] gateway];\r\
    \nif (\$newgw != \$routegw) do={\r\
    \n     /ip route set [find comment=\"ether5-LTE\"] gateway=\$newgw;\r\
    \n     /ip route set [find comment=\"special table LTE\"] gateway=\$newgw;\
    \r\
    \n     /ip firewall connection {:foreach r in=[find] do={remove \$r}};\r\
    \n}"
/tool mac-server mac-winbox
set allowed-interface-list=TRUSTED

Capture.JPG

anav · Sun Jun 02, 2024 3:53 pm

Are they connected wired or wifi,
Check the switch and AP devices, dont think its the router??

liviu2004 · Sun Jun 02, 2024 5:55 pm

All wifi and some connected to the AP which is directly wired to the router. APs are TP-Link EAP245 and their VLAN setup is pretty straight forward.

It must be the router, if I just make a simple config, one WAN, no vlans, more or less default, without changing APs and Switch, they all work fine. But I also cannot figure it out what is not ok, but I am guessing these IoTs seem pretty demanding. To be followed ...

anav · Sun Jun 02, 2024 6:39 pm

Concur on the TPLINK not too much to screw up there, but what about the switch??

Okay I went back and what troubled me was LTE was on bridge1 and not directly on an etherport on the router.
I then checked the diagram and for some strange reason its coming from the AP ????????
......

ap-router.jpg

..........

Its probably convenience but now you have an AP accepting a WIRED (not wifi) LTE signal and this is going to the ROUTER.
HOW, which AP and what are the settings on this AP.

The other issue I am having with this is ............ HOW to the LTE modem packets get tagged by vlan50. Is that done by the MODEM ( done by ISP??)
If so, good because that is one issue.
The other issue is
a. you have assigned an IP address to the vlan50
b. you also use IP DHCP client set to the same interface.........
CANT DO BOTH>

add address=192.168.8.2/24 interface=LTE19 network=192.168.8.0
/ip dhcp-client
add add-default-route=no interface=LTE19

So we have some more work to do!!!
If the 192.168.8.0 network has nothing to do with your actual assigned IP from LTE, then perhaps removed the address and stick with DHCP client???

liviu2004 · Sun Jun 02, 2024 9:28 pm

Switch is an CSS610-8P and is as well pretty simple to configure. Done one or two mistakes there in the past and it becomes very obvious when you do that, because of services not working.

switch2.JPG

switch1.JPG

Indeed, LTE is not directly connected to Router. I tried 4h to pull a cable next to another cable, in a conduit in the walls, but failed. So I searched and found that I can VLAN that WAN traffic. But yeah, it might mess things up. LTE modem needs to be up in the house in order to get some decent speeds, if I place it downstairs next to the router, I get just rubbish. For testing, I can do that, should not be a problem.

I'm willing to pull some fiber from router to LTE, one thin strand would pull, unsure about turn radius in the walls. And unaware what to buy and how to terminate. Need time to learn.

The way LTE is tagged is done by the AP, EAP245 has a second LAN port which can tag the traffic to it. So I did.

tag50.JPG

Good catch on the IP address, to my understanding I needed an IP assigned to LTE VLAN, so I can speak to the LTE for admin purposes. LTE modem LAN side is 192.168.8.1. I now removed 192.168.8.2 from LTE19 VLAN interface, and I can still access the modem, which is good. But the IoTs are still not happy.

So going forward for tomorrow evening - night, move LTE modem to router port 5, reconfig and test again. Thanks so far.

liviu2004 · Sun Jun 02, 2024 10:44 pm

For testing, I removed all to do with pppoe-out1 config, and reverted to classic config, no Policy Routing. VLANs not touched. All work as should, so the LTE VLAN50 route through AP works?

anav · Mon Jun 03, 2024 1:00 am

Too simplistic.
If you want to deviate from a logical config and measured troubleshooting steps, you are on your own.

Before I go, just to let you know from the TPLink Manual from the latest version firwmare.

3.3 Configure VLAN
Wireless VLAN is used to set VLANs for the wireless networks. With this feature, the EAP
can work together with the switches supporting 802.1Q VLAN. Traffic from the clients
in different wireless networks is added with different VLAN tags according to the VLAN
settings of the wireless networks. Then the wireless clients in different VLANs cannot
directly communicate with each other. Note that the traffic from the wired clients will not
be added with VLAN tag

It appears to be a simple switch port which allow you to attach a PC or the like to whatever is assigned to ether1.
I suppose this could be tested by plugging in a laptop and seeing which IP it gets assigned, I would assume in this case one from vlan10.

Switch looks good after a quick review.

Hence, one cannot be sure what is going on with the AP.
You would be better off with a small tplink 5 port or netgear 5 port managed switch.
It would be connected to the router, the ISP modem and the AP and do things properly.

liviu2004 · Mon Jun 03, 2024 8:12 pm

That part from the manual is new to me. It could be right, it could be wrong, I expect the later, considering my setup works stand-alone with LTE interface VLAN. Mikrotik would not accept untagged packages on a trunk port. The tagging of ETH VLAN is introduced with the latest firmware of v3 and it is only present at v3:

8. Add support for ETH VLAN in both Standalone and Controller mode

Nevertheless, I think I got some impressive support here and I should be on my way to test further. I have sufficient mikrotik gear in house, to name a few, another RB5009, RB493G, RB450, RB450x4 with which I can insert in circuit for further VLAN tagging tests.

Routing VLAN to specific WAN using Policy Routing

Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing

Re: Routing VLAN to specific WAN using Policy Routing