I cant quite wrap my head around this one...

rd228 · Sun May 13, 2018 7:41 pm

Hi all

Hoping for someone with a little more kwnoledge than myself to help me work this one out as I'm stumped!

I have BT Infinity Broadband, Fibre to the premises - 50mb Down / 10mb Up

I have replaced the BT Home Hub with a Mikrotik RB2011UiAS.

I have no problems getting connected to the BT Service using pppoe and I get my full speed when running a speed test with nice low pings below 5ms.
Here comes the interesting bit....

I have recently been seeing really high latency on my line 300ms+ which has also been slowing my broadband down to around 5-30mb down. I called BT to complain but they said they cant see a problem. So I switched out the Mikrotik for the Home Hub just to see if it was the router. I left the home hub in place for a couple hours and it was fine. I put the Mikrotik back in and after maybe 10/15 min the latency issue would come back.

I ran a test with the Mikrotik -

I disconnected everything from my network apart from the laptop I hard wired straight into the mikrotik
I ran a ping from CMD to google.com whilst running a speed test at the same time. Before the speedtest started I was seeing pings of sub 5ms as expected. When the test kicked off for the download portion, my ping jumped to around 40-50ms to google.com. Now here is the odd bit....when it got to the upload part of the speed test my CMD google ping spiked to 300ms+.

I thought that was odd so I did the same ping again but this time I downloaded an ISO from Ubuntu - Pings went up to around 30-50ms again. But for the upload test I uploaded something to dropbox.com. My pings to google.com spiked to over 300ms again.

I ran all of these tests again but with the BT Home hub this time and the pings whilst downloading jumped to around 30-50ms as they did with the mikrotik. When running an upload my pings remained at sub 5ms.

So can anyone explain why when doing an upload with the mikrotik my pings skyrocket basically making surfing for anyone else unusable? Uploads with the home hub are fine. I have just moved house and I am using the same mikrotik I did at the old house and I didn't experience this problem as far as I remember there.

Can anyone help me here?

Ross

RoadkillX · Sun May 13, 2018 8:34 pm

Can you check that you have the same MTU on the BT Home router and on the mikrotik pppoe connection?

rd228 · Sun May 13, 2018 11:16 pm

I cant find the MTU on the Home Hub 5 but I did the CMD test to work out what my MAX MTU is while I had the home hub connected

I ran ping www.google.com -f -l xxxx changing the number till I found the largest packet I could that didn't fragment and then added 28 to it. This gave me a Max MTU with the home hub of "1492"

I checked the mikrotik and the MTU settings on the pppoe interface are

MAX MTU : 1458
MAX MRU : 1458
MRRU : 1600

So my MTU is actually set less on the mikrotik to the MAX MTU I worked out with the above calculation for the Home Hub.

The MTU on the mikrotik eth1 interface; which is the interface the WAN connection for the pppoe to dial out of is as follows. (I don't know if the MTU on this port makes a difference or just on the pppoe?)

MAX MTU : 1500
MAX MRU : 1598

RoadkillX · Sun May 13, 2018 11:20 pm

Set on the mikrotik pppoe MTU=1492 and MRU=1492 and see if it improves.

rd228 · Sun May 13, 2018 11:25 pm

No Difference

Before the test starts -

Screen Shot 2018-05-13 at 21.22.44.png

During a download -

Screen Shot 2018-05-13 at 21.22.55.png

During an upload -

Screen Shot 2018-05-13 at 21.23.10.png

RoadkillX · Sun May 13, 2018 11:35 pm

How is your cpu usage on the router while running the speedtest? check your firewall if this rules exist and are enabled:

add action=fasttrack-connection chain=forward connection-state=established,related,untracked in-interface=MyPPPOE
add action=accept chain=forward connection-state=established,related,untracked in-interface=MyPPPOE

if they dont exist create them with your correct ppooe interface name make sure they are ordered before any Forward Drop Rules and repeat the speedtest.

rd228 · Sun May 13, 2018 11:37 pm

CPU is 50-60% during a speedtest download and around 30% during an upload.

I don't have these rules but will try them now and let you know!

rd228 · Sun May 13, 2018 11:45 pm

Added those rules - made no difference.

I also turned off ALL of my rules to see if it was one of those and that makes no difference either

RoadkillX · Sun May 13, 2018 11:49 pm

weird, last resource /export hide-sensitve if you can.

rd228 · Sun May 13, 2018 11:52 pm

weird, last resource /export hide-sensitve if you can.

Sorry I'm not sure what that is? Can you explain?

I copied /export hide-sensitive to the terminal but it says "expected end of command (line 1 column 9)

RoadkillX · Mon May 14, 2018 12:10 am

My bad, on the router terminal type export hide-sensitive without the / should give a dump of all the config for pasting here.

rd228 · Mon May 14, 2018 9:08 am

Hi

See attached config file - was a bit to long to put in a post.

There are quite a few firewall rules in there which I know you will probably pick up on and mention but most of them are switched off and were only used for testing. I do need to go through and tidy up the ones im no longer using.

RoadkillX · Mon May 14, 2018 10:00 am

Did you add all those config or were they there when you bought the router? And it's on version 6.34.2 latest is 6.42.1

* This was clearly a corporate router before i suggest a config reset upgrade and add your BT config.

rd228 · Mon May 14, 2018 10:23 am

Nope - Router was purchased brand new

Everything in there I have at some point put in place myself.

As I say most of it is disabled after I have tested something but I just never deleted it.

I have removed the file.

RoadkillX · Mon May 14, 2018 10:42 am

Well if you can't reset the device because you need the configs then upgrade to latest firmware, i went through the changelog and there are quite a few relevant updates for pppoe-client since 6.34.2

rd228 · Mon May 14, 2018 10:56 am

An update definitely needs to be done - I agree with you here.

I used to be on top of regular updates but now am a little apprehensive as most people are when you haven't upgraded in a while to do such a major upgrade so have kept putting it off!

I have backups so should be fine, this problem is just going to have to force me to upgrade now which is definitely a good thing.

I do need to clean up the config so will have to find a weekend some time to go through what I want to keep and export it then wipe the device and re import the configs I want to keep.

I will try an upgrade tonight and give you an update! Fingers crossed that solves the problem!

RoadkillX · Mon May 14, 2018 11:23 am

Quite sure upgrade will fix it, there are problems on that version with the pppoe-client MRU and fasttrack was not yet implemented that's why the rules added had no effect

, Give us an update when done!

rd228 · Mon May 14, 2018 11:58 am

Could you give me an explanation of the fasttrack rule? Whats its for and what it does and how it benefits?

I had a look on the wiki about it but it didn't really make sense to me....

RoadkillX · Mon May 14, 2018 12:22 pm

When you apply fasttrack to certain traffic it won't go through flter mangle or queues, so it doesn't hit the cpu which increases throughput, downside you won't be able to apply queues or firewall rules to that traffic. the default rule still gives quite a lot of control in firewall since it's only applies to rel,est connections, but you can't apply queues to it and that is just not acceptable if you have a low speed connection and need to manage bandwidth.

/ip firewall filter add chain=forward action=fasttrack-connection connection-state=established,related

This helped me understand better how it works.

https://youtu.be/8bl7V5iFVOc

*I initially had a CRS112 with 400mhz cpu and without fasttrack over pppoe fiber 300/300mbps the speedtest wouldn't go over 30mpbs and 100% cpu i even considered returning it but decided to read up before when i added the fasttrack rule to the pppoe connection i'd get 320/320mbps and around 30% cpu usage. That said ended up with an RB750Gr3 4 cores 880mhz

and can easly handle the pppoe max speed without the need for fasttrack so i can apply mangle and queue trees to all traffic.

tippenring · Tue May 15, 2018 7:31 am

I don't see a problem. Uploading at the max bandwidth of 10Mbps will result in high latency. There are ways to adjust for it with custom queuing to reduce the high latency for desired traffic, such as ping for example. It will result in a slight, not very noticeable reduced upload speed for the big packets. Basically you can give certain traffic types higher queue priorities so the desired packets get to "jump in line" ahead of the big packets in the queue.

Perhaps the router provided by your ISP already does this. I believe the MT uses a FIFO queuing by default. I'm sure someone will correct me if I'm wrong about that.

RoadkillX · Tue May 15, 2018 11:32 am

I don't see a problem. Uploading at the max bandwidth of 10Mbps will result in high latency. There are ways to adjust for it with custom queuing to reduce the high latency for desired traffic, such as ping for example. It will result in a slight, not very noticeable reduced upload speed for the big packets. Basically you can give certain traffic types higher queue priorities so the desired packets get to "jump in line" ahead of the big packets in the queue.

Perhaps the router provided by your ISP already does this. I believe the MT uses a FIFO queuing by default. I'm sure someone will correct me if I'm wrong about that.

High latency on a fiber connection is 150ms not 400ms that's just way to much and it doesn't happen when download/upload with original isp router.

rd228 · Tue May 15, 2018 4:48 pm

Didnt get a chance to upgrade the OS last night....probably wont get a chance now till Friday so will hopefully update you then!

Ross

tippenring · Tue May 15, 2018 5:45 pm

High latency on a fiber connection is 150ms not 400ms that's just way to much and it doesn't happen when download/upload with original isp router.

High latency is whatever increased delay happens as you approach 100% of the bandwidth limit. It might be 150ms worth of buffers, or it might be 500ms worth of buffers. Eventually packet loss occurs because either packets exceed the RTT limit of an endpoint host, or a device in the path drops the packet (in either direction) due to lack of buffer/queue space.

rd228 · Fri May 18, 2018 7:20 pm

I have factory reset the router and updated it to the latest configuration.

Still the same problem

.

All I configured after factory resetting it was the PPPoe Interface and the NAT rule to allow traffic out. Everything else was standard of a fresh install.

Ran a speed test - as before the ping jumped to around 40/50ms during a download but rocketed to over 500ms on an upload

Is there anything else I can try on this?

Ross

rd228 · Fri May 18, 2018 7:49 pm

Ok another test

I have put in a Simple queue against the machine I'm running the tests from

I have allowed the machine unlimited download as thats not a problem here but only allowed 5m upload from the 10m I have available to me in my package.

I ran another speed test and the queues are working as they should. During the upload speedtest the ping to google at the same time only topped out at 50-70ms during the upload test. So its better!

Again though - with the BT router I can run an upload speed test using the full bandwidth available to me and the pings stay below 5ms throughout the test. So what is going on with the mikrotik here.....

I really don't want to have to scrap this and look for an alternative...

R1CH · Fri May 18, 2018 9:46 pm

Sounds like the BT router has some AQM built in that you will need to replicate with RouterOS queue rules. Given the age of RouterOS kernel though it won't be able to compete with modern AQM like fq_codel (https://www.bufferbloat.net/projects/codel/wiki/) which is easy to set-and-forget.

rd228 · Fri May 18, 2018 9:57 pm

Thanks for your reply

Can anyone tell me how I do that?

Its odd though - as I mentioned in my first post...this router came from my old house and nothing on the config changed when it was moved to the new house with a new line. I never had this issue before I moved house! I actually have a RB750G I have now put in at my old house (parents house) to replace the one I took away with me which is running just fine and doesn't experience this upload issue as I've tested it.

The only difference is the old house was pppoe with copper to the cabinet then fibre to the exchange with 30mb down and 5mb up. The new house is pppoe as well, with fibre straight from my house to the exchange which I'm only a few meters from.

Ross

rd228 · Fri May 18, 2018 10:01 pm

As another note ....I also have a friend who has a RB750G who has just installed it on his virgin line which is something like 300mb down and 50mb up and he doesn't experience this issue either. The RB750 is older than my router I believe and he gets his full speed and no ping issues when uploading/downloading.

tippenring · Fri May 18, 2018 11:44 pm

Thanks for your reply

Can anyone tell me how I do that?

Its odd though - as I mentioned in my first post...this router came from my old house and nothing on the config changed when it was moved to the new house with a new line. I never had this issue before I moved house! I actually have a RB750G I have now put in at my old house (parents house) to replace the one I took away with me which is running just fine and doesn't experience this upload issue as I've tested it.

The only difference is the old house was pppoe with copper to the cabinet then fibre to the exchange with 30mb down and 5mb up. The new house is pppoe as well, with fibre straight from my house to the exchange which I'm only a few meters from.

It really depends what you want to do. In the screenshot of your speedtest, you are getting your full bandwidth. Do you want to reduce bandwidth consumption slightly so your ping RTTs are below a certain threshold? I wouldn't bother myself, but you can certainly prioritize ICMP traffic if you wish.

rd228 · Fri May 18, 2018 11:52 pm

Problem is that I have a server that backs up all my data and VM backups and then uploads it to the cloud for off site backups.

Because of the size of the backups its usually running constantly.

Because its running constantly my ping times are always really high and web pages lag when trying to open them due to the high ping times.

As I say before - This server was uploading constantly at my old house with the same router with no issues. Since moving its been a problem....apart from getting a new broadband line nothing else has changed. Even the new broadband line is with the same provider. Just this time its fibre to the house instead of copper.

RoadkillX · Sat May 19, 2018 1:13 am

Have you checked if the fasstrack rule is enabled and in-interface set to your pppoe connection on the rule?

rd228 · Sat May 19, 2018 1:19 am

Yep and it made absolutely no difference

RoadkillX · Sat May 19, 2018 1:36 am

Wait...made no difference even in cpu? same speeds and cpu usage with fasstrack and without? if so then the rule is not working correctly.

*I just ran a test on an RB750Gr3 fiber 300/300mbps connection with fasttrack full speed download 37MBps cpu is at 7-8% without fasstrack 22-25% i have no rules in mangle right now so no cpu use in marking connections or packets and it's bumping almost 20% cpu usage just to analyze the default firewall rules. So i'll get to the point RB750Gr3 has 4 cores 880mhz and runs at 25% cpu usage without fasttrack so the RB2011 with 600mhz single core should see significant less cpu usage with fasstrack.

rd228 · Sat May 19, 2018 9:34 am

Ok - it made a difference to CPU but no difference on my ping times or the speed I got.

Before fasttrack during a download my cpu was around 70% and 30-40% on an upload
Fasttrack enabled - download was 30% CPU and upload was between 20-30% CPU

rd228 · Sat May 19, 2018 9:56 am

I have run my ping tests again on my parents 750G router on their broadband line...

During a download the ping went to around 70ms and an upload went to around 80/90ms

So it is increasing a little on the 750G router which has a near identical setup to my 2011 but not anywhere close to the 500ms I am seeing from the 2011 router on an upload....so why am I getting poorer performance from a router that is meant to be of better spec....

If there is a better product from mikrotik that would improve upon this I would be happy to take a look at it with the intention to purchase. But I cant see why its so bad and cant be fixed with a little settings tweaks

mkx · Sat May 19, 2018 1:03 pm

My experience with broadband services (fixed and mobile) is that there are two ways of limited speed: technical and commercial.

Technical reason mostly involves lack of bandwidth on the last mile/yard/whatever and CPE/mobile terminal is fully aware of it. This can happen either with xDSL (sync rare is usually at maximum possible or at subscribed rate, whichever is lower) or with fibre (if subscribed rate is the same as SFP rate). Or with mobile bradband when signal level is not great for that matter. If CPE doesn't implement too large TX buffers, then mix of services (upload and ping in your case) actually work quite nicely.

Commercial reason means that last mile/yard/foot speed is actually quite larger than subscribed rate and bandwidth limitation happens somewhere in ISPs core network. In this case the traffic shaping device probably implements (too) large buffers and mix of services can behave in weird way. Mostly traffic shaping devices introduce additional delay to reduce tranfer rate and they do so to all packets regardless of their type (ICMP echo request included) counting on any kind of flow control (either intrinsic TCP flow control or in-application implementation in case of UDP). When that fails, they start to drop packets.
Many times, when this kind of speed limiting was in place, I could observe the following: when doing uplink (speedtest, ftp, ...) transfers, it woukd start really fast[*], but after a second or two speed would drop and stabilize at subscribed date. [*] Initial speed would mostly hit technical limit (if that one was up to a few times larger than subscribed). This kind of behaviour can be also set up on Mikrotik using simple queues.

I imagine that FTTP means that in your case rate limitation is commercial one and not technical one (any decent copper technology should allow speeds higher than 50/10Mbps on lines shorter than say 50 metres).

If ping delay really bothers you, then you might want to set up a queue (simple queue might do) on your RB to rate limit your VM backups to, say, 80% or 90% of your subscribed UL speed. This way most of time ISP's traffic shaper wouldn't touch your data packets including ICMP echo requests.

[Edit] Please bear in mind that in case of (independent) transfers in both directions, congestion in one direction almost always impacts the ohter direction as well. For example: when doing http or ftp download, a considerable number of packets (low throughput though) are sent in uplink, these are so called TCP ACK packets. If, due to UL congestion, they get delayed, then TCP flow control on sender's side slows down transmit rate. Hence rate limiting of your VM backups inside your RB can help to enhance normal DL usage and not just ping times.

rd228 · Sat May 19, 2018 1:27 pm

Hi all

Thanks for explaining all the reasons behind it but this is definitely not an issue with the ISP because if I use the BT home hub I get none of these issues.

The issue here definitely lies with the mikrotik.....

RoadkillX · Sat May 19, 2018 1:56 pm

Yeah I agree is something in the RB, have you tried connecting the ISP cable in another RB port? You probably have, but just checking, is hw-offload active on your bridge ports? RB2011 has 2 switch chips so bridge1 must have ports (ether1-ether5+sfp1) and bridge 2 (ether 6-10), is your Lan connected to a port on the same switch chip as the internet cable? And again check hw-offload. I think you use a pppoe so remove ether1 from bridge and use it for internet port and plug Ian in any other bridge1 port. Try the same with bridge2 on its ports.

rd228 · Sat May 19, 2018 3:16 pm

Can you tell me where hw-offload setting is located?

My cable config is as follows

WAN
pppoe --> Eth1 -->(CAT6) to BTs Fibre Modem (ONT)

LAN

SFP port on the mikrotik connects fibre to another SFP module on a Cisco SG300

Bridges -

Bridge 1 has the SFP port connected

Bridge 2 has ether7 and ether10 connected

However the only active ports are Ether1 for the Physical WAN connection to the BT ONT box and SFP port for my LAN connection. All other ports have been disabled as I don't use them

RoadkillX · Sat May 19, 2018 3:32 pm

Can you try doing the speedtest connected yo ether2, you can check if hw-offload is enabled using Winbox in bridge- port you'll see the H next to the port number if hwoffload is enabled. igmp snooping must be disabled on Bridge for hwoffload to work.

CZFan · Sun May 20, 2018 1:41 am

My 2c's,

This is not a hardware offload issue, but you should also not have 2 bridges, you will lose HW offload if you have 2 bridges.

Then somewhere it says your fasttrack have attribute in interface set to PPPoE? Remove that.

Then it seems your config file you placed here in earlier post is missing, can you run export hide-sensitive again, then obfuscate any sensitive items such as wan address with <wan ip> or som thing to tag effect, then paste here BUT between the code brackets, look at menu when posting for these brackets

RoadkillX · Sun May 20, 2018 1:46 am

if fasttrack is left without in-interface all traffic that meets the condition will be fastracked so firewall filter won't apply to vlans, there was a reason behind it...

CZFan · Sun May 20, 2018 1:57 am

Apologies, I did not see any mention of VLAN's in the posts, there are better way to do the fasttrack and I think we need to get to a base config where the 2011 performs as it should, then we can look at the bells and whistles, i.e. VLAN's, etc.
For now, think it is best we get the full config

RoadkillX · Sun May 20, 2018 2:00 am

Then somewhere it says your fasttrack have attribute in interface set to PPPoE? Remove that.

Just saying there was a reason not disagreeing with your suggestion.

rd228 · Sun May 20, 2018 11:29 am

Apologies, I did not see any mention of VLAN's in the posts, there are better way to do the fasttrack and I think we need to get to a base config where the 2011 performs as it should, then we can look at the bells and whistles, i.e. VLAN's, etc.
For now, think it is best we get the full config

Attached config

# may/20/2018 09:17:27 by RouterOS 6.42.2
# software id = Q9HX-XMDP
#
# model = 2011UiAS
# serial number = 608805BAE11A
/interface bridge
add admin-mac=******* auto-mac=no comment=\
    "created from master port" name=bridge1 protocol-mode=none
/interface ethernet
set [ find default-name=ether1 ] comment=WAN name=ether1-gateway
set [ find default-name=ether2 ] arp=proxy-arp name=ether2-master-local
set [ find default-name=ether3 ] disabled=yes name=ether3-slave-local
set [ find default-name=ether4 ] disabled=yes
set [ find default-name=ether5 ] disabled=yes
set [ find default-name=ether6 ] comment=DMZ disabled=yes
set [ find default-name=ether7 ] disabled=yes
set [ find default-name=ether8 ] disabled=yes
set [ find default-name=ether9 ] disabled=yes
set [ find default-name=ether10 ] disabled=yes
set [ find default-name=sfp1 ] comment=LAN
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1-gateway max-mru=1492 \
    max-mtu=1492 mrru=1600 name=pppoe-out1 user=bthomehub@btbroadband.com
/interface gre
add allow-fast-path=no comment="Site to Site VPN" !keepalive name=gre-tunnel1 \
    remote-address=******HIDDEN*****
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-256-cbc,3des pfs-group=none
/ip pool
add name=LAN ranges=172.28.6.100-172.28.6.254
add name=DMZ_Pool ranges=172.28.20.2-172.28.20.254
add name="LAB Pool" ranges=172.28.99.2-172.28.99.250
/ip dhcp-server
add address-pool=LAN authoritative=after-2sec-delay disabled=no interface=\
    bridge1 lease-time=1d name=LAN
add address-pool=DMZ_Pool authoritative=after-2sec-delay disabled=no name=\
    "DMZ DHCP"
/ppp profile
add local-address=172.28.8.1 name=PPTP remote-address=vpn-pool \
    use-encryption=yes
add change-tcp-mss=yes dns-server=172.28.8.187 local-address=172.28.9.1 name=\
    L2TP remote-address=vpn-pool
/system logging action
set 3 remote=172.28.8.108 remote-port=5544
/user group
add name=sniffer policy="ssh,read,!local,!telnet,!ftp,!reboot,!write,!policy,!\
    test,!winbox,!password,!web,!sniff,!sensitive,!api,!romon,!dude,!tikapp"
/interface bridge port
add bridge=bridge1 interface=sfp1
/ip neighbor discovery-settings
set discover-interface-list=all
/interface l2tp-server server
set authentication=mschap2 default-profile=L2TP enabled=yes
/interface pptp-server server
set authentication=pap,chap,mschap1,mschap2 default-profile=PPTP enabled=yes
/ip accounting
set account-local-traffic=yes enabled=yes threshold=2560
/ip accounting web-access
set accessible-via-web=yes
/ip address
add address=172.28.8.1/20 interface=bridge1 network=172.28.0.0
add address=172.28.7.1/24 interface=ether1-gateway network=172.28.7.0
add address=172.28.22.1/30 interface=gre-tunnel1 network=172.28.22.0
/ip dhcp-client
add dhcp-options=hostname,clientid interface=ether1-gateway
/ip dhcp-server network
add address=172.28.0.0/20 boot-file-name=undionly.kpxe comment="LAN Network" \
    dns-server=172.28.8.187 domain=****HIDDEN***** gateway=172.28.8.1 \
    netmask=20 next-server=172.28.8.252
add address=172.28.9.0/24 comment="VPN Network" dns-server=172.28.8.187 \
    domain=ad.havelockdrive.com gateway=172.28.9.1 netmask=16
add address=172.28.20.0/24 comment="DMZ Network" dns-server=8.8.8.8,8.8.4.4 \
    gateway=172.28.20.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=\
    established,related disabled=yes in-interface=pppoe-out1
add action=accept chain=forward connection-state=established,related \
    disabled=yes in-interface=pppoe-out1
add action=jump chain=forward comment="Jump To VPN Restricted Rules" \
    jump-target="VPN Restricted" src-address-list=FTPAllowedUsers
add action=jump chain=forward comment="Jump to VPN Rules" jump-target=VPN \
    src-address-list=VPN
add action=accept chain=forward comment="Allow Traffic from Tunnel to LAN" \
    in-interface=gre-tunnel1 out-interface=bridge1
add action=accept chain=forward comment=\
    "Allow new connections through router coming in LAN interface" \
    connection-state=new in-interface=bridge1
add action=accept chain=forward comment=\
    "Allow established connections through router" connection-state=\
    established
add action=accept chain=forward comment=\
    "Allow related connections through router" connection-state=related
add action=drop chain=forward comment="drop invalid connections" \
    connection-state=invalid
add action=jump chain=forward comment="Jump to DMZ" jump-target=DMZ
add action=jump chain=forward comment="Jump to LAB Network" jump-target=LAB
add action=jump chain=forward comment="jump to the virus chain" jump-target=\
    virus
add action=accept chain=forward comment="Allow BT Vision " disabled=yes \
    dst-port=5802 in-interface=ether1-gateway out-interface=bridge1 protocol=\
    udp
add action=accept chain=VPN protocol=icmp
add action=accept chain=VPN protocol=tcp
add action=accept chain=VPN protocol=udp
add action=accept chain=VPN comment="DNS Over VPN" dst-port=53 protocol=udp
add action=drop chain=input comment="GLOBAL DENY LIST" src-address-list=\
    "GLOBAL DENY"
add action=add-src-to-address-list address-list="Port Scanners DROP" \
    address-list-timeout=2w chain=input comment="Port scanners to list " \
    protocol=tcp psd=21,3s,3,1
add action=accept chain=IPTV in-interface=ether1-gateway protocol=igmp
add action=accept chain=IPTV dst-address-list=IPTV dst-port=5802 \
    in-interface=ether1-gateway protocol=udp
add action=jump chain=input comment="Jump to Port Scanner rules" jump-target=\
    "Port Scanners" src-address-list="Port Scanners DROP"
add action=accept chain=input comment=\
    "Allow everything from the LAN interface to the router" in-interface=\
    bridge1
add action=accept chain=input comment="Allow established  connections to the r\
    outer, these are OK because we aren't allowing new connections" \
    connection-state=established
add action=accept chain=input comment="Allow related connections to the router\
    , these are OK because we aren't allowing new connections" \
    connection-state=related
add action=jump chain=input comment="Allow Limited Ping" jump-target=Ping
add action=jump chain=input comment="Allow PPTP" jump-target=PPTP
add action=jump chain=input comment="jump to chain services" jump-target=\
    services
add action=accept chain=PPTP comment="Allow PPTP VPN" dst-port=1723 protocol=\
    tcp
add action=accept chain=PPTP dst-port=8291 in-interface=all-ppp protocol=tcp
add action=accept chain=PPTP comment="Allow limited pings" in-interface=\
    all-ppp limit=50/5s,2:packet protocol=icmp
# PPTP Ross not ready
add action=drop chain=PPTP comment="Drop excess pings" in-interface=\
    "PPTP Ross" protocol=icmp
add action=accept chain=PPTP protocol=gre
add action=drop chain=virus comment="Drop Blaster Worm" dst-port=135-139 \
    protocol=tcp
add action=drop chain=virus comment="Drop Messenger Worm" dst-port=135-139 \
    protocol=udp
add action=drop chain=virus comment="Drop Blaster Worm" dst-port=445 \
    protocol=tcp
add action=drop chain=virus comment="Drop Blaster Worm" dst-port=445 \
    protocol=udp
add action=drop chain=virus comment=________ dst-port=593 protocol=tcp
add action=drop chain=virus comment=________ dst-port=1024-1030 protocol=tcp
add action=drop chain=virus comment="Drop MyDoom" dst-port=1080 protocol=tcp
add action=drop chain=virus comment=________ dst-port=1214 protocol=tcp
add action=drop chain=virus comment="ndm requester" dst-port=1363 protocol=\
    tcp
add action=drop chain=virus comment="ndm server" dst-port=1364 protocol=tcp
add action=drop chain=virus comment="screen cast" dst-port=1368 protocol=tcp
add action=drop chain=virus comment=hromgrafx dst-port=1373 protocol=tcp
add action=drop chain=virus comment=cichlid dst-port=1377 protocol=tcp
add action=drop chain=virus comment=Worm dst-port=1433-1434 protocol=tcp
add action=drop chain=virus comment="Bagle Virus" dst-port=2745 protocol=tcp
add action=drop chain=virus comment="Drop Dumaru.Y" dst-port=2283 protocol=\
    tcp
add action=drop chain=virus comment="Drop Beagle" dst-port=2535 protocol=tcp
add action=drop chain=virus comment="Drop Beagle.C-K" dst-port=2745 protocol=\
    tcp
add action=drop chain=virus comment="Drop MyDoom" dst-port=3127-3128 \
    protocol=tcp
add action=drop chain=virus comment="Drop Backdoor OptixPro" dst-port=3410 \
    protocol=tcp
add action=drop chain=virus comment=Worm dst-port=4444 protocol=tcp
add action=drop chain=virus comment=Worm dst-port=4444 protocol=udp
add action=drop chain=virus comment="Drop Sasser" dst-port=5554 protocol=tcp
add action=drop chain=virus comment="Drop Beagle.B" dst-port=8866 protocol=\
    tcp
add action=drop chain=virus comment="Drop Dabber.A-B" dst-port=9898 protocol=\
    tcp
add action=drop chain=virus comment="Drop Dumaru.Y" dst-port=10000 protocol=\
    tcp
add action=drop chain=virus comment="Drop MyDoom.B" dst-port=10080 protocol=\
    tcp
add action=drop chain=virus comment="Drop NetBus" dst-port=12345 protocol=tcp
add action=drop chain=virus comment="Drop Kuang2" dst-port=17300 protocol=tcp
add action=drop chain=virus comment="Drop SubSeven" dst-port=27374 protocol=\
    tcp
add action=drop chain=virus comment="Drop PhatBot, Agobot, Gaobot" dst-port=\
    65506 protocol=tcp
add action=accept chain=services comment="Allow L2TP" dst-port=\
    500,1701,4500,137 protocol=udp
add action=accept chain=services comment="accept localhost" dst-address=\
    127.0.0.1 src-address=127.0.0.1
add action=accept chain=services comment="allow MACwinbox " dst-port=20561 \
    protocol=udp
add action=accept chain=services comment="Bandwidth server" dst-port=2000 \
    protocol=tcp
add action=accept chain=services comment=" MT Discovery Protocol" dst-port=\
    5678 protocol=udp
add action=accept chain=services comment="allow SNMP" dst-port=161 protocol=\
    tcp
add action=accept chain=services comment="Allow BGP" dst-port=179 protocol=\
    tcp
add action=accept chain=services comment="allow BGP" dst-port=5000-5100 \
    protocol=udp
add action=accept chain=services comment="Allow NTP" dst-port=123 protocol=\
    udp
add action=accept chain=services comment="Allow PPTP" dst-port=1723 protocol=\
    tcp
add action=accept chain=services comment="allow PPTP and EoIP" protocol=gre
add action=accept chain=services comment="Allow DNS request" dst-port=53 \
    protocol=udp
add action=accept chain=services comment="allow DNS request" dst-port=53 \
    protocol=tcp
add action=accept chain=services comment=UPnP dst-port=1900 protocol=udp
add action=accept chain=services comment=UPnP dst-port=2828 protocol=tcp
add action=accept chain=services comment="allow DHCP" dst-port=67-68 \
    protocol=udp
add action=accept chain=services comment="allow Web Proxy" dst-port=8080 \
    protocol=tcp
add action=accept chain=services comment="allow IPIP" protocol=ipencap
add action=accept chain=services comment="allow https for Hotspot" dst-port=\
    443 protocol=tcp
add action=accept chain=services comment="allow Socks for Hotspot" dst-port=\
    1080 protocol=tcp
add action=accept chain=services comment="allow IPSec connections" dst-port=\
    500 protocol=udp
add action=accept chain=services comment="allow IPSec" protocol=ipsec-esp
add action=accept chain=services comment="allow IPSec" protocol=ipsec-ah
add action=accept chain=services comment="allow RIP" dst-port=520-521 \
    protocol=udp
add action=accept chain=services comment="allow OSPF" protocol=ospf
add action=accept chain=Ping comment="Allow limited pings" limit=\
    50/5s,2:packet protocol=icmp
add action=drop chain=Ping comment="Drop excess pings" protocol=icmp
add action=accept chain=VPN comment=HTTPS dst-port=443 protocol=tcp
# no interface
add action=accept chain=DMZ comment="Allow DMZ Clients out to Internet (WAN)" \
    in-interface=*7 out-interface=pppoe-out1
# no interface
add action=accept chain=DMZ comment=\
    "Allow port 80 from WAN to DMZ Network only!" dst-port=80 in-interface=\
    pppoe-out1 out-interface=*7 protocol=tcp
# no interface
add action=accept chain=DMZ dst-port=443 in-interface=pppoe-out1 \
    out-interface=*7 protocol=tcp
# no interface
add action=accept chain=DMZ dst-port=1022 in-interface=pppoe-out1 \
    out-interface=*7 protocol=tcp
add action=accept chain="Port Scanners" comment=\
    "IP addresses ALLOWED to scan Router" src-address-list="Port Scanners OK"
add action=drop chain="Port Scanners" comment="dropping port scanners" \
    src-address-list="Port Scanners DROP"
add action=accept chain=forward comment="Allow Plex" dst-port=32400 protocol=\
    tcp
add action=accept chain=forward comment="Allow Home Assistant" dst-port=8123 \
    protocol=tcp
add action=accept chain=forward disabled=yes dst-port=80 protocol=tcp
add action=accept chain=forward disabled=yes dst-port=443 protocol=tcp
add action=drop chain=forward comment=\
    "Drop all other connections through the router"
add action=accept chain=LAB comment="Allow LAB Clients out to Internet (WAN)" \
    in-interface=ether10 out-interface=pppoe-out1
# no interface
add action=drop chain=DMZ comment="Disable DMZ talking to LAN" in-interface=\
    *7 out-interface=bridge1
add action=drop chain=LAB comment="Disable LAB talking to LAN" in-interface=\
    ether10
add action=accept chain="VPN Restricted" dst-address=172.28.8.82 \
    out-interface=bridge1 protocol=tcp src-address-list=FTPAllowedUsers
add action=drop chain=input comment="Drop everything else to the router"
/ip firewall nat
add action=masquerade chain=srcnat comment="PUBLIC/PRIVATE NAT (Internet)" \
    out-interface=pppoe-out1
/ip ipsec peer
add address=0.0.0.0/0 dh-group=modp1024 enc-algorithm=3des generate-policy=\
    port-override passive=yes
/ip route
add check-gateway=ping distance=1 dst-address=172.28.16.0/24 gateway=\
    172.28.22.2
/ip service
set telnet disabled=yes
set ftp disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ip traffic-flow
set active-flow-timeout=5m enabled=yes interfaces=pppoe-out1
/lcd
set backlight-timeout=never default-screen=stats read-only-mode=yes
/lcd interface
add interface=bridge1
add
add interface=pppoe-out1
/ppp aaa
set use-radius=yes
/ppp secret
add name=Ross profile=PPTP
/radius
add address=172.28.8.253 timeout=2s
/routing igmp-proxy
set quick-leave=yes
/system clock
set time-zone-name=Europe/London
/system identity
set name=008-HOME
/system logging
set 3 action=memory
add disabled=yes topics=radius
add disabled=yes topics=ppp
add disabled=yes topics=dhcp
add disabled=yes topics=pppoe
/system ntp client
set enabled=yes primary-ntp=***** secondary-ntp=*****
/system routerboard settings
set silent-boot=no

CZFan · Sun May 20, 2018 4:21 pm

Had a quick look at your config, and my personal view is your firewall rules does not make sense to me and are a mess. Anyone free to correct me.
You are jumping all over in the firewall filter table, but then you are also accepting related / established, etc before this so not sure what traffic will hit this if any.
My suggestion will be to delete all firewall rules, I will actually go as far and say reset to factory default the device. You do have a backup from what you have pasted in post above.
Then use only the following firewall rules as a starting point(Amend according to your environment), test and take from there:

/ip firewall filter
add action=fasttrack-connection chain=forward comment="Fasttrack Established, Related connections" connection-state=established,related
add action=accept chain=forward comment="Allow Related, Established" connection-state=established,related
add action=accept chain=forward comment="Allow new from LAN" connection-state=new in-interface=Bridge1
add action=accept chain=forward comment="Allow DST NATed" connection-nat-state=dstnat connection-state=new in-interface=pppoe1
add action=drop chain=forward comment="Drop Invalid" connection-state=invalid
add action=drop chain=forward comment="Default Drop"
add action=accept chain=input comment="Allow Related, Established from LAN" connection-state=established,related
add action=accept chain=input comment="Allow new from LAN" connection-state=new in-interface=Bridge1
add action=drop chain=input comment="Drop Invalid" connection-state=invalid
add action=drop chain=input comment="Default Drop"

rd228 · Sun May 20, 2018 5:06 pm

I have disabled all of my firewall rules and added your ones in place

Re ran my tests and its still the same problem....

CZFan · Sun May 20, 2018 5:36 pm

Where is PC connected that you test from, behind the Cisco switch? Have you tried as @RoadkillX suggested, test from ether2 on RB2011 to eliminate internal LAN problems?

tippenring · Sun May 20, 2018 5:49 pm

If ping delay really bothers you, then you might want to set up a queue (simple queue might do) on your RB to rate limit your VM backups to, say, 80% or 90% of your subscribed UL speed. This way most of time ISP's traffic shaper wouldn't touch your data packets including ICMP echo requests.

This would probably be what I would do. By limiting the heavy traffic to an acceptable rate, you reserve some bandwidth for interactive traffic like ping, surfing the web, etc.

rd228 · Sun May 20, 2018 5:59 pm

Whilst we are on the subject of Firewall rules - I have now tidied them up since posting my config and would like your input on them as its currently being discussed above.

I thought it might be easier to post a screenshot of the order and put an explanation as to why I have put them there and you can tell me if what I am doing is wrong? Or a better practice way of doing it? And if my ordering is completely screwed.

Input Rules

Screen Shot 2018-05-20 at 15.41.32.png

Global Deny List - I have set this up so If I have a particular IP I want to block I can add it to the address list ad-hoc and it will drop it
Port scanners to list - This rule is setup to sense if a port scanner is scanning my public IP and will add them to an address list
Jump to Pot scanner rules - This rule is setup using the Port scanners address list above. If an IP matches on that list then it will jump to the rules I have in place to drop that traffic in the Port Scanners chain.

Allow everything from the LAN interface to the router - no explanation needed here
Allow Established connection to the router - to be honest I'm not sure why this one is here (or if it should be?). I think I used a default config from the mikrotik wiki when I was new to this system and trusted that this was correct.
Allow Related connection to the router - Same reason as above

Allow Limited Ping - This one I got from there forum somewhere I think. This is setup to allow a limited ammount of ping. I guess to detect is someone is trying to DDOS the router.
Jump to services - I set this one up to put all of my services that I allow to the router such as my VPN ports and any other service ports I allow to the router. I thought putting it in a jump chain would keep the Input chain looking neat rather than loads of rules filling it up.

Forward Chain

Screen Shot 2018-05-20 at 15.51.06.png

Jump to VPN Chain - When a new device connects to my L2TP VPN their address is added to a dynamic Address list for the duration of their connection. This jump rule basically allows anyone on the VPN address list to hit this jump rule to get to my VPN chain which then has rules setup to allow access to my LAN.

Allow Traffic from Tunnel to LAN - I have a site to site VPN setup also from my house to my parents house so that services on each LAN can be shared. This rule allows anything on the GRE tunnel to access my LAN. Same rule is in place on the router at the other end.

Allow New connection from LAN to router - self explanatory, allows the LAN though the router
Allow established & Related connections - again taken from the default setup on I found on the mikrotik wiki page. Im not too sure what these rules do?
Drop invalid - again taken from the default setup on I found on the mikrotik wiki page. Im not too sure how this rule works and whats meant by an invalid connection?
Allow Plex - allows external access to my plex server
Allow home assistant - allows external access to my home automation server
Drop all other connections - I assume this rule basically drops any connection that doesn't match the above rules

rd228 · Sun May 20, 2018 6:01 pm

Where is PC connected that you test from, behind the Cisco switch? Have you tried as @RoadkillX suggested, test from ether2 on RB2011 to eliminate internal LAN problems?

I have tried cabled into the cisco switch. But yes I have also hard wired my PC into ether2 on the mikrotik and removed my lan from the router so its literally just my laptop connected and thats it when running the test...same issue.

CZFan · Sun May 20, 2018 6:14 pm

Below is my results over fibre, my PC connects to router (2011) via wireless, so yes, maybe your upload fully saturated and maybe need some QoS.

Pinging www.google.co.za [216.58.223.35] with 32 bytes of data:
Reply from 216.58.223.35: bytes=32 time=4ms TTL=59
Reply from 216.58.223.35: bytes=32 time=10ms TTL=59
Reply from 216.58.223.35: bytes=32 time=4ms TTL=59
Reply from 216.58.223.35: bytes=32 time=363ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=7ms TTL=59
Reply from 216.58.223.35: bytes=32 time=8ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=57ms TTL=59
Reply from 216.58.223.35: bytes=32 time=67ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=8ms TTL=59
Reply from 216.58.223.35: bytes=32 time=7ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=4ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=27ms TTL=59

Download test starts

Reply from 216.58.223.35: bytes=32 time=130ms TTL=59
Reply from 216.58.223.35: bytes=32 time=136ms TTL=59
Reply from 216.58.223.35: bytes=32 time=214ms TTL=59
Reply from 216.58.223.35: bytes=32 time=171ms TTL=59
Reply from 216.58.223.35: bytes=32 time=158ms TTL=59
Reply from 216.58.223.35: bytes=32 time=183ms TTL=59
Reply from 216.58.223.35: bytes=32 time=186ms TTL=59
Reply from 216.58.223.35: bytes=32 time=183ms TTL=59
Reply from 216.58.223.35: bytes=32 time=118ms TTL=59
Reply from 216.58.223.35: bytes=32 time=126ms TTL=59
Reply from 216.58.223.35: bytes=32 time=155ms TTL=59
Reply from 216.58.223.35: bytes=32 time=130ms TTL=59
Reply from 216.58.223.35: bytes=32 time=127ms TTL=59
Reply from 216.58.223.35: bytes=32 time=140ms TTL=59
Reply from 216.58.223.35: bytes=32 time=40ms TTL=59
Reply from 216.58.223.35: bytes=32 time=21ms TTL=59
Reply from 216.58.223.35: bytes=32 time=29ms TTL=59

Upload test

Reply from 216.58.223.35: bytes=32 time=162ms TTL=59
Reply from 216.58.223.35: bytes=32 time=152ms TTL=59
Reply from 216.58.223.35: bytes=32 time=429ms TTL=59
Reply from 216.58.223.35: bytes=32 time=564ms TTL=59
Reply from 216.58.223.35: bytes=32 time=188ms TTL=59
Reply from 216.58.223.35: bytes=32 time=402ms TTL=59
Reply from 216.58.223.35: bytes=32 time=629ms TTL=59
Reply from 216.58.223.35: bytes=32 time=205ms TTL=59
Reply from 216.58.223.35: bytes=32 time=159ms TTL=59
Reply from 216.58.223.35: bytes=32 time=585ms TTL=59
Reply from 216.58.223.35: bytes=32 time=375ms TTL=59
Reply from 216.58.223.35: bytes=32 time=228ms TTL=59
Reply from 216.58.223.35: bytes=32 time=97ms TTL=59
Reply from 216.58.223.35: bytes=32 time=770ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=3ms TTL=59
Reply from 216.58.223.35: bytes=32 time=18ms TTL=59
Reply from 216.58.223.35: bytes=32 time=4ms TTL=59

Ping statistics for 216.58.223.35:
    Packets: Sent = 64, Received = 64, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 3ms, Maximum = 770ms, Average = 122ms

rd228 · Sun May 20, 2018 7:13 pm

Ok so it looks like we both have the same issue then by the looks of your results?

I could setup if this is possible a QOS to limit during hours I will be using the internet and then over night or weekdays when I'm at work I can set a time shedule so that it can run full speed?

Did you see my firewall post at the bottom of page 1? Just wondering if you'd mind taking a look over what I have and if its appropriate...

CZFan · Sun May 20, 2018 7:40 pm

I will have a look through it, but not tonight. Maybe someone else will fill in before then

RoadkillX · Sun May 20, 2018 8:03 pm

I have 42ms latency to google when link is idle while at fullspeed either download or upload it increased to 45-47ms you guys should check those fiber connections

rd228 · Sun May 20, 2018 8:16 pm

My latency at idle is fine - its around 6ms or less

Download is at around 40 and upload is where the problem is at over 400ms

Nothing wrong with the fibre connection as its fine as I've mentioned above if I use the BT Supplied Router. 6ms at idle around 40 download and 6ms for an upload!

RoadkillX · Sun May 20, 2018 8:21 pm

Check for DSCP markings on the BT router.

Squire · Mon May 21, 2018 2:58 am

http://www.dslreports.com/speedtest
post results, if possible do one for each device, tik and other the isp router

rd228 · Mon May 21, 2018 9:52 pm

http://www.dslreports.com/speedtest
post results, if possible do one for each device, tik and other the isp router

Here is the result with the Mikrotik -http://www.dslreports.com/speedtest/33662838

Will get the router swapped over tomorrow and run the test again with the BT Hub as I cant take the line down this evening as I'm transferring stuff.

Ross

Squire · Tue May 22, 2018 12:28 am

The buffer bloat is pretty bad for a mikrotik, try do a basic setup without any special rules and try again the cpu might dropping packet processing through firewall..etc

My results for comparison: https://www.dslreports.com/speedtest/33630812
Even though my latency is high because of distance, no local servers, My packets are getting through without/very few Re-Xmits (re-transmitted) packets on good days when isp isn't being bad its A+ for both overall and bufferbloat, had been using a hAP lite but its cheapest mirkrotik you can get, I've recently upgraded to a 962UiGS-5HacT2HnT (hAP AC) still similar/same results.

When running the tests again start up the profile tool on Winbox, Tools -> Profile, select the "all" under cpu and start then use do a couple tests while monitoring that profile post a screen of a couple tests here is mine for comparison under load: as an example

For reference sake I'm on a 50/5mb fibre connection with removable SFP module in my ONT/CPE

CZFan · Tue May 22, 2018 1:17 am

...
Did you see my firewall post at the bottom of page 1? Just wondering if you'd mind taking a look over what I have and if its appropriate...

Firstly, it is very difficult to give pointers on firewall rules if you do not have the full picture of the network, also, from a screenshot as it does not contain all information, i.e. you might be using address-list instead of in interface and that will not show on the screenshot.

With that said, here are a couple of points:
1. make sure rules 0 & 1 are "connection-state=Related, establish" and enable them.
2. Delete rules 5 & 6 as they are taken care of in above.
3. The screenshot is only showing some of the rules, i.e. 8 - 61 is missing, so from what I can see in the screenshot, something that needs to be dropped by default drop rule in forward chain (rule 66) must go through a huge stack of rules before getting dropped, wasting unnecessary CPU / Memory resources on the router.
4. I also think your firewall rules are way to complicated and can be simplified a lot better

rd228 · Wed May 23, 2018 8:39 pm

...
Did you see my firewall post at the bottom of page 1? Just wondering if you'd mind taking a look over what I have and if its appropriate...

Firstly, it is very difficult to give pointers on firewall rules if you do not have the full picture of the network, also, from a screenshot as it does not contain all information, i.e. you might be using address-list instead of in interface and that will not show on the screenshot.

With that said, here are a couple of points:
1. make sure rules 0 & 1 are "connection-state=Related, establish" and enable them.
2. Delete rules 5 & 6 as they are taken care of in above.
3. The screenshot is only showing some of the rules, i.e. 8 - 61 is missing, so from what I can see in the screenshot, something that needs to be dropped by default drop rule in forward chain (rule 66) must go through a huge stack of rules before getting dropped, wasting unnecessary CPU / Memory resources on the router.
4. I also think your firewall rules are way to complicated and can be simplified a lot better

Hi CZFan

Thanks for looking into it, I did write an explanation of each rule under the screenshot to explain what I was trying to achieve and i listed where I was using address lists in those explanations. Did they not help at all?

Ross

CZFan · Wed May 23, 2018 8:59 pm

E.g.:

"Allow Plex - allows external access to my plex server
Allow home assistant - allows external access to my home automation server"

Based on the screenshot, you can't see if this is from WAN1, WAN2, interface-list, specific IP Address(es) specified in address-list, etc so it makes it a bit difficult to follow the flow of traffic in some situations.

Besides the above, there are many other parameters not showing in screenshots