Ok Managed to solve this by adapting the second rule set in the Mikrotik Wiki.
On each IPSEC connection 2 packets are seen on port 500 so I've made use of the Nth rule to take this into account.
Also removed connection-state=new as this prevented seeing any packets after the first attempt.
These rules go before anything else on the input chain.
/ip firewall filter
add action=drop chain=input comment="drop VPN brute forcers" dst-port=500 \
protocol=udp src-address-list=VPN_blacklist
add action=add-src-to-address-list address-list=VPN_blacklist \
address-list-timeout=4w2d chain=input connection-state="" dst-port=500 \
nth=2,2 protocol=udp src-address-list=VPN_stage3
add action=add-src-to-address-list address-list=VPN_stage3 \
address-list-timeout=1m chain=input connection-state="" dst-port=500 nth=\
2,2 protocol=udp src-address-list=VPN_stage2
add action=add-src-to-address-list address-list=VPN_stage2 \
address-list-timeout=1m chain=input connection-state="" dst-port=500 nth=\
2,2 protocol=udp src-address-list=VPN_stage1
add action=add-src-to-address-list address-list=VPN_stage1 \
address-list-timeout=1m chain=input connection-state="" dst-port=500 nth=\
2,2 protocol=udp