We discovered however that reducing the MTU on the ethernet interface on one of the computers in the office to 1400 appears to solve the problem and both SSH and HTTP work fine.
I am presuming that changing to 1400 lowers the MTU to the lowest in the path and messages are being correctly passed to web servers (SYN?) on AWS to make that work... but at 1500 there is something not working in the path as the web servers are not ever "hearing" that 1500 bytes is too big
From this it appears that PMTU is not working and devices on AWS are not reducing packet sizes correctly.
It is a relatively simple setup
Office connected to internet via PPPoE DSL (RB4011)
CHR on AWS connecting directly to AWS internet connection and creating an IPsec tunnel back to the office (ESP)
I've used ping with DF set to ascertain that the largest ping between the office and our AWS vpc is 1378. Add 28 bytes for IP and ICMP should make the MTU 1406. This traffic goes over the ipsec vpn
Pinging from the same host in the office to 8.8.8.8 gives a 1452 ping + 28 = 1480 MTU which is consistent with the MTU setting on the office router (4011) connected via ADSL modems to our ISP (Plusnet in the UK). Internet traffic works fine.
I get the same maximum ping size the other way (from AWS vpc to the office)
I'm not clear on what to configure to fix this ... despite much reaching on MTU. MSS, clamping, SYN etc. (as well as lots of posts about VPN's causing problems like this)
Lowering the MTU on the Ethernet to the modem or on the CHR does not seem right as it is working fine for all our web traffic in what is pretty much a standard configuration.
It would seem that focusing on the the IPsec tunnel and ensuring that both ends know that MTU/MSS would be sensible but I cannot find any setting in the actual tunnel config.
That seems to leave using a mangle ? But how?
I keep seeing something like this mentioned... but what PMTU size is it clamping to? Will it reduce mss on traffic that is already working fine to the internet rather than over the tunnel? on some of the traffic will go to AWS. Most just goes to the internet directly
/ip firewall mangle
add action=change-mss chain=forward new-mss=clamp-to-pmtu passthrough=yes \
protocol=tcp tcp-flags=sy