Do you do direct IPsec tunnel? It behaves a bit funny, due to the way it integrates with the network. This is quite usual in IPsec implementations.
When you don't want that, use a tunnel interface (IP Tunnel, GRE Tunnel) with IPsec protection configured.
Then you can have the usual firewall rules on the tunnel interface.
(it is possible with direct IPsec tunnel as well, but rather complicated)