What is your profile on the Fortigate? Is DPD enabled? It's possible the Fgate thinks the peer is dead so will no longer accept packets for that SA. Also, check your settings on the SA, lifetime, keepalives, etc...
JP
I enabled DPD and it did not help, tunnel still dropping. Below is my log file part:
15:18:01 ipsec ignore INITIAL-CONTACT notification, because it is only accepted after phase1.
15:18:01 ipsec ISAKMP-SA established 196.xxx.xx.73[500]-196.xx.xx.5[500] spi:b4c77b194563b6f4:a422197a08f3f312
15:18:01 ipsec respond new phase 1 negotiation: 196.xxx.xx.73[500]<=>196.xx.xx.5[500]
15:18:01 ipsec begin Identity Protection mode.
15:18:01 ipsec received Vendor ID: RFC 3947
15:18:01 ipsec received Vendor ID: draft-ietf-ipsec-nat-t-ike-03
15:18:01 ipsec received Vendor ID: draft-ietf-ipsec-nat-t-ike-02
15:18:01 ipsec received Vendor ID: draft-ietf-ipsec-nat-t-ike-02
15:18:01 ipsec
15:18:01 ipsec received Vendor ID: draft-ietf-ipsec-nat-t-ike-01
15:18:01 ipsec received Vendor ID: draft-ietf-ipsec-nat-t-ike-00
15:18:01 ipsec received Vendor ID: DPD
15:18:02 ipsec respond new phase 2 negotiation: 196.xxx.xx.73[500]<=>196.xx.xx.5[500]
15:18:02 ipsec respond new phase 2 negotiation: 196.xxx.xx.73[500]<=>196.xx.xx.5[500]
15:18:06 ipsec the packet is retransmitted by 196.xx.xx.5[500].
15:18:06 ipsec fatal INVALID-SPI notify messsage, phase1 should be deleted.
15:18:06 ipsec pfkey UPDATE failed: No such process
15:18:06 ipsec ISAKMP-SA established 196.xxx.xx.73[500]-196.xx.xx.5[500] spi:e4b77bc8f58dfa85:a00c74efa7ff0a23
15:18:06 ipsec the packet is retransmitted by 196.xx.xx.5[500].
15:18:06 ipsec respond new phase 2 negotiation: 196.xxx.xx.73[500]<=>196.xx.xx.5[500]
15:18:06 ipsec invalid length of payload
15:18:06 ipsec failed to pre-process packet.
15:18:41 ipsec initiate new phase 2 negotiation: 196.xxx.xx.73[500]<=>196.xx.xx.5[500]
15:18:42 ipsec ignore RESPONDER-LIFETIME notification.
15:18:43 ipsec IPsec-SA established: ESP/Tunnel 196.xx.xx.5[0]->196.xxx.xx.73[0] spi=127512406(0x799af56)
15:19:31 ipsec initiate new phase 2 negotiation: 196.xxx.xx.73[500]<=>196.xx.xx.5[500]
15:19:32 ipsec ignore RESPONDER-LIFETIME notification.
15:19:33 ipsec IPsec-SA established: ESP/Tunnel 196.xx.xx.5[0]->196.xxx.xx.73[0] spi=97676458(0x5d26caa)