is there any difference in setting !fin,!syn,!rst,ack for a mangle rule and setting just ack?
are there any performance issues?

there should be difference, since there is possibility to catch ack packet and syn,ack.