Would RouterOS match a packet to the rule "tcp-flags=syn" if some other flags are set as well? Is using this rule an effective proof against syn floods? If so then is there a guide out there for figuring out how to set the optimum limit?

On regular linux, enabling "syn cookies" can help a little in trying to keep the server alive during a dos attack. Usually a dos attack eats up all your available bandwidth. No software setting can fix that.