Could it be that you have masquerade somewhere without defining in\out interfaces (ip addresses)? That might cause interesting effects.
It should not, because the chains of the NAT table are only consulted for the initial packet of each connection (connection-state=new). So even if a masquerade rule made the server see the request as coming from the router's LAN IP instead of the client's IP, the server would respond to the router's address and connection tracking would "un-src-nat" the destination, same like it "un-dst-nat's" the source if dst-nat was applied on the connection.
But it's like in the joke about the fallen concrete bridge where cement says "don't blame me, I wasn't there at all" - packet 873118 goes from port 11034
to port 16666, but packet 873120 goes to port 11034 but
from port 63268. So the connection tracking cannot recognize that the two packets belong to the same connection (if they actually do which is questionable), and thus anything that relies on connection tracking, i.e. both NAT and connection marking, doesn't work.
So @ivugrinec, the first thing is to find out whether the packets actually do belong to the same connection at application level, and if they do, whether there is a way to make the server respond from the port on which it listens. If they do belong to the same connection but you cannot make the server respond from the same port, you'll need to record the address of the client to an address list logically attached to the IP address to which the request comes, and use a src-nat rule to set the right IP address to the first packet of the server->client flow which will be treated as a separate connection. As none of the connections will be "confirmed" (because the connection tracking will see only packets in one direction), shorter timeouts will apply.
And if several client connections may come from the same address, it is even more funny because the address-list-timeout will have to be long enough so that the client address would still be available in the list when the first response packet comes, but short enough that if another connection from the same remote address comes to a different local address, it would already be out of the previous list. If this is the case, enjoy the tuning process

.