I want to setup 2 VPN's that use 2 different ISP GW's on my side, but only one ISP GW on the remote side. Like so:
Where XXX.YYY.ZZZ.130 (#1) is our main connection (ISP) default route, so all traffic would go through this by default.

How could I ensure that traffic is routed through the secondary ISP (XXX.YYY.ZZZ.106)?
I am thinking along the lines of marking/mangling outgoing traffic to 10.3.7.0/24 with a routing-mark, and in Routes have an entry that routes traffic to the secondary ISP GW if it finds that routing-mark?

On the other hand, If the VPN was configured in such a way, that only the remote side would initiate the VPN, would I even need these routing rules?

Okay, let me formulate this differently.
I have two ISP's connected to the router. The main ISP is the default route.

Now I want to setup an IPsec VPN, and I want its traffic to be routed through the secondary ISP route.
This could be easily achieved by adding the following rule (since the remote's GW address is YYY.YYY.YYY.161): Let's assume I wouldn't be allowed to use this rule based on the dst-address (because there is other/ordinary traffic going to that same address, that should go through the default route / main ISP). I only want the traffic of that specific VPN to go through this secondary WAN. What other options would I have? I am thinking along the line of using "connection marks" and using that to re-route the VPN traffic?

It's still too vague, but if your router acts as a VPN server, i.e. external clients to connect to one or the other of its public addresses, a routing rule matching on src-address is sufficient to make the router respond via the second WAN.

For this use case, it only makes sense to use connection marks if you forward the incoming connections further to servers in LAN and you need that the responses from the same server in LAN were sent back via the same WAN through which the corresponding requests came in.

The VPN is a site-to-site IPsec VPN.
However, instead of the VPN tunnel being routed through the default route (main ISP), I want the traffic for this particular VPN to go through the secondary ISP (secondary default route).
What options (other than the above mentioned /ip route) do I have?

You mentioned a "routing rule matching on src-address", could you be more specific? Perhaps formulate such a rule?
With src-address you mean the local (phase 2) address, or the "SA src-address"?

I could add a default route for the secondary ISP with a routing-mark named "SECONDARY-ISP", and then add a routing-rule (based on src-address), that does a "lookup in table on"="SECONDARY-ISP".
I experimented with that, also with firewall mangle rules (adding routing mark or connection mark) - but to no avail.

I can setup a firewall mangle rule (prerouting) with src-address=my.local.ip and set a routing-mark - I can see packets being processed, but I assume, since these packets get wrapped into the "VPN" packets, the routing-mark is buried in there, so that the routing-rule has no chance to act upon it...?

Basic idea:

Great, that works!
I have always left local-address in peer empty, but I guess it should be always set for cases where a fixed local-address is used in a site-to-site VPN config?

Thank you @sob and @sindy for your help!

You need it when there's a risk that router chooses wrong one. If there's single uplink, router will take whatever is on that interface, it can't go wrong. If there are multiple, it by default uses what's on interface with default route, and it may be not what you want.

I have always left local-address in peer empty, but I guess it should be always set for cases where a fixed local-address is used in a site-to-site VPN config?

That routing rule would be sufficient alone if the router was a passive peer (only ever acting as a responder). If it also acts as an initiator, you need to specify the local-address so that it would choose that address also for the outgoing connections, as otherwise it chooses the route first and only then it assigns a source address to the packet depending on the outgoing interface of that route.

But this means that if you use the two peers for redundancy purposes, you have to make sure that both of them stick to their respective WAN IPs regardless the state of that WAN, i.e. you should specify the respective local-address for both, and you should use two routing rules:

src-address=<WAN IP from first ISP> dst-address=<IP of the remote peer> action=lookup-only-in-table table=FIRST-ISP
src-address=<WAN IP from second ISP> dst-address=<IP of the remote peer> action=lookup-only-in-table table=SECOND-ISP

Thank you @sindy, will do that!
Thanks again @sob!