Multiple EoIP Tunnels On Single WAN But With Multiple IPs

CSR460 · Tue Jan 26, 2021 4:55 am

So I have a running CHR instance and the cloud provider gave me 4 public IPv4 with different gateways.
However, these IPs are all on a "physical" ethernet port. I have 4 public IPv4 over the same interface (ether2).

I need to create 4 different EoIP tunnels to the same address, but I would like to make these 4 tunnels go to different gateways. Because each of these gateways has its own bandwidth limit. After configuration, I found that all 4 tunnels are actually going to the same gateway (The activate route).

Here is the structure I would like to achieve:

_20210125215234.png

But here is the structure it current is:

20210125215426.png

Here are my configurations:

[admin@MikroTik] /interface> print
Flags: D - dynamic, X - disabled, R - running, S - slave 
 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU
 0  R  ether1                              ether            1500
 1  R  ether2                              ether            1500
 2  X  Bonding-Amherst                     bond             1500 65535
 3  R  EoIP-Amherst1                       eoip             1458 65535
 4  R  EoIP-Amherst2                       eoip             1458 65535
 5  R  EoIP-Amherst3                       eoip             1458 65535
 6  R  EoIP-Amherst4                       eoip             1458 65535
 7  X  vlan1                               vlan      
 8  R  vlan2                               vlan             1500
 9  R  vlan3                               vlan             1500
10  R  vlan4                               vlan             1500

[admin@MikroTik] /ip address> print
Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         INTERFACE                                
 0   39.108.236.102/20  39.108.224.0    ether2                                   
 1   119.23.243.124/20  119.23.240.0    vlan3                                    
 2   120.79.186.223/19  120.79.160.0    vlan4                                    
 3   47.106.64.237/19   47.106.64.0     vlan2                                    
 4 D 172.16.0.38/24     172.16.0.0      ether1 
 
 [admin@MikroTik] /ip route> print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0   S  0.0.0.0/0                          120.79.191.253%...        3
 1 ADS  0.0.0.0/0                          172.16.0.253              1
 2   S  0.0.0.0/0                          39.108.239.247            2
 3   S  0.0.0.0/0                          119.23.255.247%...        3
 4   S  0.0.0.0/0                          47.106.95.253%v...        3

Note that I use "%" to make sure the connection goes through "vlanX" rather than directly ether2. However, VLANs do not work as a real interface. All the traffics is going through the default gateway (172.16.0.253), and since the target addresses are the same for all tunnels, I can't use the target address to route them. And I found that I can't use "prerouting" chain to mark it either. The only chain which can capture these EoIP(actually IP-gre(47) ) packets is "output" chain, there is nothing I can do.

I found that the EoIP IP packet header contains tunnel ID, but I don't know how to route the packet based on specific bytes on the header.

Any suggestions would be appreciable. Thank you.

16again · Tue Jan 26, 2021 11:23 pm

I you mark a packet before it is tunneled/encapsulated in EoIP, this marking might still be present on encapsulated EoIP. If so, use it in normal PBR rules

Your 4 different gateways.....are those 4 different devices on different WAN links? If they all have the same MAC address.......these efforts are useless

CSR460 · Wed Jan 27, 2021 4:39 am

I you mark a packet before it is tunneled/encapsulated in EoIP, this marking might still be present on encapsulated EoIP. If so, use it in normal PBR rules

Your 4 different gateways.....are those 4 different devices on different WAN links? If they all have the same MAC address.......these efforts are useless

The gateways are different servers provided by cloud, but all four IPs are on the same outgoing WAN interface (ether2), so all four IPs share the same MAC address :(

CSR460 · Wed Jan 27, 2021 7:28 am

I used "local address" properties in EoIP tunnel configuration, and now in my firewall, I can see a GRE connection from 120.79.186.223->73.238.75.222.

However, this change only affects the header of GRE packet, and it will still go to the default route. The default gatway doesn't know who is 120.79.186.223, so packet can't arrive.

Then, I tried to use Mangle rule on output chain to mark the routing. And add a routing mark on 120.79.186.223 reachable. Again, I am still able to see the connection 120.79.186.223->73.238.75.222 from the firewall and packet sniffer. But it still does not arrive. And I can't see the connection from the other side.

I checked these topics:
viewtopic.php?t=94624
viewtopic.php?t=22913
viewtopic.php?t=100322
viewtopic.php?t=112985

Still didn't solve my problem. Any idea would be helpful. Thanks!

CSR460 · Wed Jan 27, 2021 9:38 am

Problem fixed!

Turns out if I disable ether1, then remove all VLANs, without special routes settings, but just using "source address" properties of EoIP, I got what I want.

Multiple EoIP Tunnels On Single WAN But With Multiple IPs

Multiple EoIP Tunnels On Single WAN But With Multiple IPs

Re: Multiple EoIP Tunnels On Single WAN But With Multiple IPs

Re: Multiple EoIP Tunnels On Single WAN But With Multiple IPs

Re: Multiple EoIP Tunnels On Single WAN But With Multiple IPs

Re: Multiple EoIP Tunnels On Single WAN But With Multiple IPs