Two subnets, one Ethernet interface

libove · Mon Oct 29, 2012 4:32 pm

This has been discussed before. I've read everything I can find, here in the MikroTik forums and elsewhere on the Internet.
I'm still not getting it to work....

RouterBOARD 1200, firmware 2.33, RouterOS 5.5 (yes, I know, it's old; I will update it, but this is in production and I'm quite conservative, and I see nothing in the changelog to suggest that anything from 5.6 through the current 5.21 has anything to do with this problem).

One Ethernet interface (ether5), two internal IP addresses; plus public Internet address on ether6 (and the default config on ether10 which we just ignore and don't use); flat (layer 2 switches, unintelligent) network connected to ether5.

[socialpoint@MikroTik] > /ip addr print det
Flags: X - disabled, I - invalid, D - dynamic 
 0   ;;; default configuration
     address=192.168.88.1/24 network=192.168.88.0 interface=ether10 actual-interface=ether10 

 1   ;;; Orange fiber Internet connection (100Mb symmetric)
     address=xx.yy.zz.aa/30 network=xx.yy.aa.ab interface=ether6 actual-interface=ether6 

 2   ;;; LAN
     address=192.168.1.1/24 network=192.168.1.0 interface=ether5 actual-interface=ether5 

 3   ;;; LAN (also)
     address=192.168.4.1/24 network=192.168.4.0 interface=ether5 actual-interface=ether5

Just the dynamic routing table entries:
(The missing numbers in the lists below are entries which exist but are disabled, e.g. some custom routes we occasionally use when we want traffic to go out one or another Internet interface; we actually have several, but our normal configuration is for only the one on ether6 to be enabled, and all the others - as they are now - to be disabled, so they have no bearing on the current problem).

Short form

[socialpoint@MikroTik] > /ip rout print    
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 3 A S  ;;; Default Internet connection, Orange
        0.0.0.0/0                          xx.yy.zz.ab             1
 6 ADC   xx.yy.zz.aa/30      xx.yy.zz.ac     ether6                    0
10 ADC  192.168.1.0/24     192.168.1.1     ether5                    0
11 ADC  192.168.2.0/24     192.168.2.2     ether4                    0
12 ADC  192.168.3.0/24     192.168.3.2     ether7                    0
13 ADC  192.168.4.0/24     192.168.4.1     ether5                    0
14 ADC  192.168.88.0/24    192.168.88.1    ether10                   0

Long form:

[socialpoint@MikroTik] > /ip rout print det
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 3 A S  ;;; Default Internet connection, Orange
        dst-address=0.0.0.0/0 gateway=xx.yy.zz.ab gateway-status=xx.yy.zz.ab reachable ether6 distance=1 scope=30 target-scope=10 

 6 ADC  dst-address=xx.yy.zz.aa/30 pref-src=xx.yy.zz.ac gateway=ether6 gateway-status=ether6 reachable distance=0 scope=10 

10 ADC  dst-address=192.168.1.0/24 pref-src=192.168.1.1 gateway=ether5 gateway-status=ether5 reachable distance=0 scope=10 

11 ADC  dst-address=192.168.2.0/24 pref-src=192.168.2.2 gateway=ether4 gateway-status=ether4 reachable distance=0 scope=10 

12 ADC  dst-address=192.168.3.0/24 pref-src=192.168.3.2 gateway=ether7 gateway-status=ether7 reachable distance=0 scope=10 

13 ADC  dst-address=192.168.4.0/24 pref-src=192.168.4.1 gateway=ether5 gateway-status=ether5 reachable distance=0 scope=10 

14 ADC  dst-address=192.168.88.0/24 pref-src=192.168.88.1 gateway=ether10 gateway-status=ether10 unreachable distance=0 scope=200

The problem, as others have posted, is that devices on 192.168.4.0/24 cannot reach 192.168.1.0/24.
DHCP supplies IP addresses including the default gateway being the MikroTik on the same subnet as the DHCP client.
The clients can all reach anything else on their own subnet, and anything on the Internet.

Here's a sample pair which can't talk:
1. My workstation (Windows 7), DHCP-assigned IP address 192.168.4.21/24, default gateway 192.168.4.1 (the MikroTik, on ether5).
2. A simple Wi-Fi access point, 192.168.1.193 statically configured, with default gateway 192.168.1.1 (also the MikroTik, also on ether5).

Traceroute:

C:\>tracert -d 192.168.1.193

Tracing route to 192.168.1.193 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.4.1
  2     *        *        *     Request timed out.
  3  ^C

PING:

C:\>ping 192.168.1.193

Pinging 192.168.1.193 with 32 bytes of data:
Request timed out.
Request timed out.

Ping statistics for 192.168.1.193:
    Packets: Sent = 2, Received = 0, Lost = 2 (100% loss),
Control-C
^C

(Another sample pair, my same PC, and someone else's PC which has the address 192.168.1.34/24; another sample pair, that other PC, if it happens to get a 192.168.4.0/24 address, and the same Wi-Fi access point; and many other pairs, so I'm fairly confident that it's my configuration of the MikroTik, which is our LAN's only router, and not something wrong with one or another node on the LAN).

Someone commented in another thread with this problem that the MikroTik, since it can see both 192.168.1.0/24 and 192.168.4.0/24 on the same physical interface, believes that the clients should be able to do so also, and sends the ICMP Redirect in the interest of making the routing more efficient ... but the MikroTik should still be forwarding the packets anyway.

I have put in the MikroTik firewall rule to discard ICMP Redirect packets which the MikroTik wants to send on ether5:

[socialpoint@MikroTik] > /ip fire filt print
Flags: X - disabled, I - invalid, D - dynamic 
 0   chain=output action=drop protocol=icmp out-interface=ether5 icmp-options=5:0-255

This definitely stopped the clients from receiving ICMP Redirect messages (and I did remember to flush the ARP tables on the clients first, to avoid them still trying to send directly instead of going through the default router, because of having previously received and been confused by an ICMP Redirect), but nothing changed in terms of the clients' inability to talk across the 192.168.1.0/24 - MikroTik - 192.168.4.0/24 route.

I can see that the MikroTik is actually forwarding the packets. For example, if I have that other workstation, 192.168.1.34, PING me (192.168.4.21), I do receive the incoming ICMP Echo Request packets. Their source IP address is 192.168.1.34, but their source MAC is the MAC of the MikroTik, not of the sending workstation.

And yet, my workstation doesn't even try to reply. (I'm watching with Wireshark).

What could the MikroTik be doing when it routes the packets so that although they are received, have the proper source IP address, which is routable by the default route on the receiving workstation, the receiving workstation simply doesn't even respond?

Here's a received packet with the interpretation fully expanded in Wireshark:

No.     Time                          Source                Destination           Protocol Length Info
   1261 2012-10-29 15:26:06.233615000 192.168.1.34          192.168.4.21          ICMP     74     Echo (ping) request  id=0x0001, seq=129/33024, ttl=127

Frame 1261: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface 2
    Interface id: 2
    WTAP_ENCAP: 1
    Arrival Time: Oct 29, 2012 15:26:06.233615000 Romance Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1351520766.233615000 seconds
    [Time delta from previous captured frame: 0.179143000 seconds]
    [Time delta from previous displayed frame: 4.998212000 seconds]
    [Time since reference or first frame: 34.972952000 seconds]
    Frame Number: 1261
    Frame Length: 74 bytes (592 bits)
    Capture Length: 74 bytes (592 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ip:icmp:data]
    [Coloring Rule Name: ICMP]
    [Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: Routerbo_d2:c2:79 (00:0c:42:d2:c2:79), Dst: Sony_f3:a9:60 (30:f9:ed:f3:a9:60)
    Destination: Sony_f3:a9:60 (30:f9:ed:f3:a9:60)
        Address: Sony_f3:a9:60 (30:f9:ed:f3:a9:60)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: Routerbo_d2:c2:79 (00:0c:42:d2:c2:79)
        Address: Routerbo_d2:c2:79 (00:0c:42:d2:c2:79)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IP (0x0800)
Internet Protocol Version 4, Src: 192.168.1.34 (192.168.1.34), Dst: 192.168.4.21 (192.168.4.21)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
    Total Length: 60
    Identification: 0x24d6 (9430)
    Flags: 0x00
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 127
    Protocol: ICMP (1)
    Header checksum: 0x9063 [correct]
        [Good: True]
        [Bad: False]
    Source: 192.168.1.34 (192.168.1.34)
    Destination: 192.168.4.21 (192.168.4.21)
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0x4cda [correct]
    Identifier (BE): 1 (0x0001)
    Identifier (LE): 256 (0x0100)
    Sequence number (BE): 129 (0x0081)
    Sequence number (LE): 33024 (0x8100)
    Data (32 bytes)

0000  61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70   abcdefghijklmnop
0010  71 72 73 74 75 76 77 61 62 63 64 65 66 67 68 69   qrstuvwabcdefghi
        Data: 6162636465666768696a6b6c6d6e6f707172737475767761...
        [Length: 32]

No reply is sent by my workstation.

Thanks for your help!
-Jay

jandafields · Mon Oct 29, 2012 6:14 pm

In Mikrotik, you must use 2 different interfaces to route between them.

If you think this needs fixed, plaese forward your entire message to support@mikrotik.com

Feklar · Mon Oct 29, 2012 11:34 pm

In Mikrotik, you must use 2 different interfaces to route between them.

If you think this needs fixed, plaese forward your entire message to support@mikrotik.com

Actually that is flat out wrong. Using different interfaces is cleaner and can be actually secured, but is not necessary for it to route between subnets.

It could be your host firewall preventing it from responding to ICMP, that is the most likely cause, especially if you are running a version of Windows. You would also want to check what your routing table on the computers are, i.e. are they using the MikroTik as their default gateway and don't have a more specific route installed for a given subnet. The last thing to check would be you firewall rules, not just the filter rules. If you would run an export of those and pose them they can be reviewed.

libove · Tue Oct 30, 2012 12:20 am

Thank you Feklar.

I tested with the Windows firewall completely turned off. It makes no difference.

I confirm that, as stated before, the MikroTik is the default route. I also confirm that there are no more specific routes pointing a different way. And even if there were, my packet capture (on all interfaces) would have seen the responding packets.

I also tried explicitly setting a subnet route pointing at the MikroTik (although this would be redundant and make no difference, given the default route). As expected, this made no difference.

As stated, I cleared the ARP table, to assure that the host wasn't trying to send the packets directly instead of having to rely on a route. I note that no ARP request is sent out by the host to try to find a way to reach the IP address from which the incoming ICMP Echo Request packets arrive, and no ARP entry is created for that IP address on the host.

What else?
-Jay

jandafields · Tue Oct 30, 2012 1:59 am

Feklar, I'm sorry. You are completely correct. I just tried this, and it works perfectly.

The following is the ONLY configuration on the router:

add address=10.10.1.1/24 interface=ether2
add address=10.10.2.1/24 interface=ether2

I have 2 computers:

10.10.1.100/24
gatewate 10.10.1.1

-and-

10.10.2.100/24
gateway 10.10.2.1

They can both reach each other. They are both plugged into a switch that is plugged into ether2 on the routerboard (1 interface only).

So, if it isn't working, then some rule (either in the router or on the PC) is blocking it, or there is an error in the PC's network configuration.

libove · Tue Oct 30, 2012 9:03 am

Hi jandafields,
Thanks for testing and updating the thread. Could I ask you to go one step farther and run a packet capture on one of the machines to check whether they are receiving and honouring the ICMP Redirect which the MikroTik router is surely sending (honouring meaning that thereafter they communicate directly with each other, which the switches will allow) or whether they continue to talk through the MikroTik, all packets being routed? (We can check which of the two is the case by looking at the source MAC address of received packets. If the all have the MAC address of the other station, then they're bypassing the MikroTik and talking directly with each other; if they have the MAC address of the MikroTik, then they're being routed).

You could even go one step further, if you're curious to try it, by putting in the firewall filter rule to block the MikroTik from ever sending the ICMP Redirects out

/ip firewall filter add action=drop chain=output disabled=no icmp-options=5:0-255 out-interface=etherN protocol=icmp

Then remember to clear the ARP tables on both sides, and try the ping again, watching with a packet capture (like Wireshark).

Anyone, any other ideas on the weirdness that I'm suffering? I'm going bald(er!) trying to figure this out... (and the natives [users] are getting restless...).

Thanks,
-Jay

Feklar, I'm sorry. You are completely correct. I just tried this, and it works perfectly.

The following is the ONLY configuration on the router:
Code: Select all
add address=10.10.1.1/24 interface=ether2
add address=10.10.2.1/24 interface=ether2
I have 2 computers:

10.10.1.100/24
gatewate 10.10.1.1

-and-

10.10.2.100/24
gateway 10.10.2.1

They can both reach each other. They are both plugged into a switch that is plugged into ether2 on the routerboard (1 interface only).

So, if it isn't working, then some rule (either in the router or on the PC) is blocking it, or there is an error in the PC's network configuration.

Feklar · Tue Oct 30, 2012 5:19 pm

Thank you Feklar.

I tested with the Windows firewall completely turned off. It makes no difference.

I confirm that, as stated before, the MikroTik is the default route. I also confirm that there are no more specific routes pointing a different way. And even if there were, my packet capture (on all interfaces) would have seen the responding packets.

I also tried explicitly setting a subnet route pointing at the MikroTik (although this would be redundant and make no difference, given the default route). As expected, this made no difference.

As stated, I cleared the ARP table, to assure that the host wasn't trying to send the packets directly instead of having to rely on a route. I note that no ARP request is sent out by the host to try to find a way to reach the IP address from which the incoming ICMP Echo Request packets arrive, and no ARP entry is created for that IP address on the host.

What else?
-Jay

The what else would be to provide an export of your firewall rules, filter, nat, and mangle. You can also run torch on the router to see that the packets are being sent and if it is receiving anything back, also you can see what IP it is sending from and to. If it's sending and your computer is reviving them, but not sending anything back, then it's an issue with the computer not sending reply traffic.

libove · Tue Oct 30, 2012 6:59 pm

Thanks everyone. Given the amount of time this was taking, and that we didn't have a strong reason to keep the two separate subnets on the one same physical medium, we solved the issue by dodging it entirely -

We changed from 192.168.1.0/24 and 192.168.4.0/24 to 192.168.0.0/21 ....
:-/

So I guess this one just gets left hanging. I really do appreciate the time people took to try to help.

saludos,
-Jay

TroyQ · Wed Nov 02, 2016 12:00 pm

I am still struggling with mine, i need both sub nets communicating.

I cannot change the networks because both have servers and software set up for them.

Two subnets, one Ethernet interface

Two subnets, one Ethernet interface

Re: Two subnets, one Ethernet interface

Re: Two subnets, one Ethernet interface

Re: Two subnets, one Ethernet interface

Re: Two subnets, one Ethernet interface

Re: Two subnets, one Ethernet interface

Re: Two subnets, one Ethernet interface

Re: Two subnets, one Ethernet interface

Re: Two subnets, one Ethernet interface