Problems with DNS for www.google.com

jcremin · Tue Oct 20, 2009 1:21 am

Hello all,
I have a very strange issue that has been driving me crazy for the last week or two. I really hope someone here can shed some light.

The internet will be working great for all of my customers, but then I'll start getting phone calls saying that they can't get to www.google.com. I've even had it happen to me while I had myself setup with a CPE. I can ping google.com just fine, but when I add the www, ping acts as if the domain doesn't exist. If I do an nslookup to www.google.com, it usually finds the IP address, which makes me think that the DNS server is actually working fine. I ran Wireshark and watched when I tried to ping www.google.com, and the strange thing is that it didn't even look like it was sending a DNS query. I would see a couple netbios requests, but that was it. just going to google.com worked just fine.

Ok, here is how my system is setup. I have a Mikrotik PPPoE server. My customers are wireless Mikrotik (900 mhz) or UBNT (2.4 ghz) CPE's. They login via PPPoE and are configured to use OpenDNS for their DNS servers and perform NAT to the customer's LAN. It doesn't seem to matter if they have a router at their place or not. So far I have heard of problems from a couple dozen Mikrotik clients, and only one UBNT client. Most of them use Windows, but a few have Mac's and I have heard from one person who has a Mac and is having the same problem.

When a DNS query is performed, the client's computer should ask the CPE, which should directly forward the request to OpenDNS's servers. Has anyone else seen any similar issues? I'm going crazy trying to figure out is it is a problem caused by a windows update (unlikely since it appears to be happening on a Mac too), if it is something wrong with my CPE's (strange considering it just started out of the blue even for clients who haven't been touched in many months), or if Google is having a major problem (unlikely since I'm assuming it would be all over the news.

One last thing I have noticed, when I am logged into winbox and go to tools/ping, I can always ping google.com, but sometimes www.google.com gives me an error saying that it "expected an IP address".

Any thoughts?

Thanks,
Joe

skillful · Tue Oct 20, 2009 2:09 am

Do you have a NAT rule to redirect all DNS request in place? If yes, disable the rule and check if the issue is resolved.

roc-noc.com · Tue Oct 20, 2009 2:36 am

Have you tried another dns provider during the "outage"? OpenDNS works by caching your lookup for a long time. It could be easily poisoned with bad info.

Sounds like a DNS issue to me.

Tom

jcremin · Tue Oct 20, 2009 3:39 am

skillful: I only have one NAT rule in place for each customer, and that is simply to masquerade all traffic out the PPPoE connection.

Tom: Yes, I have tried a few different DNS providers.

Like I mentioned, running nslookup has always worked as far as I am aware, so the DNS server itself appears to be working fine... It just seems as if the DNS requests simply aren't getting to where they need to be.

One more weird thing I should mention... I have 3 computers at my house, which is setup like any other client of mine. One desktop and two laptops. At any given time, all three can be working, all three can be not working, or any combination of working/not working. I can sit two of the machines side by side and type "ping www.google.com" and one will work while the other gives me a "ping could not find host" message.

I'm almost at my wits end trying to figure this out, and a handful of my customers are breathing down my neck to get it fixed, and I honestly can't even tell them if it's my problem or not.

Thanks,
Joe

adrianatkins · Tue Oct 20, 2009 10:17 am

If your DNS works sometimes, then not, then it's clearly an intermittent DNS problem.

Check all the bits down the chain.

OpenDNS, then the Mikrotik that's acting as your gateway, any APs in the middle, then the CPE.

Each one can/will cache the DNS query results (depending on how you set it), as will your client PC/Mac.

It sounds like the DNS sometimes works, so the 'correct' DNS answer is cached somewhere, so it works. Then DNS fails, and the 'failure' is cached too, so it doesn't work.

On my nets i set one Mikrotik AP to act as a DNS proxy for all devices on the network.

That narrows down the search for DNS problems a lot.

LatinSuD · Tue Oct 20, 2009 10:50 am

My bet is about big DNS replies and PPPoe MTU.

Does that DNS traffic go over UDP or TCP?

I once was using a wrong DNS server which eventually cut TCP support, and big requests didn't work.

changeip · Tue Oct 20, 2009 7:25 pm

http://forum.mikrotik.com/viewtopic.php?f=1&t=35584

libor777 · Tue Oct 20, 2009 8:55 pm

Hallo, I have in network about 100 mikrotik routers and I have problem with http://www.google.com too in some sectors of network. This is not bug in DNS becouse I put http://www.google.com IP from OpenDNS: 208.69.34.231 to browser in two different parts of network in same time and in one part working google.com good and in other part not work. I have problem in 2 parts of network where is new version mikrotik 3.30. Another parts have 3.20 and older versions and http://www.google.com have no problem. If I reboot mikrotiks in part where google.com not work, it working perfect some times and than problem back again. Now I going to downgrade and I will test if problem will removing.

Good luck Libor.

Now I test it again and problem really can be in DNS, because IP: 208.69.34.231 working in all parts network but if I want finding with google.com it is not working and it jump to link: http://www.google.cz/#hl=cs&source=hp&q ... b8fee82121 and this using DNS.

omidkosari · Wed Oct 21, 2009 10:01 am

The temporary solution for this problem is

/ip dns static
add address=208.69.34.230 disabled=no name=www.google.com ttl=1d
add address=208.69.34.231 disabled=no name=www.google.com ttl=1d

jrecabeitia · Wed Oct 21, 2009 2:43 pm

When things are so strange, the way that I solved by doing the following:
a) Make a backup of the entire router
b) Reset default
c) Re-build the backup.

At least a couple of chances so I could solve similar problems. It's like a bug that is generated within the router's impossible to get otherwise.
Suerte!

jcremin · Thu Oct 29, 2009 12:21 pm

Well, as suddenly as things quit working... they have started working again. No rhyme or reason. I had given up and quit messing with things, and 2 days later, nobody was having the problem. I have still seen it once or twice, but then it goes away again.

The temporary solution for this problem is

/ip dns static
add address=208.69.34.230 disabled=no name=www.google.com ttl=1d
add address=208.69.34.231 disabled=no name=www.google.com ttl=1d

I tried that on the very first mikrotik box that the requests should have gone to. The PCs were configured to use the MT router as the DNS server, so nothing else should have been in between caching the requests. Pinging www.google.com still failed, so I'm clearly confused as to why things weren't working.

Thanks for the responses everyone. If it crops up again, I'll try to revive this thread with any new info I can find out.

Joe

mramos · Tue Nov 03, 2009 1:49 am

Hi ...

I noticed this behaviour some time ago, not only for Google but for a couple of web sites.

Since flushing DNS cache woked for me when I was trying to figure what was going on, my dirty turn around was a small 'script' that flushes the DNS cache from time to time, typed directly on scheduler, something like this:

/ip dns cache;
flush

Scheduler run each 10 or 15 minutes, I don't remember. May be once an hour works too, donno. Ok, may be I miss the main caching feature but ... better some miliseconds to a name resolution each 10 minutes than people screaming because google.com does not load.

Besides that most of those vy 'dynamic' sites defines some ip addresses ttl to 5 minutes or so anyway ...

Regards;

jcremin · Thu Nov 12, 2009 8:48 pm

Well, the DNS problem seems to be slowly reappearing for me and the biggest thing I notice is that when people have problem, it looks like their DNS record for www.google.com has a TTL of a week. Normally the TTL is only 30 seconds or so.

Does anyone have any idea on what would cause the occasional record to get cached for way too long?

iam8up · Mon Dec 07, 2009 6:17 pm

Hi ...

I noticed this behaviour some time ago, not only for Google but for a couple of web sites.

Since flushing DNS cache woked for me when I was trying to figure what was going on, my dirty turn around was a small 'script' that flushes the DNS cache from time to time, typed directly on scheduler, something like this:
Code: Select all
/ip dns cache;
flush
Scheduler run each 10 or 15 minutes, I don't remember. May be once an hour works too, donno. Ok, may be I miss the main caching feature but ... better some miliseconds to a name resolution each 10 minutes than people screaming because google.com does not load.

Besides that most of those vy 'dynamic' sites defines some ip addresses ttl to 5 minutes or so anyway ...

Regards;

Try

/ip dns set max-cache-ttl=15m

rgunderson · Wed Dec 09, 2009 2:04 am

I tried this:
/ip dns
set udp-packet-size=768 cache-max-ttl=15m
But I get a "expected end of command" error.
Where am I going wrong?

chapex · Wed Dec 09, 2009 6:39 pm

I tried this:
/ip dns
set udp-packet-size=768 cache-max-ttl=15m
But I get a "expected end of command" error.
Where am I going wrong?

te right sintax is "set max-udp-packet-size=768 cache-max-ttl=15m"

regards

jandafields · Thu Dec 10, 2009 2:11 am

RB493AH Version 4.2

I wanted to mention that I recently had a similiar problem with Google. I could ping google, but typing it into the address bar on Internet Explorer, it was unable to load. I could load it using the IP address returned by ping, but not google.com

A reboot seemed to fix it.

Thu Dec 10, 2009 11:11 am

I've gone through different topics, it seems the problem is caused by the DNS server, which reports 1W as TTL for google, which then brakes communication with the server (correct me if I wrong). The solution for the problem is to set lower TTL (is it correct?).

taglio · Thu Dec 10, 2009 12:12 pm

Hi there,

A cuple of months ago also I've noticed the problem with www.google.com.

I've noticed it at the same time at the office and at home.

In the two case the design is a simple router NAT that connect to a network and give connection to other clients, using diffent type of medium and passing different middle switch or bridge. The two networks are bridged using PPTP.

Like EXACTLY jcremin said:

The internet will be working great for all of my customers, but then I'll start getting phone calls saying that they can't get to www.google.com. I've even had it happen to me while I had myself setup with a CPE. I can ping google.com just fine, but when I add the www, ping acts as if the domain doesn't exist. If I do an nslookup to www.google.com, it usually finds the IP address, which makes me think that the DNS server is actually working fine. I ran Wireshark and watched when I tried to ping www.google.com, and the strange thing is that it didn't even look like it was sending a DNS query. I would see a couple netbios requests, but that was it. just going to google.com worked just fine.

Also i want to say that i use OPENDNS and a central DNS proxy build with mikrotik [all the station/client connect to the central DNS proxy].

I goes completly crazy and digging i've found something about a strange error of OPENDNS. For me just change DNS provider resolve the problem.

The true is that i don't found a real solution of the problem.

Best Regards,

RG.

jcremin · Thu Dec 10, 2009 5:03 pm

As an update to all of my earlier posts. I now know a little more about this problem. The issue definitely is that www.google.com (and occasionally other Google domains) is cached for 1 week. Once the problem has affected one router, anything that relies on that routers cache is affected because the record (along with the wrong TTL) is passed along right down to the customer's computer. The only fix is to flush the dns cache all the way down the chain.

The cause of the issue seems to only occur on Mikrotik routers. I originally saw the problem on other brands of routers too, but that was because all of my CPE routers were set to use my main Mikrotik router as their DNS server which was forwarding and caching DNS requests.

I have since changed my network so that all of my CPE devices currently point right to OpenDNS. What I have seen is that occasionally, a small handful of Mikrotik CPE still get the 1 week problem. The problem still gets passed along to any routers and computers they may have, but the scope of who gets affected is much smaller now. The problem definitely does not stem with OpenDNS. I have used a handful of different DNS servers (even tried setting my own server which directly used the root servers) and the problem still crops up from time to time, but again, only on Mikrotik routers.

This is a tough problem to troubleshoot since it only happens once in awhile and there is no way to duplicate it that I know of. Manually overriding the max TTL's on the Mikrotik settings SHOULD work in theory, but I haven't actually tried it as I'd prefer not to mess too much with workarounds.

Hopefully this info helps. It's a frustrating problem and I wish it could be gone for good.

Joe

jandafields · Thu Dec 10, 2009 5:07 pm

So, WHY is this causing the problem? It's not like Google is changing their IP address, so why isn't the old entry ok?

taglio · Thu Dec 10, 2009 5:14 pm

And why changing from OPENDNS to ISP provider in my mikrotik dns proxy solve COMPLETLY the problem......

jandafields · Thu Dec 10, 2009 5:28 pm

And why changing from OPENDNS to ISP provider in my mikrotik dns proxy solve COMPLETLY the problem......

Probably because it is such a random problem... as said earlier, there is no guaranteed way to duplicate it. So, the problem going away for you doesn't necessarily have anything to do with changing dns providers...

taglio · Thu Dec 10, 2009 5:31 pm

like 2 months. using it every day for work and at home [like 10 hour a day].

No problem since i've change it... For sure look here

http://googleblog.blogspot.com/2009/12/ ... c-dns.html

I've just change my DNS to the google one

jcremin · Thu Dec 10, 2009 5:38 pm

And why changing from OPENDNS to ISP provider in my mikrotik dns proxy solve COMPLETLY the problem......

This is just a guess, but maybe the ISP already overrides Google's TTL or responds in a way that the MT doesn't get confused. It's a strange problem, and I all can confirm is that for ME, using different ISP's or DNS services did not cause the problem to go away.

changeip · Thu Dec 10, 2009 7:33 pm

try setting the max udp size on the dns service to 4096 and see if that helps. if you dont want to, add a mangle log rule for udp/53 > 768 and watch how many packets trigger it.

jkevlin · Fri Dec 11, 2009 10:14 pm

I have had this problem also. I am switching from opendns to google public dns.

jandafields · Sat Dec 12, 2009 2:49 am

What is wrong with a 7-day TTL? They don't change the IP address that often. Why does it cause a problem?

Also, what is wrong with max udp over 4096?

roadracer96 · Tue Dec 15, 2009 7:26 am

I have had this happen randomly on a handful of the 40-50ish MTs I have out in the field.

Its OpenDNS, not Mikrotik. If you flush the DNS cache, it might start working, or if you just wait.

I use it for free hotspots, so it isnt a big deal. The people dont pay for the service anyways.

jcremin · Tue Dec 15, 2009 7:42 am

I hate to keep repeating this but I do NOT think that the problem is OpenDNS.... This problem first started for me before I was even using OpenDNS, and persisted with 2 OTHER dns servers.

The reason flushing the DNS cache works is because it goes and does a new dns query, usually caching the proper TTL after a flush.

jandafields · Tue Dec 15, 2009 5:58 pm

I have had this happen randomly on a handful of the 40-50ish MTs I have out in the field.

Its OpenDNS, not Mikrotik. If you flush the DNS cache, it might start working, or if you just wait.

I use it for free hotspots, so it isnt a big deal. The people dont pay for the service anyways.

Nope, not the fault of OpenDNS... like was stated above, some DNS providers probably correct for this before you receive it, so it seems like they work fine. However, just because OpenDNS does not do this does not mean it is the fault of OpenDNS. Other DNS providers have this save issue.

Pada · Wed Dec 16, 2009 1:33 am

Some people in South Africa experienced the same - see a thread on one of our local forums: http://mybroadband.co.za/vb/showthread.php?t=207826

roadracer96 · Wed Dec 16, 2009 4:54 am

How about the fact that OpenDNS redirects google.com to their own google search servers...

Using bind w/ hints to query root servers directly:

;; ANSWER SECTION:
www.google.com. 220388 IN CNAME www.l.google.com.
www.l.google.com. 292 IN A 64.233.169.103
www.l.google.com. 292 IN A 64.233.169.104
www.l.google.com. 292 IN A 64.233.169.105
www.l.google.com. 292 IN A 64.233.169.106
www.l.google.com. 292 IN A 64.233.169.147
www.l.google.com. 292 IN A 64.233.169.99

Using OpenDNS servers:

;; ANSWER SECTION:
www.google.com. 30 IN CNAME google.navigation.opendns.com.
google.navigation.opendns.com. 30 IN A 208.69.36.230
google.navigation.opendns.com. 30 IN A 208.69.36.231

From the same computer.

Now. Whois results.

whois on 64.233.169.99 returns an IP block owned by google.
whois on 208.69.36.230 returns an IP block owned by opendns.

The TTLs that bind shows are the same as what the Mikrotik shows in the cache for either address. It is just doing what the DNS response is telling it to do.

Its simply OpenDNS hacking stuff. Not a lot you can do about it.

Maybe set the cache size to 0? Lots of cheapy routers dont have any substantial dns cache.

roadracer96 · Wed Dec 16, 2009 5:08 am

I should point out that they do the same thing to search.live.com. This is how they make money.

interpoint · Sat Dec 19, 2009 6:34 pm

I too am having the same problem.

Problem:

MT with OpenDNS settings:
www.google.com works for a while and then suddenly is inaccessable
I ping www.google.com from the MT terminal console and do not get a response. Host unknown error. Clear the cache and all works again.

MT with local ISP DNS settings:
Everything works fine.

Solution:

I schedule the MT dns cache to flush every 20 minutes and this seems to solve the issue. I believe the issue is not MT but is OpenDNS and they way they hack the DNS response back to clients. This is how OpenDNS works and what it is designed to do. I am unsure why MT has a problem with the hacked response but at this stage don't really care as long as I can content filter for free and surf www.google.com reliably.

jcremin · Sat Dec 19, 2009 7:02 pm

I believe the issue is not MT but is OpenDNS and they way they hack the DNS response back to clients.

The thing is, it happens with other DNS servers other than OpenDNS. If it works with your ISP's servers, it might just be luck of the draw... but it is not isolated to only OpenDNS.

Also, there is nothing special about the results OpenDNS returns for www.google.com. All they do is return different IP addresses so their content filtering features will work with Google. The problem isn't the IP addresses that are being returned... the problem is the fact that MT decides to cache the results for 7 days instead of what OpenDNS tell it to.

I have a little over 200 CPE devices on my network. Around half of them are MT, and the other half are UBNT. Ever since I stopped routing all of my clients through a single MT DNS server, the only clients that have had any problems are the MT CPEs. The UBNT devices have had zero problem caching the wrong TTL and needing to be flushed.

Joe

jcremin · Sat Dec 19, 2009 7:21 pm

Here's more info about what OpenDNS is doing, and why they are doing it: http://blog.opendns.com/2007/05/22/goog ... -the-page/

interpoint · Sat Dec 19, 2009 7:49 pm

The UBNT devices have had zero problem caching the wrong TTL and needing to be flushed.

Joe

I didn't think that UBNT CPE had a DNS cache. I thought UBNT devices just had a DNS forward function only. Not 100% sure about this. Maybe this is why UBNT is working and MT is not. I feel that the cached entries are the problem. Can you do some more tests to rule this theory 100% in or out.

I think the issue is that the MT cached record entry for http://www.google.com (source OpenDNS) is not the same as the record entry MT would have recieved if it was not using OpenDNS.

I am using Google's DNS servers 8.8.8.8 and 8.8.4.4 on my central MT and all CPE and AP's on the network are using this central server for resolution.

I have a few Hotspots though that are set to use OpenDNS directly.
OpenDNS returns these results when I query http://www.google.com from behind the hotspots
http://www.google.com = CNAME for google.navigation.opendns.com TTL 30 seconds
google.navigation.opendns.com = A Record 208.69.34.230 TTL 30 seconds

Now.... Alternatively.

This is the result I get when I query http://www.google.com from behind my normal CPE that are using 8.8.8.8 and 8.8.4.4

http://www.google.com= CNAME for http://www.l.google.com TTL 3d
http://www.l.google.com = A Record 216.239.59.103 TTL 5m
http://www.l.google.com = A Record 216.239.59.104 TTL 5m
http://www.l.google.com = A Record 216.239.59.99 TTL 5m
http://www.l.google.com = A Record 216.239.59.147 TTL 5m

The results are totally different between OpenDNS and Third Part DNS servers.

These are my observations and as yet I have not concluded what could cause the http://www.google.com website from not working when OpenDNS is used on a MikroTik cached DNS compared to a MikroTik cached DNS using an alternative upstream DNS referance other than OpenDNS.

I know that http://www.google.com does alot of IP address load balancing and DNS roundrobin tricks to load balance the worlds traffic to their servers and anything that messes with that system [OpenDNS or MT on their own is fine but when combined the resulting inaccurate TTL or out of date A Records may the root of the issue] is causing the site to become unresolvable.

For the moment I am just going to clear my cache every 20 minutes for OpenDNS hotspots as the rest of my network 99.99% is unaffected.

I still believe that OpenDNS, MT Cache and certain host website security "redirect and spoofing avoidance protection etc" when combined together is causing the issue we are seeing. How to fix it is another story..

Keep the MT cache fresh is my solution at the moment

jcremin · Sat Dec 19, 2009 8:52 pm

I'll look into the UBNT cache thing...

The worst part about this problem is that it is so random and only happen for a day or two at a time, and then will be fine for weeks. I'll try to dig in and do some more troubleshooting next time it crops up.

jargon · Sun Dec 20, 2009 2:06 am

is possible that this problem is by blocking p2p?
i have the same problem... for some reason and at any time... DNS doesnt respond... I was totally baffled... i tried everything... and one of the things i tried was to disable a pair of p2p filter rules... Once I did that my DNS became operational...
As i was not sure ... I went back to enable the rule ... and my DNS stopped working again ... obviously disable it again ... and so far everything is going well ... about 1 hour ago

What i was using to block p2p is: http://www.taringa.net/posts/info/18988 ... rotik.html

some feedback would be great!!!

interpoint · Sun Dec 20, 2009 2:30 am

I am not blocking P2P on any router that has seen this dns issue with www.google.com

I do not have issues when I use dns servers other than OpenDNS ones.

The problem occurs after a few days or weeks and I can only fix it by clearing the dns cache or changing the dns servers to different settings and then back to the OpenDNS ones.

Caci99 · Mon Dec 21, 2009 10:16 pm

I have seen this today in my RB411
I changed dns settings, as "changeip" recomeds, a couple of weeks ago.
So my dns settings are max-udp-size=4096
Today again I experienced same problem, www.google.com wan't open and wan't ping

So, I entered routerboard and started a ping from winbox with google.com
and it answered with the error that IP is expected. After this I tried a ping to
www.l.google.com and it was ok. Again I tried ping to www.google.com and this
time it was ok too.
I went back to my PC, and opened succesfully www.google.com

I don't know what name to put into this, but someone with greater knowledge about
DNS servers could give us a clue.
My DNS server is opendns.com

omidkosari · Tue Dec 22, 2009 8:29 am

/ip dns static
add address=208.69.34.230 disabled=no name=www.google.com ttl=1d
add address=208.69.34.231 disabled=no name=www.google.com ttl=1d

and then clear all dns cache.
it solves the problem as i said before .

changeip · Tue Dec 22, 2009 6:54 pm

guys, lets get to the bottom of this. please go to terminal and export some things when this happens so we can determine what the problem is. Run some of the following WHEN THIS HAPPENS and post the output here.

/ip dns print
/ip dns cache all print terse where name="www.google.com"
/ip dns cache all print terse where name="www.l.google.com"
/ip dns cache flush
:delay 5
:resolve "www.google.com"
:delay 2
:resolve "www.l.google.com"
:delay 2
/ip dns cache all print terse where name="www.google.com"
/ip dns cache all print terse where name="www.l.google.com"

Paste the entire command set above into terminal and paste the results here. For ease, open a new terminal, paste the above, and then right click the terminal window and say 'copy all'.

also look in your cache at the www.google.com and www.l.google.com entries. Are any of them unknown type? Do you have both a CNAME and a list of A records?

Sam

rmichael · Tue Dec 22, 2009 6:58 pm

Can you script it? i.e if ping test fails run above commands.

jcremin · Tue Dec 22, 2009 8:02 pm

This happened to me once a few hours ago before I saw your script.

While I was checking, I didn't remember seeing any A records for www.l.google.com or www.google.com. There was a CNAME for www.google.com, but I don't remember what it's data was. I'll post more when I get a chance to see it happen again.

Caci99 · Tue Dec 22, 2009 8:31 pm

Sam, it is getting really frustrating today. Can't leave my post for an hour, when I turn back
I see again http://www.google.com does not open. This gives also the false idea that there is no internet.

So, I will do what you explain, but I also checked the dns cache carefully yesterday and today.
I can find http://www.google.com entry as CNAME related to http://www.l.google.com. This entry has
a TTL=24h
I found the entries of http://www.l.google.com, these were type A related to four or five IP-s with TTL=4m 25s

I noticed that when time expires for http://www.l.google.com it will trigger the TTL of http://www.google.com and restart it.
For example if TTL of http://www.google.com was 23h55m00s (sth like this) when the time of http://www.l.google.com
had expired and the entries were reissued in cache, the TTL of http://www.google.com would be reset at 24h.

I think this is normal. Only that in some cases I didn't see that happen.

Caci99 · Tue Dec 22, 2009 8:55 pm

It just happened again, here is what I got from the script posted above

/ip dns print
primary-dns: 80.78.66.66
secondary-dns: 208.67.222.222
allow-remote-requests: yes
max-udp-packet-size: 512
cache-size: 1024KiB
cache-max-ttl: 1d
cache-used: 103KiB
[admin@mikrotik] > /ip dns cache all print terse where name="www.google.com"
13 name=www.google.com type=CNAME data=www.l.google.com ttl=23h49m43s

[admin@mikrotik] > /ip dns cache all print terse where name="www.l.google.com"

[admin@mikrotik] > /ip dns cache flush
[admin@mikrotik] > :delay 5
[admin@mikrotik] > :resolve "www.google.com"
[admin@mikrotik] > :delay 2
[admin@mikrotik] > :resolve "www.l.google.com"
[admin@mikrotik] > :delay 2
[admin@mikrotik] > /ip dns cache all print terse where name="www.google.com"
14 name=www.google.com type=CNAME data=google.navigation.opendns.com ttl=25s

[admin@mikrotik] > /ip dns cache all print terse where name="www.l.google.com"
15 name=www.l.google.com type=A data=216.239.59.103 ttl=1m39s
16 name=www.l.google.com type=A data=216.239.59.99 ttl=1m39s
17 name=www.l.google.com type=A data=216.239.59.104 ttl=1m39s
18 name=www.l.google.com type=A data=216.239.59.147 ttl=1m39s

missinlnk · Tue Dec 22, 2009 9:09 pm

You seem to be pretty frustrated, could you for the moment stop using OpenDNS and instead use a different set of DNS servers? That doesn't fix the problem, but it would at least resolve your frustration until we can pin down exactly what is happening.

Has anyone ever seen this behavior when using OpenDNS in any setup without a Mikrotik in the loop?

Caci99 · Tue Dec 22, 2009 9:46 pm

Well, OpenDNS isn't my primary DNS server at the moment
I can change it for the sake of testing

fewi · Tue Dec 22, 2009 9:49 pm

Well, OpenDNS isn't my primary DNS server at the moment
I can change it for the sake of testing :)

RouterOS will automatically balance requests between the primary and secondary DNS and not just use the secondary when the primary does not respond. So if any of the configured nameservers are OpenDNS, some (half?) of your queries will be answered by it.

changeip · Tue Dec 22, 2009 10:09 pm

if one is opendns and the other is something else, opendns google answers are different, differing answers might be ignored by the cache resolver. better to point them both at opendns, or both somewhere else.

Caci99 · Tue Dec 22, 2009 10:56 pm

I changed the DNS servers to my ISP, but anyway, they were both previosly
pointed to OpenDNS when things happened. What you see in my post
is one my tryes to figure out what was happening.

For the last two hours looks like things are ok

jcremin · Wed Dec 23, 2009 5:18 am

You seem to be pretty frustrated, could you for the moment stop using OpenDNS and instead use a different set of DNS servers? That doesn't fix the problem, but it would at least resolve your frustration until we can pin down exactly what is happening.

Has anyone ever seen this behavior when using OpenDNS in any setup without a Mikrotik in the loop?

As I mentioned shortly after I began this thread, I first starting having the Google problem before I had ever used OpenDNS. Switching to OpenDNS was one of the things I tried, hoping it would fix the problem. So how does not using OpenDNS help resolve any frustration when it has been reported that this still happens while not using OpenDNS? Either way, I just switched my entire network over to using Google's own public DNS service, so I'm now getting the records right from the horse's mouth, as the saying goes.

Two more things that I would like to mention: 1) When this problem occurs, I quite often find that the dns cache on the client computer needs to be flushed as well as the MT cache because the bad response has been passed along and cached by the computer. If this is not a Mikrotik bug, then why do none of my clients on non-MT devices have problems? and 2) if this problem is OpenDNS's fault, why is there nothing on their forums reporting this problem, other than from a couple other Mikrotik users?

I'm just about to dig into a customer's router who reported the google problem, so I'll report what I find in a few minutes.

Joe

jcremin · Wed Dec 23, 2009 5:26 am

Ok, here's what I see...

[admin@CPE] > /ip dns print
            primary-dns: 8.8.8.8
          secondary-dns: 8.8.4.4
  allow-remote-requests: yes
    max-udp-packet-size: 512
             cache-size: 2048KiB
          cache-max-ttl: 1w
             cache-used: 83KiB

[admin@CPE] > /ip dns cache all print terse where name="www.google.com"
 0   name=www.google.com type=CNAME data=www.l.google.com ttl=5d23h46m32s 

[admin@CPE] > /ip dns cache all print terse where name="www.l.google.com"

[admin@CPE] > /ip dns cache flush
[admin@CPE] > :delay 5
[admin@CPE] > :resolve "www.google.com"
[admin@CPE] > :delay 2
[admin@CPE] > :resolve "www.l.google.com"
[admin@CPE] > :delay 2
[admin@CPE] > /ip dns cache all print terse where name="www.google.com"
 1   name=www.google.com type=CNAME data=www.l.google.com ttl=2d23h57m51s 

[admin@CPE] > /ip dns cache all print terse where name="www.l.google.com" 
 2   name=www.l.google.com type=A data=209.85.225.147 ttl=2m34s 
 3   name=www.l.google.com type=A data=209.85.225.105 ttl=2m34s 
 4   name=www.l.google.com type=A data=209.85.225.106 ttl=2m34s 
 5   name=www.l.google.com type=A data=209.85.225.104 ttl=2m34s 
 6   name=www.l.google.com type=A data=209.85.225.99 ttl=2m33s 
 7   name=www.l.google.com type=A data=209.85.225.103 ttl=2m33s

chenier · Wed Dec 23, 2009 3:47 pm

I am having same problem.

Have been using Tic for several months.
Just last night configured to point to OpenDNS for my teenager's sake

cannot reach google.com
but can reach mail.google.com, etc

will look further and post if I figure anything out

jcremin · Thu Dec 24, 2009 9:11 am

I didn't think that UBNT CPE had a DNS cache. I thought UBNT devices just had a DNS forward function only. Not 100% sure about this. Maybe this is why UBNT is working and MT is not. I feel that the cached entries are the problem. Can you do some more tests to rule this theory 100% in or out.

You were correct about the UBNT devices. Mike Ford from UBNT confirmed that they do not cache the responses and only forward them. I agree that the cached entries are what keeps www.google.com from working, but even if the UBNT devices don't cache the responses, the client's router or computer should be caching the responses that the UBNT devices pass along, right?

Do you know if you can flush one record at a time from the MT cache instead of a full flush? It appears that there is only one cached entry that is causing the whole problem... The CNAME entry for www.l.google.com is the only record that appears for www.google.com while the problem is happening.

interpoint · Thu Dec 24, 2009 1:48 pm

I think the best way is to put in a static record and this will take precedence over the cache.
What concerns me is there any other sites that we do not know about ?

Caci99 · Thu Dec 24, 2009 2:42 pm

I think the best way is to put in a static record and this will take precedence over the cache.
What concerns me is there any other sites that we do not know about ?

This is a work around. We need to come to the bottom of the problem and look for its solution.
I switched to the ISP DNS and it is working okay during these last dayes, but (there is always a but

)
the ISP DNS does not resolve some other pages correctly from time to time.

I guess there could be some incompatibility between the method of OpenDNS and Mikrotik Cache.
I am not an DNS expert so can't find what is going wrong.

changeip · Thu Dec 24, 2009 7:09 pm

If someone can get me a pcap of port 53 while this is happening I will look at them closely and see if I can determine the cause. It would be nice of you to :resolve "test.example.org" before and :resolve "test2.example.org" after while this is happening. Kind of like a marker in the pcap file for me to find the steps you did.

mersudin · Fri Dec 25, 2009 12:39 am

u could use Comodo secure dns it's free as secundary dns, or if u are ISP u might wana Bind9 as DNS and multiple forwarders

WirelessRudy · Tue Jan 12, 2010 1:27 am

OK guys,

I have been busy with other problems lately so missed this discussion entirely. But today the "google-dns" issue hit my network too again. The last time was in November and at that time not a lot of noise about it on this forum.

All my 200 clients have dns servers pointing to my edge gateway running ros3.30 where even requests to other servers is cathed and re-directed to the cache. So if the problem is there it is there basically for everybody.

My both dns server of the edge router point to the both OpenDNS server and all symptoms mentioned in this tread are mine too.

But anyway, it is January 12th now and the last post on this tread is dec.25th.....
Is the problem now found and the issue solved and thus the discussion closed?

Or is eveybody still in the X-mas and NY mood and too tired to write on the forum....

rgds.

WirelessRudy · Tue Jan 12, 2010 1:32 am

Forgot to mention: Where is the input on the issue from MT? I believe to see one small remark from sergeis. The rest... dead silence...

jcremin · Tue Jan 12, 2010 4:44 am

As I mentioned in my last post, I have changed my DNS servers to Google's public DNS service and I haven't see the problem since then, however I quite often went weeks or a month between issues before, so I'm still not going to rule it out. Assuming the problem never comes back for me, but does for other MT users using OpenDNS, the problem would have to be a combination of the two because I still have yet to hear about a non-MT user having any of these problems.

I still have issues assuming that the problem is OpenDNS because I wasn't using OpenDNS when the problem first cropped up.

Joe

WirelessRudy · Tue Jan 12, 2010 5:03 am

Well, using the Google servers don't sound all too good for me. Its like asking the crook the best path through down town area. I wouldn't be surprised you end up where you didn't want to go.....
And basically what you did is avoiding the symptoms, we need to cure the disease.

The idea of using OpenDNS is to have real independent search results. By Google you have no guarantee that the page you get is also the best one, more the best one in their eyes.... But that beside...

I have been reading what OpenDNS has been writing about what google does (and Dell) and how they bypassed it.

But the issue needs to be cleared by MT (or the users it looks like). I hope to hear more input on this issue.

Rudy

jcremin · Tue Jan 12, 2010 5:47 am

I agree 100% with what you said, but if you read through the whole thread, so many people just want to point their finger at OpenDNS and can't get past that. I don't want to use Google's dns servers, but I figure if www.google.com gets screwed up while using their own servers, the problem HAS to be with MT.

At least I know if the problem never crops up, that there has to be a problem with the combination of OpenDNS and MT. If it does happen again, then hopefully we can all get on the same page that the problem is with MT and try to solve this thing.

Joe

Caci99 · Wed Jan 13, 2010 12:50 pm

And who is the crook?

As I have mentioned before on this thread, I had problems too.
But swiching to ISP DNS server does not give any more that issue.

I think the problem is between Open DNS and MikroTik cache.

Caci99 · Wed Jan 13, 2010 1:58 pm

This is really funny, as I was writing before to this thread I found out that google.com
wasn't working again. My college had used a back-up to restore some settings and along
with those settings were changed the DNS ones, going back to Open DNS.
The router has worked with these DNS settings for 10 days almost and here is the result:

/ip dns print
primary-dns: 208.67.222.222
secondary-dns: 208.67.220.220
allow-remote-requests: yes
max-udp-packet-size: 4096
cache-size: 8192KiB
cache-max-ttl: 1d
cache-used: 22KiB
[admin@mikrotik] > /ip dns cache all print terse where name="www.google.com"

[admin@mikrotik] > /ip dns cache all print terse where name="www.l.google.com"

0 name=www.l.google.com type=A data=74.125.39.105 ttl=8s
1 name=www.l.google.com type=A data=74.125.39.99 ttl=8s
2 name=www.l.google.com type=A data=74.125.39.104 ttl=8s
3 name=www.l.google.com type=A data=74.125.39.106 ttl=8s
4 name=www.l.google.com type=A data=74.125.39.147 ttl=8s
5 name=www.l.google.com type=A data=74.125.39.103 ttl=8s

[admin@mikrotik] > /ip dns cache flush
[admin@mikrotik] > :delay 5
[admin@mikrotik] > :resolve "www.google.com"
[admin@mikrotik] > :delay 2
[admin@mikrotik] > :resolve "www.l.google.com"
[admin@mikrotik] > :delay 2
[admin@mikrotik] > /ip dns cache all print terse where name="www.google.com"
6 name=www.google.com type=CNAME data=google.navigation.opendns.com ttl=26s

[admin@mikrotik] > /ip dns cache all print terse where name="www.l.google.com"

[admin@mikrotik] >

WirelessRudy · Wed Jan 13, 2010 11:46 pm

.
But swiching to ISP DNS server does not give any more that issue.

I think the problem is between Open DNS and MikroTik cache.

I switched to OpenDNS because my ISP's servers were very unreliable and OpenDNS has more usefull features.

R.

doush · Thu Jan 14, 2010 1:09 am

The problem hit me aswell today. I cant reach google.com or gmail.com. Im using open dns and 8.8.8.8 as secondary DNS. Anyone found a reliable solution for this problem ?

changeip · Thu Jan 14, 2010 6:46 am

you cannot mix opendns and another resolver. since opendns forges responses they will conflict with someone elses answers. stick with opendns only, or no opendns at all, but do not mix them. If you get conflicting NS or SOA records back your cache will go crazy.

; <<>> DiG 9.5.1-P3 <<>> @208.67.222.222 -t any www.google.com

;; QUESTION SECTION:
;www.google.com. IN ANY

;; ANSWER SECTION:
www.google.com. 30 IN CNAME google.navigation.opendns.com.

OpenDNS gives you back a CNAME with no additional records. This forces the resolver to make yet another DNS query, which not all clients probably will obey. RouterOS dns resolver sure doesnt.

; <<>> DiG 9.5.1-P3 <<>> @8.8.8.8 -t any www.google.com

;; QUESTION SECTION:
;www.google.com. IN ANY

;; ANSWER SECTION:
www.google.com. 86400 IN CNAME www.l.google.com.
www.l.google.com. 300 IN A 72.14.213.104
www.l.google.com. 300 IN A 72.14.213.106
www.l.google.com. 300 IN A 72.14.213.105
www.l.google.com. 300 IN A 72.14.213.103
www.l.google.com. 300 IN A 72.14.213.147
www.l.google.com. 300 IN A 72.14.213.99

Google and other dns servers hand back the CNAME and the A records in a single response. My guess is that OpenDNS fudging the responses and handing back answers you didnt ask for is breaking things in RouterOS's half inplemented DNS cache.

Has anyone confirmed that not using OpenDNS (and not anyone elses dns cache that fudges results) still results in this problem?

missinlnk · Fri Jan 15, 2010 1:11 am

*Edited cause I didn't read the last post properly*

From everything we've seen, OpenDNS is the only DNS provide that has caused this issue. So does anyone know if response from OpenDNS is a proper response from a DNS server per the DNS specs? Or should the Mikrotik DNS service not need to expect this kind of response to a query?

roadracer96 · Fri Jan 15, 2010 1:50 am

In my research, it is a proper response. Just not an accurate response. If it really chaps yer ass, setup your DNS sever to forward all requests to OpenDNS, and *.google.com/*.gmail.com to root servers. Problem solved.

The only reason I use OpenDNS is for the content filtering. Works great to proxy-redirect block.opendns.com to http://www.disney.com. Those trying to surf for hardcore smut will end up getting Pochohontas...

hilton · Fri Jan 15, 2010 11:08 am

Works great to proxy-redirect block.opendns.com to http://www.disney.com. Those trying to surf for hardcore smut will end up getting Pochohontas... :D

Poke oh who? :-)

roadracer96 · Fri Jan 15, 2010 5:11 pm

There ya go. The first response with the 24hr TTL is the one for the real google from real servers. The second one after you flush is the one from the OpenDNS servers.

I bet the problem is that Mikrotik is round-robining the connections between DNS servers. One request gets the long TTL, so MT sticks with it, but the next request goes to OpenDNS with the different TTL.

What happens when you dont mix DNS servers? Use JUST OpenDNS.

lukkes · Fri Jan 15, 2010 5:38 pm

As an update to all of my earlier posts. I now know a little more about this problem. The issue definitely is that http://www.google.com (and occasionally other Google domains) is cached for 1 week. Once the problem has affected one router, anything that relies on that routers cache is affected because the record (along with the wrong TTL) is passed along right down to the customer's computer. The only fix is to flush the dns cache all the way down the chain.

The cause of the issue seems to only occur on Mikrotik routers. I originally saw the problem on other brands of routers too, but that was because all of my CPE routers were set to use my main Mikrotik router as their DNS server which was forwarding and caching DNS requests.

I have since changed my network so that all of my CPE devices currently point right to OpenDNS. What I have seen is that occasionally, a small handful of Mikrotik CPE still get the 1 week problem. The problem still gets passed along to any routers and computers they may have, but the scope of who gets affected is much smaller now. The problem definitely does not stem with OpenDNS. I have used a handful of different DNS servers (even tried setting my own server which directly used the root servers) and the problem still crops up from time to time, but again, only on Mikrotik routers.

This is a tough problem to troubleshoot since it only happens once in awhile and there is no way to duplicate it that I know of. Manually overriding the max TTL's on the Mikrotik settings SHOULD work in theory, but I haven't actually tried it as I'd prefer not to mess too much with workarounds.

Hopefully this info helps. It's a frustrating problem and I wish it could be gone for good.

Joe

I have the same problem with google AND MICROSOFT:COM, sometimes i try to udate the msn and other from microsoft and give me load error, BUT THE SURPRISE IS THAT I WAS WORKING WITH LINUX ROUTER NOT MIKROTIK.... but if i make a nslookup microsoft.com an put the ip in the browser the page load perfect. but the internal links give error again because the link point to a dns names.

Mon Jan 18, 2010 12:30 pm

roadracer96, yes it is true, that /ip dns can work round-robin for both configured DNS servers. When router needs to get IP/DNS for new address, which is not in the cache, round-robin is true, when both servers sent good replies for long time.

WirelessRudy · Thu Jan 21, 2010 3:07 am

Sergeis:

Is MT working on this issue?
It seems a lot of MT users use OpenDNS and since most of the world users use google this is definitely an issue that needs to be solved.

Does anybody already have a more permanent workaround for this issue? And I am not asking a solution that means abandon OpenDNS. And flushing the dns cache is also not always helping.
Right now I flushed my dns cache 4 times in a row and flushed my browser's history, even made a reboot on my PC. Still not able to reach www.google.com. Other nationality google's no problem, but the .com stays unreachable.
I can ping the site which then give me the IP and the browser then reaches it fine so it is not an issue their server is down (what would seem to be a very rare and unlikely event to me anyway!)

I think MT should step in here to give a more permanent solution.

roadracer96 · Thu Jan 21, 2010 3:54 am

How many people having this problem are ISPs? Just curious.

hilton · Thu Jan 21, 2010 9:13 am

How many people having this problem are ISPs?

I'm not an ISP but I use Mikrotik as LAN routers for my customers and I've experienced this problem. However not since moving away to another DNS (either Google or the ISP DNS).

Thu Jan 21, 2010 12:26 pm

WirelessRudy, currently I do not see, where is MikroTik fault, as it is working as middle chain.
User requests www.google.com, DNS cache is asking for DNS server, DNS server is replying.
If DNS server reply is not too good, why MikroTik DNS cache should change it.

Have you tried another DNS server, do you have the same problem?

jcremin · Thu Jan 21, 2010 4:23 pm

WirelessRudy, currently I do not see, where is MikroTik fault, as it is working as middle chain.
User requests http://www.google.com, DNS cache is asking for DNS server, DNS server is replying.
If DNS server reply is not too good, why MikroTik DNS cache should change it.

Have you tried another DNS server, do you have the same problem?

Have you not read this whole thread? This is getting really frustrating and nobody at MT seems to care. Here's the basic summary of this thread:

1) We can confirm that while using an MT router with OpenDNS, www.google.com won't load properly.
2) We aren't positive yet, so we can't confirm that it only happens with OpenDNS, but that seems to be the case.
3) But we CAN confirm that it only happens with MT routers, nobody else using OpenDNS is having this problem.
4) Something in MT's DNS goofs up www.google.com with the combination of MT and OpenDNS.

Did I miss anything?

Caci99 · Thu Jan 21, 2010 4:37 pm

WirelessRudy, currently I do not see, where is MikroTik fault, as it is working as middle chain.
User requests http://www.google.com, DNS cache is asking for DNS server, DNS server is replying.
If DNS server reply is not too good, why MikroTik DNS cache should change it.

Have you tried another DNS server, do you have the same problem?
Have you not read this whole thread? This is getting really frustrating and nobody at MT seems to care. Here's the basic summary of this thread:

1) We can confirm that while using an MT router with OpenDNS, http://www.google.com won't load properly.
2) We aren't positive yet, so we can't confirm that it only happens with OpenDNS, but that seems to be the case.
3) But we CAN confirm that it only happens with MT routers, nobody else using OpenDNS is having this problem.
4) Something in MT's DNS goofs up http://www.google.com with the combination of MT and OpenDNS.

Did I miss anything?

You are right

Actually I have switched to the recently google DNS servers, so far so good.

rmichael · Thu Jan 21, 2010 8:35 pm

Anyone tried to apporach OpenDNS about it? I would think it would be in self interest to work with as many setups as possible.

missinlnk · Thu Jan 21, 2010 8:58 pm

Have you not read this whole thread? This is getting really frustrating and nobody at MT seems to care. Here's the basic summary of this thread:

1) We can confirm that while using an MT router with OpenDNS, http://www.google.com won't load properly.
2) We aren't positive yet, so we can't confirm that it only happens with OpenDNS, but that seems to be the case.
3) But we CAN confirm that it only happens with MT routers, nobody else using OpenDNS is having this problem.
4) Something in MT's DNS goofs up http://www.google.com with the combination of MT and OpenDNS.

Did I miss anything?

One thing, it's been covered several times in this thread but since we're recapping it'd be good to double check. In this setup, both the Primary and Secondary DNS servers are pointing to OpenDNS servers, correct?

(For those coming into the thread late, if you have one OpenDNS server and one DNS server from another provider there will be a problem. OpenDNS returns different results compared to other DNS servers, and Mikrotik will bounce between using both primary and secondary DNS servers regularly when doing its querying.)

Scott

jcremin · Thu Jan 21, 2010 9:32 pm

I haven't been in direct communication with OpenDNS, but browsing through their forums a handful of times, the only reports of this problem that I could find were from other MT users. I haven't been on their forums in a couple weeks, so I'll check them again later to see if I can find any direct correspondence from OpenDNS regarding these issues.

roadracer96 · Thu Jan 21, 2010 11:12 pm

WirelessRudy, currently I do not see, where is MikroTik fault, as it is working as middle chain.
User requests http://www.google.com, DNS cache is asking for DNS server, DNS server is replying.
If DNS server reply is not too good, why MikroTik DNS cache should change it.

Have you tried another DNS server, do you have the same problem?
Have you not read this whole thread? This is getting really frustrating and nobody at MT seems to care. Here's the basic summary of this thread:

1) We can confirm that while using an MT router with OpenDNS, http://www.google.com won't load properly.
2) We aren't positive yet, so we can't confirm that it only happens with OpenDNS, but that seems to be the case.
3) But we CAN confirm that it only happens with MT routers, nobody else using OpenDNS is having this problem.
4) Something in MT's DNS goofs up http://www.google.com with the combination of MT and OpenDNS.

Did I miss anything?

Ive read the whole thread and everything that I can see is that MT follows exactly what the response from the DNS server is to a T.

You say that something in MT's DNS goofs up google.com with the combination of MY and OpenDNS. My contention and from what I gather, the contention of the people at Mikrotik is that it is something that OpenDNS is doing.

My bet is that OpenDNS is doing something odd or just outside of RFC for DNS and other DNS clients are more forgiving to it.

Mikrotik has ZERO problems with any other DNS server that I have ever seen. With 50 RBs using OpenDNS in production, I have only seen it happen 1 time. Before I used OpenDNS, I had ZERO problems.

Maybe a proof of concept would be to use BIND forwarding to OpenDNS and point MT to BIND and see what happens.

rmichael · Fri Jan 22, 2010 1:30 am

If DNS server reply is not too good, why MikroTik DNS cache should change it.

Could you or someone else explain which response is not "not good". From what I gather the response is unusual but not out of RFC spec (I'm talking about CNAME reply).

thank you.

WirelessRudy · Fri Jan 22, 2010 3:14 am

Last night I spend some hours in just googling and reading all kinds of forums I came across mentioning the same or sort of same problem.

I could not see that the many treads on the OpenDNS forums regarding the subject have been from MT users. In contrary, some stated specifically they used D-link or other domestic type of adsl routers.

Although I got the impression by this forum it is a MT problem with OpenDNS I now embrace the idea it is more an issue between OpenDNS and Google.

I have been reading an article (can`t find it no more) from OpenDNS were they explained why resolving to google.com (and the caching in their cache system) is done a bit different then on most other sites. If I remember well it has to do with the way google is trying to track visitors to their servers and the way google reacts when possible visitors make typo's.

OpenDNS has, to stop this and assure a ´neutral´ browsing experience set-up some filter/proxy system that should guard against this ´phising´ practises of google.
Normally their system works fine but now and then it is causing problems. In most treads of the OpenDNS forum the given solution is basically ´to flush´ your cache and flush your assigned OpenDNS cache and the problem solved.

No further solutions are given by OpenDNS. Very frustrating and by the amount of closed but not solved treads very un-satisfaction-airy.

Who is now the culprit in all this? Google probably blames OpenDNS or the router manufacturer while OpenDNS points out to the ´questionable´ politics of google. The router manufacturer hides behind the argument it is not their problem because they don't have the problem... But we, the users, are left in the middle!

Maybe we should all start putting the issue at the OpenDNS forums and google forums. Maybe this thing becomes so big it will surface into the normal news and some parties are forced to do something about it!

In my opinion the issue is a result of the attempts of google to gain access to the interesting dns resolving market. As biggest advertisement company on the net and recently issued a competing (but less sophisticated) public dns system as OpenDNS their interest is obvious and I can see why they made things such that people reaching their servers from a competing public resolver are having problems to such extend these users might drop the competitor's system in favour of their own.

Proof that their tactics work is that some on this forum already showed they moved to the google system!
Very short sighted though! Didn`t we all dislike some years ago how powerful Microsoft became? Why should we now help yet another company trying to achieve the same.

I think we all should help OpenDNS and/or this community to find a workable solution.

roadracer96:

I don't know why you want to know how many members with the issue are ISP's. I am one, a small one but I'll bet most members of this forum are small or bigger ISP's. But at the same time treads on forums elsewhere show the problem also exist amongst single users of all kind of equipment.
In general the single user coming across the problem just end his session and goes on browsing the internet the next day when the problem usually has disappeared. Or they are so illiterate they don't know what happens to them anyway.
ISP's on the other hand control usually more clients and thus if there is any problem amongst their users they will also have a bigger change to get the notification from a user and out of professional point of view also will take bigger effort to solve the problem or bring it under attention amongst us...
So that's why you probable see more relative small ISP's putting this issue to this forum while the bigger ones probably have more resources to find solutions or workarounds themselves and the ample single user joining in the discussion will probably a curiosity. (Some call them ´nerds´

)

roadracer96 · Fri Jan 22, 2010 3:52 am

If you are a small ISP or a large ISP, you shouldnt be using OpenDNS for your DNS servers. You should have your own caching servers that query root.

That is my point.

jcremin · Fri Jan 22, 2010 4:45 am

Last night I spend some hours in just googling and reading all kinds of forums I came across mentioning the same or sort of same problem. I could not see that the many treads on the OpenDNS forums regarding the subject have been from MT users. In contrary, some stated specifically they used D-link or other domestic type of adsl routers.

Ok... Post links to them. Most of the issues I have seen with the users of consumer routers have been mis-configured settings or ISP issues. Here's what I find Looking through all the threads back to the beginning of August:

http://forums.opendns.com/comments.php? ... ionID=5242 - This one the problem ended up being a messed up hosts file
http://forums.opendns.com/comments.php? ... ionID=5211 - The users couldn't even diagnose the issue and didn't really know what they were talking about.
http://forums.opendns.com/comments.php? ... ionID=5014 - This was an international connectivity issue with the user's ISP.
http://forums.opendns.com/comments.php? ... ionID=4562 - This appears to be a DDOS/Comcast issue.
http://forums.opendns.com/comments.php? ... ionID=5150 - The user didn't even try troubleshooting and the thread is closed.
http://forums.opendns.com/comments.php? ... ionID=5828 - This is the one where you chimed in about the problems, but until that point, it was obvious that the original poster had no idea how to troubleshoot and gave up.

http://forums.opendns.com/comments.php? ... ionID=5726 - MT users having this problem....
http://forums.opendns.com/comments.php? ... ionID=5750 - MT users having this problem....

I have been reading an article (can`t find it no more) from OpenDNS were they explained why resolving to google.com (and the caching in their cache system) is done a bit different then on most other sites.

I don't think that is the problem at all. All they are doing is returning results that point to a different IP address to "proxy" google traffic through their servers so you can use the filtering options they provide.

In my opinion the issue is a result of the attempts of google to gain access to the interesting dns resolving market.

Ok, let's stop the speculation about the "big bad giant". This is pure speculation and just an easy attempt to point the finger at Google. If Google were doing this, all the "nerds" around the world would have picked up on this and this would have been a major news issue two or three months ago.

I don't know why you want to know how many members with the issue are ISP's. I am one, a small one but I'll bet most members of this forum are small or bigger ISP's. But at the same time treads on forums elsewhere show the problem also exist amongst single users of all kind of equipment. ISP's on the other hand control usually more clients and thus if there is any problem amongst their users they will also have a bigger change to get the notification from a user and out of professional point of view also will take bigger effort to solve the problem or bring it under attention amongst us...

Again, I disagree. I can't find enough threads to support your claim about different kinds of equipment. Think about how small the MT market is compared to Linksys, D-link, Netgear, etc combined. Why am I finding just as many posts from the very small MT userbase as the rest of those combined.

Also, there are some very smart networking people that use consumer brand routers. Not everyone who uses a Linksys router is an idiot. Let me put it this way, I have a TON of other brand routers in my network... NOT ONE OF THEM HAVE HAD THIS ISSUE even while they were pointed at OpenDNS. Almost every single MT router on my network had problems, and if you look way back to just after I started this thread, I reported that I switched to OpenDNS AFTER the problem first cropped up.

I'm not trying to yell at you... I just want this problem to be solved.

jcremin · Fri Jan 22, 2010 4:45 am

If you are a small ISP or a large ISP, you shouldnt be using OpenDNS for your DNS servers. You should have your own caching servers that query root.

That is my point.

And this is a rule that we're all supposed to obey? Yes, if you are a very large ISP, you are right, that it probably makes more sense to run your own DNS server, but for a small ISP, there is zero reason why running your own DNS server is a requirement. Should every small business have their own DNS server?

Don't forget, the primary target audience for OpenDNS is for those TRYING TO GET AWAY FROM USING THEIR ISP'S DNS SERVERS.

I apologize for my recent posts being snotty, but I'm getting tired of everyone trying to point the finger instead of helping to troubleshoot this problem...

changeip · Fri Jan 22, 2010 4:53 am

Should every small business have their own DNS server?

Yes

roadracer96 · Fri Jan 22, 2010 4:56 am

If you are a small ISP or a large ISP, you shouldnt be using OpenDNS for your DNS servers. You should have your own caching servers that query root.

That is my point.
And this is a rule that we're all supposed to obey? Yes, if you are a very large ISP, you are right, that it probably makes more sense to run your own DNS server, but for a small ISP, there is zero reason why running your own DNS server is a requirement. Should every small business have their own DNS server?

Don't forget, the primary target audience for OpenDNS is for those TRYING TO GET AWAY FROM USING THEIR ISP'S DNS SERVERS.

I apologize for my recent posts being snotty, but I'm getting tired of everyone trying to point the finger instead of helping to troubleshoot this problem...

So use ISPs reliable DNS servers, or use OpenDNS and have failures.

The solution seems pretty obvious to me.

Remember when you were a little kid and hurt yourself? You went to mom and said "It hurts when I do this..." And she said "Dont do that, then."

And yes. If you run an ISP, you should have your own DNS servers. Period. I run a small business and even I have my own DNS servers. My business isnt even internet based. I have 4 DNS servers... 2 Linux and 2 Windows AD.

It takes all of 15 minutes from empty hard drive to a caching BIND DNS server.

EDIT:

Its reliable, too...

[root@luke ~]# uptime
21:59:49 up 673 days, 23:40, 1 user, load average: 0.04, 0.04, 0.00

That is an ISP that I do side work for. The other DNS server is about 20 minutes less uptime. The authoritative servers are at 5xx days now.

I havent flushed the cache once.

jcremin · Fri Jan 22, 2010 5:16 am

No offense roadracer96 and changeip, but I disagree on that requirement. I suppose I should start including a dedicated DNS server to each and every one of my business clients. Actually that would be a huge waste of resources, potential space, and electricity. It's like saying everyone alive should own their own car even when riding the bus or carpooling works just as well.

At any rate, that's not here or there... the point is that I'm trying to figure out how to FIX the problem and not just avoid it. When I started this thread, I didn't ask for workarounds. This thread is getting so long that I would like to ask that all future posts be directly related to troubleshooting.

Thanks,
Joe

roadracer96 · Fri Jan 22, 2010 5:42 am

OK. Troubleshooting.

What happens when you dont use OpenDNS?

EDIT: And if you are a small ISP and dont have your own DNS server, you are doing something severely wrong.

rmichael · Fri Jan 22, 2010 5:57 am

OK. Troubleshooting.

What happens when you dont use OpenDNS?

Question should be: what happens withen you use MT with OpenDNS?

EDIT: And if you are a small ISP and dont have your own DNS server, you are doing something severely wrong.

It's perfectly fine to use someone else for DNS it's called outsourcing

changeip · Fri Jan 22, 2010 7:43 am

yep, lets stick to the topic, figuring this out. we really need pcaps of dns queries to understand the responses. can anyone having this problem packet sniff on udp/53 on the wan interface so we can investigate more? I can work directly with David over there and get this fixed if its something they are doing wrong.

jcremin · Fri Jan 22, 2010 7:59 am

I switched a few customers and my office back to the OpenDNS servers. I'll watch to see if the problem crops up again and will wireshark as much as I can (fingers crossed that it shows something useful!)

Fri Jan 22, 2010 12:08 pm

Yes, I agree with changeip.
Please, give us OpenDNS servers which produce the problem with Google.
We will see pcaps what opendns are replying for Goggle requests, and see whether we can do anything for it or not.

P.S. jcremin, I read the whole post, as problem is not present with not openDNS servers, MikroTik DNS is not server, but cache.
It would be great you can provide with few OpenDNS servers, which produce problem 100% for you or at least pcaps with the problem present.

Caci99 · Fri Jan 22, 2010 3:47 pm

Very short sighted though! Didn`t we all dislike some years ago how powerful Microsoft became? Why should we now help yet another company trying to achieve the same.

I think we all should help OpenDNS and/or this community to find a workable solution.

What do you mean by very short sighted? We have sticked with opendns more than needed, just to give a chance.
I have wrote to the OpenDns forums, and couldn't found any solution to that.
I changed then back to ISP Dns servers, but they have their problems too, some pages would not work.

So, what am I left to do?

Sergejis, the OpenDns servers are:
208.67.222.222
208.67.220.220

WirelessRudy · Tue Jan 26, 2010 1:49 am

jcremin

I have been reading your replies to my post with interest.
I'm not going into all of them. At some point you are right, at some we disagree.

But When it comes to finger pointing I stay put on what I wrote before.
I explain why I think so and also give an in my opinion plausible explanation what might be happening here in this context.
Readers that think we are drifting off topic here again; please read on.....

Big players like Google are smart enough to make somebody else's system malfunction while they stay out of the heat.... They have all reasons to ride a competing system like OpenDNS in the wheels. They do the same with MS so why not OpenDNS? They made it clear they are looking to operate in the same field becuase there is an opportunity to make money and to know what users are doing. Very strong argument!

There are several articles in important publications where ´expert writers´ raise eyebrows on Google's intentions when it comes to their behaviours in the ´search´ and now ´dns´ world, to say the least...
Google might have broken MS's supremacy in some fields but it's no secret that they are looking to play a mayor role in the internet world. Sabotaging a honest system like OpenDNS because it interferes with what they want themselves is nothing more then a ´tool of war´.

So, to give a new, maybe more fruitful, input to the issue in relation to this last:

Since MT has dns cache only, it stores lots of info for sometimes relative long times. So if there is somewhere wrong info, like we see with google resolving, this error can stay around for some longer time too.

OpenDNS has a cache as well, so here the same argument counts. Wrong info is stored for longer times.

Most service providers probably use their own dns servers (like "bind") or just use their upstream ISP's
servers. If there is an error then new requests overwrite these the moment the error is gone.

But what about the following hypothetical scenario:

Google indeed supplies faulty resolving info regarding their own servers to requests coming from Opendns. But only now and then to stay under the ´radar´. Just some minutes a month, and everytime another OpenDNS server. And off course, very erratic. They are smart enough to achieve that.

Single client users that use OpenDNS might occasionally come across the faulty info in the OpenDNS servers cache, but as most users will do, ´try again later´ or, on advice of OpenDNS themselves, flush every dns cache along the pipeline including the OpenDNS cache and the problem is gone.... The bulk of users live with errors on their system for years without taking action. So they don't bother too much if google once every so many days or weeks give no return for a change...

Corporate, campus or cooperative users that use OpenDNS but are not using MT product might come across the error, but most consumer level routers have small cache (is overwritten relative fast by new info) or they leave their clients pointing towards their ISP's servers, or run their own servers.
Only some of them, a minority, are using MT routers with its cache...

And only a small part of these also uses OpenDNS upstream...

So, for this small group of MT users using a dns cache filled with info coming from the OpenDNS cache, they get the faulty info in theirs cache and it stays there until it is basically erased or overwritten only after a long time...

If users of these networks are pointed to this MT cache, flushing their local systems won't help. If operators now flush their gateway's cache its picking it up straight again if the error is still around in the OpenDNS cache.
Since network operators probably are not monitoring their dns cache minute by minute it might be the error stays around for a relative long time until operator starts ´sweating´ because he gets so many complaints from the clients...
Since he then don't seem to nail the problem he ends up in MT forum or else with the problem....

Not using OpenDNS any longer solves the problem thus makes it easy to point the finger to the MT-OpenDNS combination. In fact it is even true, but not necessarily because one of these two is responsible for that... Now it is no suprice users of the very active MT forum are complaining, also at OpenDNS forum, which then supports the idea it has something to do with MT.

I think most other brand routes don't have such capacity to store so much cache as the MT routers can. (Correct me here if I am mistaken. I can't really state this as a fact).
In cases were really big cache would be needed probably network operators already choose to run their own dns servers... so they don't come across the problem neither.

My conclusion:

After reading all the info from both MT experts, OpenDNS forum and articles related to googles intentions I wouldn't be surprised the issue in not MT, is not OpenDNS, but google....

In support to this idea I dare to state that if it really was a ´code´ problem or any other issue related to MT-ROS in combination with OpenDNS, it should be reproducible on command.
So far nobody seems to be able to do this. Everybody reporting the issue also reports the problem exists only now and then.... even our ´nerds´

on the forum have to wait to see if the problems pops up again....

I my humble opinion this would clear MT's products or ROS as being the problem. (Unless you argue running a dns-cache is the wrong approach as some other in this forum state.)

Anyway, I am a busy guy and since my OpenDNS ´google´ resolving crashed last weekend again I switched back to my ISP's dns servers. They work (for the time being) so I can focus on making money again....

I hope to read one day in this forum someone found the real problem and solution so I can revert back to OpenDNS.

R.

jcremin · Tue Jan 26, 2010 2:30 am

WirelessRudy,
While I don't disagree that there is a possibility Google would want to do this, I do have a few reasons to highly doubt it. Here are my observations:

1) One of the first thing my users did when Google quit working was switch to Bing, Yahoo, or Ask.com. In fact a few discovered they preferred these other search engines and reported that they now use them instead. The average user wouldn't realize it were a DNS issue, and all they know is Google is down. This makes Google look unreliable and keeps people from using their services (lost advertising revenue). Huge loss to Google.

2) The error occurs on end user's PC's cache, but from what I have seen, only those who point to an MT cache. Both PC's which have the OpenDNS servers manually entered and those who have a low end router or UBNT device which relays or caches the responses haven't seen any symptoms on my network. Because the problem flows all the way to the Windows or Mac cache, that means that even PC's who have gotten Google's DNS info directly from OpenDNS themselves by pointing the computer at their DNS servers should show this problem. Again, I only see it on computers which flow through an MT cache.

Another observation, which I don't remember I it has been stated (this thread is getting quite long): Once the problem has "infected" a computer, it won't do the DNS lookup for some reason. I can clear the MT cache, and the problem stays until the PC's cache is cleared also.

One of the reasons (other than only seeing this occur on MT caches) that I still point the finger at MT is because I run across strange bugs from time to time that don't make sense. For instance: I keep all my towers saved in Winbox, but because winbox sorts IPs based on all 3 digits of each address, I like to save them like this: 192.168.10.001, 192.168.10.002, 192.168.10.003, so that when I get up to 192.168.10.010 or 192.168.10.100, they stay in the right order. If you do that, some IP addresses don't connect properly with the leading zeros (008 is one, I think) while most of them do. Same way that when you use a domain name in some boxes in winbox it will work fine, and other times it will give an error saying "error, IP address expected". These are just goofy errors that are inconsistent and makes me wonder if it has something to do with the actual IP address of one of the OpenDNS servers. Does it add up to some magic number that MT interprets incorrectly? Does it have anything to do with the length or TTL of the returned record? I don't really have any idea, but it does linger in the back of my mind through all of this.

Daniel from OpenDNS has told me that their administrators are looking into this on their end. I will make sure he knows about the updated posts in this thread and the theory you have suggested and maybe they can shed some light.

Thanks,
Joe

accmap · Tue Jan 26, 2010 3:09 am

I haven't seen anybody suggest my thought, so here it is:

Your DNS requests are being hijacked, or someone is trying to run the trick of beating the response from the correct server and hoping they can get the correct sequence number. The 1 week TTL is a bit of a clue that something unsavory is going on, not a technical problem per se.

If I were going to try to attack OpenDNS responses, I'm sure I would start with Google as well. Once you get that working, you can steer the victim anywhere you want.

jcremin · Tue Jan 26, 2010 3:16 am

Your DNS requests are being hijacked, or someone is trying to run the trick of beating the response from the correct server and hoping they can get the correct sequence number. The 1 week TTL is a bit of a clue that something unsavory is going on, not a technical problem per se.

Ok, but I still fail to understand this: why do only my clients with MT routers end up with this problem. Somebody hijacking DNS requests wouldn't (and probably couldn't) discriminate between requests from XP, UBNT, MT, etc.

I'm trying my best to not get frustrated say "that's wrong", but instead point out the "flaws" that I see in each theory. So far, every theory (even my own) have holes in them that can't conclusively point the issue to one specific thing. The only thing in common that I can find is the MT dns cache.

Joe

fewi · Tue Jan 26, 2010 3:19 am

The 1 week could be perfectly normal, that's what Google's nameservers give as a TTL for http://www.google.com:

$ dig @ns1.google.com www.google.com | grep CNAME
www.google.com.		604800	IN	CNAME	www.l.google.com.

http://www.l.google.com then has 5 minute TTLs on its A records.

roadracer96 · Tue Jan 26, 2010 6:23 am

I haven't seen anybody suggest my thought, so here it is:

Your DNS requests are being hijacked, or someone is trying to run the trick of beating the response from the correct server and hoping they can get the correct sequence number. The 1 week TTL is a bit of a clue that something unsavory is going on, not a technical problem per se.

If I were going to try to attack OpenDNS responses, I'm sure I would start with Google as well. Once you get that working, you can steer the victim anywhere you want.

1week TTL is normal. It is a CNAME. No reason not to cache it for a long time.

missinlnk · Tue Jan 26, 2010 6:30 pm

I'm wondering if we have our answer mixed in with all of the conversation.

- Mikrotik round-robins their DNS requests.
- OpenDNS resolves Google in a way that the Mikrotik has to make two requests
- When the Mikrotik makes the second DNS request, they use the other DNS server listed, not the original one

We know this causes problems if you're using a mix of one OpenDNS and one standard DNS server. I'm wondering if that two step request to two different OpenDNS servers is also causing problems. Has anyone noticed the DNS problems if you're using OpenDNS with only one DNS server listed (Secondary is completely blank) on your Mikrotik?

Scott

Caci99 · Tue Jan 26, 2010 7:56 pm

I guess you are wrong in here Scott

It is not true that we are using two different DNS servers.
In my case I have always used DNS servers of one provider, only a couple of times
I tried to use two different servers because I wanted to see if this could have
been a solution to the problem.

So basically, the problem exist when using OpenDNS servers.

I have problems with my ISP DNS servers also (for other webpages)
so now I am testing GoogleDNS servers.

WirelessRudy · Tue Jan 26, 2010 8:58 pm

My understanding of the use of two (or more) dns servers in the client settings is that the second server is only to be used in case the first one is not giving a reply.
But if the the first server did give a reply, even if it is wrong or corrupt, this wrong info is stored in the cache.
It works for Windows dns clients this way and I mean to remember I red some of the same regarding MT's dns clients server settings too.

Apart from that, in winbox it is not even accepted to leave the second field ´blank´.

On the other hand, running torch on the WAN side shows me both dns servers are used by my main router to resolve. But in a sort of time shift... a couple of seconds only the first, then a couple of seconds the second, then the first, the second etc. etc.
So what process is behind this I don't know. Anybody?

fewi · Tue Jan 26, 2010 9:28 pm

roadracer96, yes it is true, that /ip dns can work round-robin for both configured DNS servers. When router needs to get IP/DNS for new address, which is not in the cache, round-robin is true, when both servers sent good replies for long time.

Unlike the Windows DNS client, if you configure two DNS servers under "/ip dns" RouterOS will use them round robin to resolve. Setting secondary-dns to 0.0.0.0 should disable it, I think.

roadracer96 · Wed Jan 27, 2010 6:16 am

I'm wondering if we have our answer mixed in with all of the conversation.

- Mikrotik round-robins their DNS requests.
- OpenDNS resolves Google in a way that the Mikrotik has to make two requests
- When the Mikrotik makes the second DNS request, they use the other DNS server listed, not the original one

We know this causes problems if you're using a mix of one OpenDNS and one standard DNS server. I'm wondering if that two step request to two different OpenDNS servers is also causing problems. Has anyone noticed the DNS problems if you're using OpenDNS with only one DNS server listed (Secondary is completely blank) on your Mikrotik?

Scott

Maybe this IS the problem. What happens if OpenDNS changes the IP that it resolves to but it hasnt propagated to the secondary server by the time the subsequent request is made?

Maybe a good test would be to set MT to use only one OpenDNS server. Maybe the round-robin nature of MT IS the catalyst to a problem in the OpenDNS infrastructure.

(Thinking about OpenDNS being multi-homed with only the 2 ip addresses. You are really talking to any of potentially 100s of servers all over the planet. Propagating a change to that many servers over so much physical distance doesnt happen in milliseconds)

rmichael · Wed Jan 27, 2010 7:50 am

Maybe this IS the problem. What happens if OpenDNS changes the IP that it resolves to but it hasnt propagated to the secondary server by the time the subsequent request is made?

A: at worst you'll receive stale data - but you'll still get some reply. The issue here isn't wrong resolution however, so I don't think that is the case.

missinlnk · Wed Jan 27, 2010 8:47 pm

I looked back through this thread and I noticed that everyone keeps checking www.l.google.com for an IP address. This DNS name is not used when you're running OpenDNS.

When you use OpenDNS's servers, www.google.com actually points to google.navigation.opendns.com. Can you guys running OpenDNS show what IP address is being resolved for google.navigation.opendns.com when you have problems?

Scott

BobcatGuy · Thu Jan 28, 2010 5:48 am

I have had a few issues at wierd times with Internet connectivity, where interal lan works fine, and external doesn't. I always thought it was the MT router where NAT failed, becuase a VoIP phone that I had on a External bridge ( Bridged 3 ports on MT) to allow the Voip internet IP instead of behind firewall.

I always thought the NAT rules failed, but today it happened again, Its been up for 34 days, roughly the same time before, of 31 days, and it failed.

I was so tempted to reboot like I did before which seemed to fix the isssue, but I waited, tried a bunch of things.

I used my black berry to lookup a free dns, happed to be 8.8.8.8 which is google's free dns. I put that as the seconday DNS on the MT, and what do you know, it worked.

I tried removing it, and no dns look ups were working, I tried to ping an IP, that works, switched back to 8.8.8.8 as secondary dns on MT again, everything works.

Obviously my ISP's DNS servers were not working, again, but they kep telling me it was a problem with my router, becuase they knew I was on the VoIP phone, so internet connectivity was there, at the time it made sense... I felt like an idiot,so i assumed it was the MT router. Upon deeper searching, I find out that my Voip Adaptor from the telco that runs it uses IP only and no name lookups, thats why it kept working.

So when in doubt, use someone elses DNS server. Its not often that this has happened, but when it does it effects alot of systems for me, even down to the Blackberry Browser on the Blakberry's which use the BES server, which is Behind the MT router.

At the end of the day I truested the MT router more then the ISP, and I was right, it seemed to be a wierd problem to just all of the sudden stop the NAT translation.

So now I have Primary as ISP dns, and Secondary as the 8.8.8.8

DeVerm · Thu Feb 11, 2010 4:55 am

Well, here goes for my first post here

I admit that I retired in 2002 but I used to be one of those DNS guru's (I started first commercial ISP in Holland) and actually worked on the RFC's with RIPE and IETF. So, I'm a bit rusty but I know this stuff. I just checked and see my servers are still there even though the name of the company changed 8 years ago to EasyNet

Even the IP addresses are still the same... they are authoritive for so many domains that they can't change them ha! (ns.wirehub.net and note that the secondary ns4.wirehub.net is actually on an OpenDNS server

The problem is very simple. OpenDNS is not authoritive for the google.com domain. You can ask the authoritive nameservers for the google.com domain at any root-server and it'll present you with some Google nameservers. If you understand that, the rest is easy: only authoritive servers for google.com are allowed to enter A or CNAME records for that domain. All other servers must ask a google.com nameserver for this info and persent that info to the client. The Google servers tell the non-authoritive server how long this info is valid and that server can decide to cache it for that period.

Now, OpenDNS decided to enter a CNAME record for www.google.com which isn't allowed and which results in everyone using OpenDNS for DNS resolution to receive false information. If things didn't change since 2002, Google could report this and have this practice stopped.

Even though I am sympathetic to the OpenDNS effort, they are breaking the RFC's and thus the functionality of DNS.... as you all found out. Their practice is the same as when let's say UUnet puts a CNAME in their nameservers for www.mikrotik.com pointing to microtik.uu.net where they show a page that looks just like the mikrotik page. Now, everyone using ns.uu.net gets false resolution and another website. Not funny.

Also, I see that most here have little to no clues about DNS, let alone bind. If you are serious about this, you should study this because it is one of the basic building blocks of the Internet. I can recommend the O'Reileigh book on DNS and BIND if that still exists.
If you study this, you will find that indeed you should run your own DNS server. How would you ever be able to create a better Internet service by limiting your best shot to the level (which you laugh at) of another ISP... you will always come 2nd or later, i.e. your plan is flawed. The key point is this: never never ever query a DNS server that is not authoritive for the query except for the official root servers... which ___will___ direct you to the official authoritive nameservers for your query. Or in other words: your server(s) should be the only ones doing recursive lookup's for you and your customers.
Also, you can run your own primary dns for your domain now and you can even request your upstream ISP to delegate DNS for your assigned address space to your server so that you control your own reverse-lookup results. This is key, the first thing we did for any network change or expansion.
And it's so easy, find the current config file for the rootservers and start bind.

The DNS load balancing that MT does is fine because every server you define in your config should answer with the same data.

Hope this clears things up a bit.

Now, if you still want to find out what is really happening the "script" y'all use and which output is posted here over and over is not gonna help. I counted one person noticing this: OpenDNS doesn't reply with CNAME www.l.google.com so it's very silly to try to find that in your cache. If this isn't enough clue for modifying the script: O'Reileigh, "DNS and BIND".

I know that I'm probably the oldest geek on this forum but I bet that the RFC's didn't change much if at all and that the info above is correct today. Take it and use it to your advantage.

cheers,
Nick.

rmichael · Thu Feb 11, 2010 11:32 am

Now, OpenDNS decided to enter a CNAME record for www.google.com which isn't allowed and which results in everyone using OpenDNS for DNS resolution to receive false information. If things didn't change since 2002, Google could report this and have this practice stopped.

I guess times are changing and RFC are just that. One might argue that sending false information, as you put it, is what OpenDNS does for business. But that's not an a problem. The issue is that on occasion MT DNS does not resolve www.google.com at all (not even to "false" OpenDNS servers).

WirelessRudy · Thu Feb 11, 2010 2:55 pm

Well, here goes for my first post here

I admit that I retired in 2002 but I used to be one of those DNS guru's (I started first commercial ISP in Holland) and actually worked on the RFC's with RIPE and IETF. ............

..

cheers,
Nick.

Wow, nice info. First of all I am delighted to find another Dutch on this forum. I have the feeling there aren't a lot. (En wanneer dat wel het geval is, zend me een mail naar info@marucom.es. I wil wel eens weten met wie ik dan van doen heb!) Great! Any Dutch emigrated and working with MT? Even better! "Laat je horen!"

I swapped to my IPS's dns servers three weeks ago and since then everything fine. But yes, I need to build Bind. I'd only wish I had the time to do it (and the knowledge and hardware.....)
What machine's suggested to use for Bind? And how can that machine made redundant (with another machine?) DNS is definitely not my field, nor is setting up Linux on PC's. I tried that once and ended up throwing the PC through the room out of sheer frustration!

Nick; "als je interesse hebt, ik woon in de Costa Blanca, misschien interesse om me met het één en ander uit de brand te helpen in ruil voor een spotgoedkoop vakantie adress?

DeVerm · Thu Feb 11, 2010 3:40 pm

I guess times are changing and RFC are just that. One might argue that sending false information, as you put it, is what OpenDNS does for business. But that's not an a problem. The issue is that on occasion MT DNS does not resolve http://www.google.com at all (not even to "false" OpenDNS servers).

So, that's how it is today...?? Someone, OpenDNS in this case, breaks the RFC which results in a problem on a MT device/software that does conform to the RFC and now you say that you don't care and demand that MT takes care of the problem? Do you know how many DNS servers there are on the Internet? Without them all conforming to the RFC's and guidelines, there would be no Internet, there is no way that it'll all work together that way. Standardization is not a luxury or frill; without it, it'll come tumbling down.

I think you would think differently when some DNS server administrators start messing with your domain, like pointing http://www.yourdomain.com to their own server instead of yours!! You would go mad and take them to court probably

But anyway, MT already stated that they think MT works good and only relays/caches incorrect data that is received from the DNS server that you choose to use. If you do not agree and want to find the problem, you should modify that "script" so see what you actually have in cache (like A-type records for the fake google servers that OpenDNS announces for http://www.google.com). You could also query OpenDNS with tool to debug this, like dig or nslookup.

This is what Scott (missinlnk) wrote and he is right:

I looked back through this thread and I noticed that everyone keeps checking http://www.l.google.com for an IP address. This DNS name is not used when you're running OpenDNS.

When you use OpenDNS's servers, http://www.google.com actually points to google.navigation.opendns.com. Can you guys running OpenDNS show what IP address is being resolved for google.navigation.opendns.com when you have problems?

Now, just to show you that stuff is broken, you should query the SOA records (Start Of Authority):

#nslookup

> server 208.67.222.222
Default Server:  resolver1.opendns.com
Address:  208.67.222.222

> set q=SOA

> www.google.com.
Server:  resolver1.opendns.com
Address:  208.67.222.222

Non-authoritative answer:
www.google.com  canonical name = google.navigation.opendns.com
>
> server 8.8.8.8
Default Server:  google-public-dns-a.google.com
Address:  8.8.8.8

> www.google.com.
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Non-authoritative answer:
www.google.com  canonical name = www.l.google.com
www.l.google.com        canonical name = www-tmmdi.l.google.com

l.google.com
        primary name server = ns4.google.com
        responsible mail addr = dns-admin.google.com
        serial  = 1406548
        refresh = 900 (15 mins)
        retry   = 900 (15 mins)
        expire  = 1800 (30 mins)
        default TTL = 60 (1 min)

So, this shows that OpenDNS fails to show the SOA record, no serial number, no nothing. The SOA record also shows who is responsible (authoritive) for this domain: dns-admin[at]google.com so you could send an Email and ask!

The first thing you must do when there is DNS trouble is ask your server the SOA and look at the serial number. Next you find the primary server for that domain and ask it for the SOA and compare the two. It must be the same. So, we just found that ns4.google.com is the primary, let's ask it:

> server ns4.google.com.
Default Server:  ns4.google.com
Address:  216.239.38.10

> www.google.com.
Server:  ns4.google.com
Address:  216.239.38.10

www.google.com  canonical name = www.l.google.com
l.google.com
        primary name server = ns3.google.com
        responsible mail addr = dns-admin.google.com
        serial  = 1406549
        refresh = 900 (15 mins)
        retry   = 900 (15 mins)
        expire  = 1800 (30 mins)
        default TTL = 60 (1 min)
google.com      nameserver = ns1.google.com
google.com      nameserver = ns3.google.com
google.com      nameserver = ns2.google.com
google.com      nameserver = ns4.google.com
ns1.google.com  internet address = 216.239.32.10
ns3.google.com  internet address = 216.239.36.10
ns2.google.com  internet address = 216.239.34.10
ns4.google.com  internet address = 216.239.38.10
>

Now, here we learn that there are multiple primary servers (big responsibility to keep those in sync!!) and that the serial has increased already. I quickly checked 8.8.8.8 again and they have the new record too. If TTL is messed with, this wouldn't work either.

If you start testing yourself you should understand that I was lazy with these tests because the SOA isn't for a hostname but for a domain. I should have queried for l.google.com. instead but these servers understood and forgave my lazyness... others might not so query for the (sub)domain if there's trouble and don't forget the last dot.

I find the whole thing suspicious when I see something like "resolver1.opendns.com" because the resolver is the client and yet here we have a server that calls itself resolver.

Also, some have reported that they have this problem when they use other DNS servers too! Now, how do you know what the admin of those servers does with the config? for all I know, he/she is caching wrong data too or forces them to use OpenDNS or whatever.... everything is possible!

So, next time it happens, see what's in your cache for google.navigation.opendns.com and there should be type-A records. Then query your cache with dig or nslookup for it etc. etc. I would also check by querying from a linux box because the windows resolver is, like anything windoze, suspect too.

ciao!
Nick.

DeVerm · Thu Feb 11, 2010 4:45 pm

Wow, nice info. First of all I am delighted to find another Dutch on this forum. I have the feeling there aren't a lot. (En wanneer dat wel het geval is, zend me een mail naar info@marucom.es. I wil wel eens weten met wie ik dan van doen heb!) Great! Any Dutch emigrated and working with MT? Even better! "Laat je horen!"

I swapped to my IPS's dns servers three weeks ago and since then everything fine. But yes, I need to build Bind. I'd only wish I had the time to do it (and the knowledge and hardware.....)
What machine's suggested to use for Bind? And how can that machine made redundant (with another machine?) DNS is definitely not my field, nor is setting up Linux on PC's. I tried that once and ended up throwing the PC through the room out of sheer frustration!

Nick; "als je interesse hebt, ik woon in de Costa Blanca, misschien interesse om me met het één en ander uit de brand te helpen in ruil voor een spotgoedkoop vakantie adress?

Hi Rudy,

This is me: http://www.google.com/search?hl=en&rlz= ... f&aqi=&oq=

Bind will use very little resources and any reliable machine (or even routerboard when you have enough RAM) will run just fine. Bind is the only one you want and it's in every distribution of Linux or freeBSD or whatever unix flavour you like. I always used freeBSD because I come from the Berkeley Unix camp

But bind is the same on Linux.
The server won't crash unless there's hardware trouble. For redundancy, you can create a 2nd one with the same config. Only for domains you enter authoritive records (your domains or domains of your customers) you do this different on the back-up machine: you make it a secondary for those domains. This is very easy. You should also have a secondary elsewhere... we always asked a friendly ISP in other parts of the world to run secondary for us, until our network reached those parts of the world and we installed our own DNS server there. But all that is only for authoritive data, not for caching.

You need to find a Linux geek to help you with this. There's plenty!

About vacation: I am retired, live on a sailboat and sail around the world!

We're in Panama now, on the edge of sea and jungle and there's 3 wireless links between me and civilization. Wifi is all we have, wouldn't really work dragging DSL cables behind us

But I think we'll sail to Spain one day so who knows, we might meet you there!

cheers,
Nick.

missinlnk · Thu Feb 11, 2010 6:33 pm

Nick, kinda off topic, but just wanted to say thank you for jumping in here. If you can bring that kind of background and knowledge into any of the other discussions on this forum we'd love to see you post more! We'll even be nice and overlook the periods of silence while you're sailing to a new port.

roadracer96 · Thu Feb 11, 2010 8:31 pm

Now, OpenDNS decided to enter a CNAME record for http://www.google.com which isn't allowed and which results in everyone using OpenDNS for DNS resolution to receive false information. If things didn't change since 2002, Google could report this and have this practice stopped.
I guess times are changing and RFC are just that. One might argue that sending false information, as you put it, is what OpenDNS does for business. But that's not an a problem. The issue is that on occasion MT DNS does not resolve http://www.google.com at all (not even to "false" OpenDNS servers).

No. The issue is, OpenDNS doesnt return consistent results and MT is following what they say.

This isnt an MT problem, but an OpenDNS problem.

In response to BIND being reliable...... Check it....

[root@ns1 ~]# uptime
13:29:49 up 582 days, 3:33, 1 user, load average: 0.18, 0.09, 0.02
[root@ns1 ~]#
[root@ns2 ~]# uptime
13:30:15 up 582 days, 2:55, 1 user, load average: 0.06, 0.06, 0.01
[root@ns2 ~]#

I installed those servers... Guess... 582 days ago.

Kernel is out of date, but considering the only thing that can get through to them from the outside is port 53 udp and icmp 8, im not worried about it.

DeVerm · Thu Feb 11, 2010 9:15 pm

Yes, nameserver stability is great. Our freeBSD nameservers set a record of > 800 days (all of them). We took them down for upgrading to a newer version of freeBSD or they would still be running I think.

But... does anyone here have bind running on a routerboard?

cheers,
Nick.

WirelessRudy · Thu Feb 11, 2010 11:53 pm

That was my next question going to be: How to install Bind on RB? I´v got two rb600's doing nothing. Maybe a nice job for them and together will all my other stuff can run on 12-24V and battery with Solar.

Anybody can shine some light on that?

roadracer96 · Fri Feb 12, 2010 1:08 am

I would say use metarouter and an openwrt image, but that wont fly on PPC. Do you have a couple old pentium 2/3 laying around with a 4gig hd? Install CentOS and install the caching-nameserver package. Done.

jcremin · Fri Feb 12, 2010 4:25 am

Also, some have reported that they have this problem when they use other DNS servers too! Now, how do you know what the admin of those servers does with the config?

When I first started having this problem (when I originally started this thread) I was using my own DNS server. I know for a fact that I didn't have a problem with the config.... Simple DNS caching server which queried the root servers, not any other ISP's server. Because of that one simple issue, which is what caused me to try OpenDNS in the first place, it made me strongly believe that the problem is with MT.

While I was quite ambitious for trying to find a solution, I don't have the time to test this much farther. I did switch back to OpenDNS for a few days, but then problems started cropping up again and this is not worth losing customers over. I switched to using Google's Public DNS servers, and haven't had a problem since. So I'm just sticking with what works unless I get more time to test.

DeVerm · Fri Feb 12, 2010 7:23 am

When I first started having this problem (when I originally started this thread) I was using my own DNS server. I know for a fact that I didn't have a problem with the config.... Simple DNS caching server which queried the root servers, not any other ISP's server. Because of that one simple issue, which is what caused me to try OpenDNS in the first place, it made me strongly believe that the problem is with MT.

While I was quite ambitious for trying to find a solution, I don't have the time to test this much farther. I did switch back to OpenDNS for a few days, but then problems started cropping up again and this is not worth losing customers over. I switched to using Google's Public DNS servers, and haven't had a problem since. So I'm just sticking with what works unless I get more time to test.

Yes, I've been there. What software did you use for your nameserver? Remember that if it ain't "bind" it is suspect!

On the other hand, when you try your own server again, you may never get the problem again.... You never know what happened (cache is easy to go dirty when you're playing with the server/config). One thing I am sure of: if you use bind and give it a config with just the OFFICIAL root nameservers, you will not have trouble!

cheers,
Nick.

jcremin · Fri Feb 12, 2010 7:31 am

What software did you use for your nameserver? Remember that if it ain't "bind" it is suspect!

It is Windows 2003 DNS server. But I'm sure that since it isn't bind/linux it doesn't make any difference and I was screwed from the beginning.

DeVerm · Fri Feb 12, 2010 8:00 am

You use a windows nameserver and find it strange all goes fubar??!!! pls, m$ is still bitter that the Internet stayed with DNS & bind instead of their crap so stay away from that. beat & burn it!

ciao!
Nick.

jcremin · Fri Feb 12, 2010 8:31 am

My Windows server (which is also my web hosting/email server) has been running solid for years (I was close to your 800 days on my server as well when I finally rebooted for a ton of updates) and the only thing that I have ever had a problem with was MT's DNS client. It is pretty apparent that you are a Microsoft hater and I have to wonder if you've ever used a MS DNS server since you're obviously married to bind. I'm not trying to call you on this, but you're not really doing much to help troubleshoot. All you've really done is point fingers and use a ton of words to say that Bind=God.

I'm not trying to attack you, but as soon as anyone (Apple/Mac fanboys are a good example) says that you should only ever use one product regardless of what their needs are, I instantly question their credibility, as it usually means that is the only thing they have ever used and don't know any better.

I've been in some discussions on the dslreports.com forums where a few people were saying that the only way you could run a high end PPPoE server was with a $10k Cisco router, when there are obviously plenty of ways to do it just as well with Mikrotik for a fraction of the price.

Back to the original topic... As rmichael stated: "The issue is that on occasion MT DNS does not resolve http://www.google.com at all (not even to "false" OpenDNS servers)." It appears MT doesn't even send the request to the DNS servers.

roadracer96 · Fri Feb 12, 2010 8:41 am

What software did you use for your nameserver? Remember that if it ain't "bind" it is suspect!
It is Windows 2003 DNS server. But I'm sure that since it isn't bind/linux it doesn't make any difference and I was screwed from the beginning.

Kind of. BIND has pretty much unlimited flexibility. MS DNS has almost no flexibility.

changeip · Fri Feb 12, 2010 8:46 am

appears MY doesn't even send the request to the DNS servers.

Thats exactly why it would be nice if someone could get a pcap of it. I will help diagnose if someone can get it captured. This ENTIRE uncertainty about OpenDNS being the cause is ONLY becuase they give back answers other than what google wants. It could be something totally different and it just so happens google is the busiest and most noticed. How much mail isn't flowing because of a bug like this?

Let's not argue and bicker about whats better, let's just solve the problem in RouterOS by figuring out whats broken. The half implemented DNS code in MT has always been lacking and it's where my finger is pointed.

jcremin · Fri Feb 12, 2010 8:47 am

What software did you use for your nameserver? Remember that if it ain't "bind" it is suspect!
It is Windows 2003 DNS server. But I'm sure that since it isn't bind/linux it doesn't make any difference and I was screwed from the beginning.
Kind of. BIND has pretty much unlimited flexibility. MS DNS has almost no flexibility.

Just for clarification, I was being sarcastic when I posted that. I didn't need flexibility. All I needed was a simple DNS server that could resolve addresses for me. In my case, the less to configure, the better. MS DNS has worked, and continues to work fine for me... Just not with any MT boxes pointing to it as it's DNS server.

jcremin · Fri Feb 12, 2010 8:50 am

appears MT doesn't even send the request to the DNS servers.
Thats exactly why it would be nice if someone could get a pcap of it. I will help diagnose if someone can get it captured. This ENTIRE uncertainty about OpenDNS being the cause is ONLY becuase they give back answers other than what google wants. It could be something totally different and it just so happens google is the busiest and most noticed. How much mail isn't flowing because of a bug like this?

Let's not argue and bicker about whats better, let's just solve the problem in RouterOS by figuring out whats broken. The half implemented DNS code in MT has always been lacking and it's where my finger is pointed.

AGREED!!!! As I mentioned earlier, I have run short on time to troubleshoot right now, but I will do my best to find time to see if I am able to get a wireshark capture while the problem is happening. Hopefully someone else get's a chance to capture it too (or first) as the sooner the better

taglio · Fri Feb 12, 2010 12:08 pm

FreeBSD rules!

[taglio@tsunami]/home/taglio(101): uptime
11:11AM  up 821 days, 15:44, 1 user, load averages: 0.02, 0.01, 0.00
[taglio@tsunami]/home/taglio(102):

Fri Feb 12, 2010 12:20 pm

RouterOS is no different in this regard. See that topic about RouterOS uptime. So many posts with 400 days + uptime

WirelessRudy · Fri Feb 12, 2010 4:16 pm

RouterOS is no different in this regard. See that topic about RouterOS uptime. So many posts with 400 days + uptime

Well Normis,

Why not make a special dns package for ROS that implements a full dns server that can do the same as bind on a RB and can be setup by us ´dummy's´.
It would solve some of the cache problems and would make a nice selling tool for your stuff!

Or am I now talking weird here...?

Fri Feb 12, 2010 4:19 pm

RouterOS is small and fast as a router. If we start adding DNS server, Webserver, Mailserver, Antivirus ... it will become something else.

Sob · Fri Feb 12, 2010 6:14 pm

If we start adding DNS server, Webserver, Mailserver, Antivirus ... it will become something else.

It would become something even better and cooler than it's now.

I mean, web and mail is probably too much, but dns is pretty close to low level network stuff. I wish for real recursive caching resolver in MikroTik for a long time. Just some nice optional package for those who need it. And in my wildest dreams, I could sometimes use even authoritative server. But I guess there would be much lower demand for this than for resolver.

SurferTim · Fri Feb 12, 2010 6:17 pm

RouterOS is exactly what its name implies. This is a router, not a server. Or am I wrong?

taglio · Fri Feb 12, 2010 6:30 pm

Correct! routeros is a router. Or you see some IOS OS with a bind? To run bind you have to use a dedicated hardware [running other SO like linux or FreeBSD {that is better}].

my 2 cents.

Sob · Fri Feb 12, 2010 6:46 pm

If RouterOS can contain web proxy (what does it have to do with routing?) in default install, then optional dns resolver wouldn't hurt. Forget about authoritative server, I'll keep that in my wild dreams. But recursive resolver does make sense. Assuming it would be reasonably configurable, it would be solution for things like http://forum.mikrotik.com/viewtopic.php?f=1&t=25569.

SurferTim · Fri Feb 12, 2010 6:52 pm

@Sob: It does that already. Entries in "/ip dns static" will override any external dns. I will leave it up to you to figure out the rest of that.

Sob · Fri Feb 12, 2010 7:26 pm

@SurferTim: Not exactly. It's not about overriding address for one host. It's about telling RoS that some complete domain is handled by different nameservers. There can be any number of subdomains, different nameservers for them, thousands of records. You can't do that in current RoS and it's impossible to do without recursive resolver. The clever hack in the other thread works only when one server is involved. But it's OT in this thread, so any further discussion about this should go to the other thread.

SurferTim · Fri Feb 12, 2010 7:35 pm

You are correct. This is a server issue, not a router issue.

DeVerm · Fri Feb 12, 2010 8:20 pm

For those who want to defend running a nameserver on windows: a nameserver is as alien to microsoft as Alf to Earth: there will be trouble. If you really really can't admin a unix flavour box and have to use windows, I understand that bind has been ported to windows.

It has nothing to do with me hating m$... I hate no one and nothing, I have no reason to. Just ask yourself: what do the root nameservers run? what do all the TLD nameservers for every ccTLD in the world run? answer: bind. They run it on hardware from every manufacturer and every unix/linux flavor that exists... but it is bind. Now, look at all the big ISP's: they use..... bind. So, if you want to be 100% compatible with the world's DNS system, what do you use? microsoft?

About uptime, reliability: a long long time ago freeBSD was more stable than Linux. Today, they are equal. Sure RouterOS fits in the list of stable platforms because as long as the hardware is good it is stable because the kernel is Linux.

I would love to see bind- and (micro/mini-what is it)httpd packages for RouterOS. I would buy a routerboard with fast cpu and plenty memory and it would run all that I need on our boat. Hotspot login page with links to the local httpd where other's can find our website, photo's etc. All on solid state disk. It should be trivial to make those packages.

Mailserver? sure, why not, a fast board will do that for the little traffic there is when people want this. Antivirus: I can't see that work

For the developers I can understand they have their eyes on creating the meanest & fastest router solution in the market; they just want to beat Cisco & Juniper. This is good and it's what brought RouterOS to what it is today. But it wouldn't hurt to ask a marketing-type about this and check the current customer base. I think that a big percentage of the customers are not the big ISP but small networks that could use these features and will most likely convince them to upgrade to the newest fastest routerboard yet

I would buy!

cheers,
Nick.

Sob · Fri Feb 12, 2010 8:28 pm

@SurferTim: And are you sure that MikroTik is only router? Web proxy, already mentioned, has nothing to do with routing. Similar SOCKS proxy, although a little closer to routing, it's not really routing. NTP server, no connection to routing at all. The whole virtualization thing, yes, you can run virtual routers inside it, but by itself, it's not routing. After all, the whole dns has nothing to do with routing. And still MikroTik does all that.
What I'm saying is not to trash current dns cache implementation and force users to use recursive resolver even if they don't want it. Do it the same way as with NTP. Basic client is part of system package. If it's not enough for someone, then there's separate NTP package that can do more.
I think I said all I could. It's up to MT guys to decide what's best for their product. Clearly I'm not the only one who thinks that some more dns stuff would be good, as I wasn't the first one who said that wants dns server on MikroTik.

jcremin · Fri Feb 12, 2010 8:59 pm

RouterOS is small and fast as a router. If we start adding DNS server, Webserver, Mailserver, Antivirus ... it will become something else.

I happen to agree with the others here when they say that a full-blown DNS implementation would be practical. What about pppoe SERVER.. Web Proxy, Hotspot, Radius, etc. All of these are beyond the scope of a basic router and PPPoE alone is way more CPU intensive than DNS.

This would give us an awesome DNS server that can run on a "server" which requires very little power and has no hard drive, which should provide to be a rock solid platform, offering us one more incentive to buy Mikrotik routerboards.

jcremin · Fri Feb 12, 2010 9:03 pm

Do it the same way as with NTP. Basic client is part of system package. If it's not enough for someone, then there's separate NTP package that can do more.

Yep, like the user-manager package. Not installed by default, but if someone wants to use it, just install it.

rmichael · Fri Feb 12, 2010 9:32 pm

So, that's how it is today...?? Someone, OpenDNS in this case, breaks the RFC which results in a problem on a MT device/software that does conform to the RFC and now you say that you don't care and demand that MT takes care of the problem? Do you know how many DNS servers there are on the Internet? Without them all conforming to the RFC's and guidelines, there would be no Internet, there is no way that it'll all work together that way. Standardization is not a luxury or frill; without it, it'll come tumbling down.

I think you would think differently when some DNS server administrators start messing with your domain, like pointing http://www.yourdomain.com to their own server instead of yours!! You would go mad and take them to court probably

As some suggested - if you don't trust quality of responses from your provider run your own.

So, this shows that OpenDNS fails to show the SOA record, no serial number, no nothing. The SOA record also shows who is responsible (authoritive) for this domain: dns-admin[at]google.com so you could send an Email and ask!

The first thing you must do when there is DNS trouble is ask your server the SOA and look at the serial number. Next you find the primary server for that domain and ask it for the SOA and compare the two. It must be the same. So, we just found that ns4.google.com is the primary, let's ask it:
Code: Select all
> server ns4.google.com.
Default Server:  ns4.google.com
Address:  216.239.38.10

> www.google.com.
Server:  ns4.google.com
Address:  216.239.38.10

www.google.com  canonical name = www.l.google.com
l.google.com
        primary name server = ns3.google.com
        responsible mail addr = dns-admin.google.com
        serial  = 1406549
        refresh = 900 (15 mins)
        retry   = 900 (15 mins)
        expire  = 1800 (30 mins)
        default TTL = 60 (1 min)
google.com      nameserver = ns1.google.com
google.com      nameserver = ns3.google.com
google.com      nameserver = ns2.google.com
google.com      nameserver = ns4.google.com
ns1.google.com  internet address = 216.239.32.10
ns3.google.com  internet address = 216.239.36.10
ns2.google.com  internet address = 216.239.34.10
ns4.google.com  internet address = 216.239.38.10
>
Now, here we learn that there are multiple primary servers (big responsibility to keep those in sync!!) and that the serial has increased already. I quickly checked 8.8.8.8 again and they have the new record too. If TTL is messed with, this wouldn't work either.

If you start testing yourself you should understand that I was lazy with these tests because the SOA isn't for a hostname but for a domain. I should have queried for l.google.com. instead but these servers understood and forgave my lazyness... others might not so query for the (sub)domain if there's trouble and don't forget the last dot.

I find the whole thing suspicious when I see something like "resolver1.opendns.com" because the resolver is the client and yet here we have a server that calls itself resolver.

Also, some have reported that they have this problem when they use other DNS servers too! Now, how do you know what the admin of those servers does with the config? for all I know, he/she is caching wrong data too or forces them to use OpenDNS or whatever.... everything is possible!

So, next time it happens, see what's in your cache for google.navigation.opendns.com and there should be type-A records. Then query your cache with dig or nslookup for it etc. etc. I would also check by querying from a linux box because the windows resolver is, like anything windoze, suspect too.

ciao!
Nick.

Your theory is missing a reason why it only happens sporadically and why MT stops resolving http://www.google.com all together. If someone can capture traffic as changeip suggests we might get an answer otherwise number of possible points of failure is too many.

So, next time it happens, see what's in your cache for google.navigation.opendns.com and there should be type-A records. Then query your cache with dig or nslookup for it etc. etc.

I agree with you here 100%

jcremin · Fri Feb 12, 2010 9:34 pm

For those who want to defend running a nameserver on windows: a nameserver is as alien to microsoft as Alf to Earth: there will be trouble.

You're making a generalization. It's like saying that just because Ford was established in 1903, you will definitely have problems driving a GM vehicle since they were founded in 1908. Microsoft didn't always have a web server either, but the newer versions of IIS are rock solid and work great. Same with DNS. As long as they follow the RFC's it shouldn't matter what software you are running. Post some real proof that Microsoft's DNS totally breaks the internet and I will listen, but until then, I'm just going to ignore any scare-tactic generalizations.

Just ask yourself: what do the root nameservers run? what do all the TLD nameservers for every ccTLD in the world run? answer: bind.

They probably use bind because that's what they started with and it works. I'll be the first the admit that Windows probably won't scale to nearly the capacity that bind can, but for me, I don't need anything to handle thousands of site, so I'm sticking with what works.

About uptime, reliability: a long long time ago freeBSD was more stable than Linux. Today, they are equal. Sure RouterOS fits in the list of stable platforms because as long as the hardware is good it is stable because the kernel is Linux.

Yes, a solid Kernel and good drivers makes all the difference in the world. I know you keep giving me the 3rd degree about running Windows, but this isn't windows ME I'm running. It used to be less stable, but like you admit, the times change. My server has been solid for 4 years and hasn't crashed once. RouterOS is solid too, and if it ran bind, I would use it. I'm not against Bind or Linux. I just can't justify setting up a new box just to handle my minimal DNS traffic when what I have works fine.

I don't plan to drag on this MS vs Bind debate anymore... Neither is 100% right in every situation... You know bind and that's what you're comfortable with... I know MS and that's what I'm comfortable with. While it may be a good discussion, it is way off topic. Cheers!

DeVerm · Fri Feb 12, 2010 10:11 pm

@jcremin: I can generally agree with our post, but don't you find it curious that you are the only one in the world that had this problem without using any OpenDNS servers?? Think!

And like I said: there's bind for windows too.

cheers,
Nick.

WirelessRudy · Sat Feb 13, 2010 2:36 am

RouterOS is small and fast as a router. If we start adding DNS server, Webserver, Mailserver, Antivirus ... it will become something else.

Normis,

As you must have noticed on the reactions after this reply of yours I think the point of adding 100% compatible dns server functionality in a separate ROS package is a definite pro. If you want it you use it, if you want to build the fastest router possible, you can. Let the customer decide. Probably most of the users use only half of the supplied packages anyway. I'll bet the some of the ´high-end´ routing packages and options are only used by a relative small group of users.
Some already mentioned, MT customers base is probably more to be found in the field of small to medium sized providers that would surely appreciate an add-on package performing dns.
We all want fast networks, fast reliable dns resolving is just one of the tools to achieve that. Like web proxy can, like MPLS can etc. (And indeed, how does NTP make routers faster? And what is the relation with routing?)

Routerboards are also the perfect platform to run small add-on packages. They are reliable, have long lifetime, are small, are relative cheap, consume little power and are almost perfect to use in low energy (solar, wind) environments. They are perfect even on a sailing boat drifting the Caribbean!

I have to run a PC now to do mail, dns and web server for my little network. So this machine has to run 24/7 and consumes each hour probably as much power as all my other 8 or so routerboards here in my masts! I have to buy an expensive UPS to make it survive small power cuts while all my boards run at least for 24 hours on one 200A Gel battery costing half the price of the UPS.

These are all factors that especially small and medium sized business are looking for. The moment I could do everything I need to do as a small WISP on rb's this PC-server is the first that will retire....

MT aimed with their RB750(G) to the interesting SOHO market. Well, with dns server (and yes indeed small mail and webserver as extra would make it even more interesting) they will definitely be appreciated by the same group and everybody that wants to run a small network or to become a small and growing provider.

Fine tuning and keeping up with the competition to deliver the ´best price/quality router' with the latest protocols etc. must cost a lot of energy of development in Latvia. Steer a bit of this energy into more widely needed and used functionality like dns, mail, 802.11n (where is the first MT deployment of PtMP setups with 802.11n? So far I can see MT starts to miss the boat here...) etc. and your sales will probably go up faster then by having the high-end users satisfied with latest developments in routing. These big boys having the need to run routers that do the best possible in networking probably stick with their contractual suppliers of Cisco stuff anyway...
It's not Ferrari that satisfies most drivers, its Toyota or Volkswagen!

So MT, stick your heads together and do us (and yourselves) a favour and listen to your wide base of users and make something we want. We want a vehicle that brings us to where we want to go. We are not waiting for a F1 race car, we need a Ford, Volkswagen, Toyota or whatever......

rmichael · Sat Feb 13, 2010 4:50 am

Let's not forget that most of the routerboard's storage is based on flash memory with limited number of writes. With the emergence of flash drives (low power high reliability) and fast MIPS cpus it's perhaps time for MT to rethink their strategy and allow for more write happy applications when using an external drive?

I can see MT competing well in situations where I need to use cisco (btw, there are cheap cisco routers out there now too) and additional PC where I can do it with one or two MT appliances (winbox is a plus as well). Yes I do consider MT not a router but an appliance.

As to DNS, here's a link where adding static dns entries have revealed wakness in current implementation:

http://forum.mikrotik.com/viewtopic.php ... +static+ad

Sob · Sat Feb 13, 2010 4:52 am

@WirelessRudy: Not that I enjoy shooting in own lines, but don't get too carried away.

Web server is already in MT, only you can't use it for yourself. But even if you could, you'd quickly realize that it's just not it without php, mysql and the list goes on. It's getting a little too complicated to distribute all this, keeping it up to date, etc.. Similar with mail server. It really calls for spam filter and even that antivirus. Also on RB, it would be quick and reliable way to kill the flash memory, because it writes to disk a lot. And it's not really so much disk space on RB anyway. And what applies to both of them, there are tons of different settings that MT would have to make available if those servers should not be only very limited.
Dns on the other hand, does not depend on anything else. It's simple and there are only few needed settings. It doesn't need to write to disk at all (talking about recursive resolver). All it needs is some memory and little cpu time, no problem for almost any RB. Once it gets going, there isn't really much to update, it just works. Many years ago, when I didn't know yet about MT, I set up company's inet gw on Linux and put DJB's dnscache on it. No one touched it since and it'll probably work until the hw goes to Silicon Heaven or some major "once in twenty years" change happens in dns (DNSSEC comes to mind).

DeVerm · Sat Feb 13, 2010 5:26 am

Yes, DNScache is rock stable, don't hesitate to use it.

On the rest: they have httpd for other routers and it's doing just fine. Even a couple of pages with stuff to download etc. is nice. no server-side stuff, no Apache.

Disk: what does some USB flash cost? I have thrown these out because they were too small in capacity for today but never because they wear out! I nice tight little SMTP server with a POP3 daemon, some way to rewrite a couple of headers: plenty choice in open source! Blacklisting/anti-virus can be done on the next big SMTP server or on the clients. It is not very productive to go from no functionality to the world's most advanced SMTP server... such statements are only brought forward in an attempt to kill the initiative.

cheers,
Nick.

WirelessRudy · Sat Feb 13, 2010 7:15 pm

hmm, maybe I was getting carried away.
dns, that's really what we want. The other servers is not really important for small WISP's anyway. Plenty of third party soluciotions available on the web so why should I want to do that myself.
But dns, yes we just want a good and fast working dns solucion for clients.

cheers

roadracer96 · Sat Feb 13, 2010 7:21 pm

hmm, maybe I was getting carried away.
dns, that's really what we want. The other servers is not really important for small WISP's anyway. Plenty of third party soluciotions available on the web so why should I want to do that myself.
But dns, yes we just want a good and fast working dns solucion for clients.

cheers

Then install BIND. I dont know what power costs in Spain, but running a PC with bind should run you more than about $1USD/month. Expensive battery backups? $300 USD will buy you a backup that will run a standard PC for over an hour.

jcremin · Sat Feb 13, 2010 8:50 pm

hmm, maybe I was getting carried away.
dns, that's really what we want. The other servers is not really important for small WISP's anyway. Plenty of third party soluciotions available on the web so why should I want to do that myself.
But dns, yes we just want a good and fast working dns solucion for clients.

cheers

Then install BIND. I dont know what power costs in Spain, but running a PC with bind should run you more than about $1USD/month. Expensive battery backups? $300 USD will buy you a backup that will run a standard PC for over an hour.

I'm not sure what you are using to calculate $1/mo, but in April 2009, the average US electric cost was about 12 cents per kilowatt hour, and the trend has been that the price keeps rising. The average desktop computer takes between 100 and 200 watts, so lets compromise at 150 watts. Running it 24 hours a day would cost about $12/mo, or $150/year in electricity. I'm sure some places have cheaper electricity, but I'm also sure that a lot of countries cost much more. Don't forget about some who need to run off of solar or wind power where every watt is valuable.

Now that $300 battery backup might keep it running for an hour, but if you're running a serious ISP, you probably want a lot more than 60 minutes of uptime. While my small towers may only have 2 hours of uptime, I make sure my main tower and servers can run at least 18, and I'm working to get that up to 36 hours when I get a chance to install a second battery. For that kind of uptime, we're now talking about thousands or tens of thousands of dollars for batteries or generators to keep your network online. A routerboard at about 5 watts of consumption will cost a lot less to keep running in an outage, and my $500 battery backup system can now keep my network up for a day and a half.

Also take into account that you typically have a much higher up-front cost for a computer than a routerboard, $500-$1000 for a high grade reliable computer (you don't want to skimp because a DNS server failure would bring your whole network to it's knees). I'm sure a 450G for around $100 including a case and power supply is powerful enough to run quite a snappy DNS server, and it would be much less prone to component failure than a regular PC.

My 2 cents...

WirelessRudy · Sun Feb 14, 2010 12:51 am

Then install BIND. I dont know what power costs in Spain, but running a PC with bind should run you more than about $1USD/month. Expensive battery backups? $300 USD will buy you a backup that will run a standard PC for over an hour.
I'm not sure what you are using to calculate $1/mo, but in April 2009, the average US electric cost was about 12 cents per kilowatt hour, and the trend has been that the price keeps rising. The average desktop computer takes between 100 and 200 watts, so lets compromise at 150 watts. Running it 24 hours a day would cost about $12/mo, or $150/year in electricity. I'm sure some places have cheaper electricity, but I'm also sure that a lot of countries cost much more. Don't forget about some who need to run off of solar or wind power where every watt is valuable.

Now that $300 battery backup might keep it running for an hour, but if you're running a serious ISP, you probably want a lot more than 60 minutes of uptime. While my small towers may only have 2 hours of uptime, I make sure my main tower and servers can run at least 18, and I'm working to get that up to 36 hours when I get a chance to install a second battery. For that kind of uptime, we're now talking about thousands or tens of thousands of dollars for batteries or generators to keep your network online. A routerboard at about 5 watts of consumption will cost a lot less to keep running in an outage, and my $500 battery backup system can now keep my network up for a day and a half.

Also take into account that you typically have a much higher up-front cost for a computer than a routerboard, $500-$1000 for a high grade reliable computer (you don't want to skimp because a DNS server failure would bring your whole network to it's knees). I'm sure a 450G for around $100 including a case and power supply is powerful enough to run quite a snappy DNS server, and it would be much less prone to component failure than a regular PC.

My 2 cents...

All true. We have at least once a year a serious power cut lasting several hours. I have been lucky each time that I was home to start my gen after some hours to charge my UPS. Units running on batteries could last at least several hours more....
Price/quality of battery-charger compared to UPS is 300% better option I learned the hard way...
Same counts for rb hardware compared to PC hardware.

And yes, why not save some bucks on electricity. "Think green" is the fashion nowadays and if this can be achieved without any loss of service and quality, why not go for low energy demand hardware like rb?
But now we are seriously drifting off-topic here.

The message and conclusion of this topic should be that the problem of the initiator gets solved and it seems to me the best route to go for MT would be to make a dns-server package for ROS
There are plenty arguments given for this and solucions to the initial problem given so far are basically nothing more then sympton curing, or just avoidance by going another road.

The intial question was "How can MT solve a problem with dns." and the answer should read; "MT develops a full dns-server package."

roadracer96 · Sun Feb 14, 2010 8:18 pm

MT has a DNS caching package. It just doesnt work with ONE provider.

Sorry. I forgot to carry a 1 when multiplying the power consumption. Regardless. It isnt a big deal. Cheap is cheap.

Where do you run your mail servers, webpages, etc, etc, etc.

I guess Im done with this argument. Ive worked for ISPs that cheap out on shit or cant bring themselves to pay someone to set something up right or take the time to learn to do it themselves. Ive had the same basic argument many times over the years. At this point, we have been arguing it for months, applying workarounds, and pointing the finger at MT. The whole time, you and your customers have been the ones suffering. Is that worth $10ish/month in electricity? I dunno.

Personally, I would rather see expansion of the VPN components of MT WAY before a full featured DNS or mail server. At least those are "native" to a router/firewall.

jcremin · Sun Feb 14, 2010 8:40 pm

MT has a DNS caching package. It just doesnt work with ONE provider.

Yeah, this has gone off topic from the specific problem. While not directly related to the original topic, it still would be nice to have a little more dns functionality than what is currently offered.

Sorry. I forgot to carry a 1 when multiplying the power consumption. Regardless. It isnt a big deal. Cheap is cheap.

Depends, electricity costs much more in some countries, and like we discussed, there can be a heck of a lot more to the cost of a server than just the electricity. Plus, we're talking about reliability too, not pure costs.

Where do you run your mail servers, webpages, etc, etc, etc.

I lease my servers from a real datacenter, but there are many other services available. Once I grow large enough, I will probably move my email over to Google apps. There's nothing saying that an ISP has to offer email or web sites in the first place, so a DNS server could be the only "real" server that is required for the network to be operational.

Not trying to say that anyone is right or wrong, just stating that running DNS on a real server might be appropriate for some, while alternative solutions might be better for others.

WirelessRudy · Sun Feb 14, 2010 8:55 pm

Not a single network has the same necessities. That's why we will never find standard answers or solutions to many issue or questions we come across in this forum.

As user in the past I never understood why I could not have my own mail address but was bound to that one of my provider. Nowadays we have hotmail, gmail and many others so why should I actually as a provider want to run mail servers? Or web or file servers? Unless you want to make the extra bug I personally think others can do that much better then me and it leaves the client its freedom.

On the other hand, dns is a basic tool for networking and browsing on the internet. That has to work flawless and fast.
It did it for me with MT-cache in combination with OpenDns for a long time. And then the google issue popped up last November.
I tried to install BIND two years ago but ended frustrated due lack of knowledge on how to set a reliable system up to run something like BIND.

Most of the knowledge needed for running networks has to be learned. Everybody has been or is in that boat and up to some months ago I never had the real need to have an alternative for MT-dns chache.
Now I have I simply lack the time to spend in studying BIND again. I need to run my network and have higher priorities on my work list then to solve a problem that is cured by a simple switch to my ISP's dns servers.
But it nevertheless would be a great add-on if MT could make something needed to close this topic.

R.

djmuk · Fri Feb 19, 2010 12:58 am

Hey guys can we move the argument to a new thread and get on with working out what is happening and fixing the problem... I'll use whatever's to hand & yes I have a windows XP box running bind...

I have an install with the same problem, What I did notice was:

my dns servers are resolver1.opendns.com and resolver2.opendns.com which are the ones the ISP gave me!!!

www.google.com is a CNAME for www.l.google.com

www.l.google.com wasn't in the cache.

I didn't think I had the opendns redirection entry initially but it was there just now and when I flushed the cache it was the only entry for www.google.com with a 30sec TTL. Before the cache flush I had CNAME entries for www.google.com that pointed to both the opendns redirect AND the www.l.google.com so I wonder if that is the root of the problem - at some point the MT has managed to get the 'proper' google CNAME entry as well as the opendns one?

Prior to the cache flush it was the same problem:-

doing an nslookup (against the MT) for www.google.com returned ' No address (A) records available for www.google.com'
doing an nslookup for www.l.google.com returned several (6) IP addresses which were then returned by an nslookup for www.google.com.

Looking in the cache the www.l.google.com A records have a TTL of 5 minutes which counted down, when it reached 0 it started counting up...?? & the query for www.google.com failed again...

It seems to be the l.google.com domain that is causing the problems as I just noticed some entries for
youtube-ui.l.google.com that are showing the same counting up behaviour...

Why did does the TTL count up???? Why doesn't MT flush & requery an expired entry??

aaargh....

David

jgilcas · Thu Feb 25, 2010 12:34 pm

Hi.

I still having DNS problems but not with google, i made static entries and everithing it´s working ok, now the problems is with es.search.yahoo.es

I made the same procedure to fix it, add stratic entrie but no luck.

Any idea?

Thnks

mocart · Thu Feb 25, 2010 7:38 pm

I have the same problem i.e. unable to reach google.com. I am using simple workaround which is changing IP by using Mask Surf application. But I can not explain why this works:)

DeVerm · Fri Feb 26, 2010 12:31 am

Hi.

I still having DNS problems but not with google, i made static entries and everithing it´s working ok, now the problems is with es.search.yahoo.es

I made the same procedure to fix it, add stratic entrie but no luck.

Any idea?

Thnks

Making static entries for domains that you are not authoritive for is the beginning of the end. You break a system that has proven to work for 30 years.

What is your DNS config and what does dig or nslookup show?

cheers,
Nick.

jgilcas · Fri Feb 26, 2010 11:54 am

Hi.

I still having DNS problems but not with google, i made static entries and everithing it´s working ok, now the problems is with es.search.yahoo.es

I made the same procedure to fix it, add stratic entrie but no luck.

Any idea?

Thnks
Making static entries for domains that you are not authoritive for is the beginning of the end. You break a system that has proven to work for 30 years.

What is your DNS config and what does dig or nslookup show?

cheers,
Nick.

I don´t know if it´s the beginning of the end but this worked for me with google.

"The temporary solution for this problem is
Code:
/ip dns static
add address=208.69.34.230 disabled=no name=www.google.com ttl=1d
add address=208.69.34.231 disabled=no name=www.google.com ttl=1d"

Respuesta no autoritativa:
Nombre: rc.europe.fyeu.b.yahoo.com
Address: 87.248.121.75
Aliases: es.yahoo.es

>

DeVerm · Fri Feb 26, 2010 5:17 pm

Sure looks like the same problem. What is your DNS config, which servers are you using?

Also, what does a dig/nslookup tell you about es.yahoo.es ?

cheers,
Nick.

jgilcas · Tue Mar 02, 2010 2:43 pm

Sure looks like the same problem. What is your DNS config, which servers are you using?

Also, what does a dig/nslookup tell you about es.yahoo.es ?

cheers,
Nick.

First DNS Server 8.8.8.8
Second DNS Server 8.8.4.4

nslookup

Non-Authorative answer
Name: rc.europe.fyeu.b.yahoo.com
Address: 87.248.121.75
Aliases: es.yahoo.es

Thanks.

und3ath · Tue Apr 20, 2010 5:29 pm

Nothing new with this issue? I am still having problems with all google services (google.com, gmail.com,...)

hilton · Tue Apr 20, 2010 5:41 pm

I gave up on OpenDNS and I now use my local ISP's DNS and it's much better. Pity there's no local cloud based web filtering service that's friendly on the pocket.

jcremin · Tue Apr 20, 2010 8:03 pm

I gave up on OpenDNS

Same here. No problems using Google's Public DNS service. Wasn't worth my time to keep fighting to find the problem.

pedja · Tue Apr 20, 2010 11:49 pm

There are other public DNS servers. I tried several and got rid og problem. I've published long list of puiblic DNS at http://pedja.supurovic.net/kako-upotreb ... provajdera

You can also use http://www.grc.com/dns/benchmark.htm (free tool) to check speed of number od DNS servers which would help you choose which one to use.

om5495 · Mon Mar 16, 2020 6:56 am

Hi, I am Om Prakash Yadav
When i try to ping 8.8.8.8 it says request timeout but when i ping google.com it's working fine.
I am using mikrotik router rb 1100ah. How to solve this issue??