I'm having issues with HTTPS and the hotspot system we use for hotels
I've taken over as lead network engineer, the previous employee had put together a hotspot system that I am slowly understanding, piecing together and fixing issues with
The main issue I have at the moment is HTTPS sites don't redirect to the hotspot login page
I have come to realize that we need a signed certificate. So i've followed this https://wiki.mikrotik.com/wiki/Manual:H ... PS_example and managed to get an initial redirect request for 'some' HTTPS sites, but many of the larger ones i.e. google throw an SSL error
I can bypass some sites by manually adding walled garden entries. But i'm still having issues with our hotspot page not loading when accessing https: redirects but works fine with http: - i'm not sure if its the hotspot system or a MikroTik issue

I want to know what people are doing with hotspot systems these days? In an ideal world i'd like users to get redirected to the hotspot page no matter what they try and do (as I personally HATE support calls about this sort of thing). Is this currently possible with Mikrotik's hotspot implementation or are people using another service for it?
I don't mind completely changing our hotspot system and paying for something that works correctly 99.9% of the time

I'm not sure what our current hotspot system is based on, but it runs on a hosted linux server, has a management GUI and allows us to create sites and edit a lot of parameters. It allows us to download a config file specifically for mikrotik devices and it contains a hotspot folder with a bunch of images etc that we apply on the main management page, as well as a config file that contains a bunch of information - including scripts and scheduler items. It routinely updates itself and replaces things like walled garden entries, downloads and ensures images and text for the hotspot page are up-to-date etc. After following the example above I noticed it does not work with https at all. It simple says

Redirecting to Welcome Portal...

Redirect error!
Connection lost, Welcome Portal not reachable or walled garden not updated!

I have enabled 'wwl-ssl' service and enabled HTTPS in ip->hotspot. I don't know if this is a mikrotik issue or an issue with the hotspot files - i'll have to look into it when I get some time
Problem is time is extremely limited and every day I have a very full plate that often spills over for days/weeks, so i'd like some input from people to get this all working as smoothly as possible. With or without our current hotspot system

In general, you will _never_ be able to reliably redirect HTTPS (for purpose of Captive Portal CP or anything else), unless you are able to install special certificate on the users device.
Which is not possible for a public hotspot.
This is just a feature of HTTPS, especially designed to inhibit such interference.
However, most of users devices trigger the CP because of probing the internet connection using standard http.
In case, user ignores the devices hint (similar to popup in IOS, Android, Windows), but directly opens the browser, having https://anydomain.com as homepage, user might be lost.

would it be possible to have a page within hotspot server, when user enters HTTPS address, they get redirected to this page and it will say that please use HTTP or http://anydomain.com instead?

No. As you wrote, it also needs an (impossible) https-redir. "Best" you could do is to trigger some error messages from browser, regarding invalid cert. But this frightens the user even more.
So just to ignore https here only viable solution.

Make sure your hotspot is intercepting requests to hotspot-detection services that any modern OS has. This includes HTTP requests to URLs such as http://gstatic.com/generate_204 and intercepting all DNS requests eg for invalid / random hostnames like "xgjaiobman"

Make sure your hotspot is intercepting requests to hotspot-detection services that any modern OS has. This includes HTTP requests to URLs such as http://gstatic.com/generate_204 and intercepting all DNS requests eg for invalid / random hostnames like "xgjaiobman"

If the hotspot server is a mikrotik router, how do you accomplish this?

Are additional rules manually added somewhere?
Hotspot rules are dynamically created in firewall filter & NAT, as such those rules can't be moved down

Can you give examples on how to get this working properly?

In part, HTTPS exists exactly to prevent such silent interception of web browsing.

If the hotspot server is a mikrotik router, how do you accomplish this?

Sorry, no idea, but doing this for long time already, on openwrt-based devices.
Which are much better suited for hotspots with "advanced features", like this one.

Just make a poster in your location, saying "scan this code to log in", and the QR code leads to http://neverssl.com

In part, HTTPS exists exactly to prevent such silent interception of web browsing.

Doesn't change the fact that other hotspot devices have far, far better hotspot handling than MikroTik. It seems to 'just work' far more often.
Whereas we constantly get the odd device that just doesn't play ball with MikroTik's hotspot implementation and doesn't detect it
How do they do it? I don't know, but MikroTik needs to either improve their implementation, or post information in the Wiki on how to improve it themselves (i.e. publish the mechanisms that all devices are using to detect captive portal systems so we can manually do redirects etc)

We tell people to manually go to the IP address of the router but it's not something we should have to do

Other devices break standards or do things considered illegal in some countries?
See my post above yours.

If the hotspot server is a mikrotik router, how do you accomplish this?

Sorry, no idea, but doing this for long time already, on openwrt-based devices.
Which are much better suited for hotspots with "advanced features", like this one.

Any recommendations for a package we can put on a low cost or low form factor device like a Raspberry Pi3/4?
At the end of the day if I can put something in that 'just works' i'm happy to entire ditch MikroTik for the hotspot functionality. I really don't care that much, I just want it to work and to stop getting complaints from hotel managers. But it has to be something that is hierarchical and centrally managed. We use a third party package through a web GUI that handles all the hotspot data like user accounts and images/HTML pages as well as access for hotel managers to generate vouchers etc

At the moment this functionality outweighs the problems with MikroTik hotspot handling, but I want to improve the reliability and it seems this is a dead end for that unless I can get some further help from MikroTik or someone who's worked it out for themselves

As I wrote already,
you will _never_ be able to reliably redirect HTTPS (for purpose of Captive Portal CP or anything else), unless you are able to install special certificate on the users device.
Which is not possible for a public hotspot.
Valid for MT-hotspots, openwrt-based ones etc. No difference.

I do not know implementation of CP (Captive Portal) on MT, whether based on iptables rules (wifidog) or much smarter, like using coova-chilli, which is the most advanced CP, as it easily integrates radius features, like speed- or volume limits, accounting etc.
MT might even use coova, but the real problem is the fact, that RoS is _NOT_ open source. For example, on openwrt I can combine a real powerful proxy to coova, like squid, to do some "http-mangeling". Or I can combine a local websever (on openwrt-device) with coova-chilli, to fake an internet connection, i.e. to suppress the annoying "popups" on Android, asking to connect to possibly available WiFi networks. Or I can keep the landing page open, on Android, after connection to web, for ads to be displayed. Or I can even serve local content (movies, music etc.) to the hotspot user, i.e. on a mobile hotspot, in public transport.
I started with MT-hotspots couple of years ago, but switched because of the limitations, being not open source.
However, the special features I mentioned are not ready-made in an openly available package for openwrt, more like customized features for clients.
Which usually also include remote firmware updates.

My main focus here is not in actually trying to redirect HTTPS, I really honestly don't give a flying stuff about that

The real issue is simply when hotspot detection fails, the user gets no prompt or no notification in any way that they need to first 'sign in' and the normal behavior is they just open up a web browser and by default it'll usually be a HTTPS site, this just leads to a the connection being dropped. So the idea of HTTPS redirection is just a band-aid for the real problem
Once again, I really don't at all care about a redirect, but I want the captive portal/hotspot detection to JUST WORK so that this never happens, and they ALWAYS get a notification to sign in. MikroTik sucks at this

HTTPS redirection is just a band-aid for the crap hotspot detection.
I have seen it done in some places, where manually trying to go to i.e. google.com does infact have an intercept successfully happen saying to first sign in, without just dropping the connection like there's no internet, or having certificate warnings. What system it was I don't know, how it 'actually' worked (probably not a HTTPS redirect but infact a graceful negotiation with the device/browser)
Regardless if it can't be done, fine, don't care, lets move on to actually fixing the underlying problem
Whether that be manual rules in the MikroTik router to improve hotspot detection, or just ditching it entirely and going with something that can be spun up on a low footprint device like a rasp pi and installing that inplace instead i'm all for it
I just want a solution

So the CP-detection is your real issue. I am a bit wondering about this, as I thought, MT-hotspots are quite often used,
and I expected MT to have a reasonable solution, at least.
Your issue(s) need more detailed discussion, it looks like, as CP-detection
also is dependent upon clients device, although the general principle is always the same:
Clients device tries to connect to "well-known" server, and expects certain feedback.
In case, it fails, considering other conditions (DNS, WiFi), a CP is assumed, and a "minibrowser" is opened.
Now slightly different scenarios, especially for IOS and Android, even different between Andoid versions.
I.e. IOS practically forbids JS in the "minibrowser", before successful login. Newer Androids do _NOT_ allow
redirection after login (connection to internet), simply closing the "minibrowser" automatically. Which can be delayed, if required, using special tricks.
So, careful integrated design of the "Landing Page" and the hotspot-device/CP to be done.

As we are now going off-topic, you can reach me via my antispam adrs augustus_meyerATyahooDOTde

Make sure your hotspot is intercepting requests to hotspot-detection services that any modern OS has. This includes HTTP requests to URLs such as http://gstatic.com/generate_204 and intercepting all DNS requests eg for invalid / random hostnames like "xgjaiobman"

Wait a moment... this should read: make sure your hotspot is not intercepting....
It is very important that you do not handle such special detection services in a special way!
So do NOT return a fake IP on those DNS requests, do NOT redirect those special requests to local servers, do NOT place these special sites in the walled garden, etc!

Those requests should fail. When they do, the software behind it knows they are on a network without immediate internet access, and they use this to show the user a special page that informs them about the issue and lets them proceed to the login page of your hotspot (by trying to load a dummy http page).
Once that has been done, they can go back to their https homepage.

Any special handling you put on those special requests (like allowing them through) will break the clever software and cause problems for your users!

I have the same issue when i connect with ssid "hotspot services" With computer's, After connect to ssid and i go to broswer like google chroum (If i set before in browser login page like "https://www.google.com", In this case the hotspot login page can't change automatic to hotspot server), Whats the problem or how can resolve the issue?, In mobile when i connect to ssid hotspot service the mobile direct go to login page of hotspot but in computer's can't change?
I haven't certificate.

As written many times above, that issue cannot be solved!
However, the writers of software like Chrome and Android do know that, and they use requests that you do not enter yourself (some DNS and some HTTP requests) to detect this situation.
When they find that they are on a hotspot/portal network, they allow the user to enter the login info instead of going to the homepage site.

My main focus here is not in actually trying to redirect HTTPS, I really honestly don't give a flying stuff about that

The real issue is simply when hotspot detection fails, the user gets no prompt or no notification in any way that they need to first 'sign in' and the normal behavior is they just open up a web browser and by default it'll usually be a HTTPS site, this just leads to a the connection being dropped. So the idea of HTTPS redirection is just a band-aid for the real problem
Once again, I really don't at all care about a redirect, but I want the captive portal/hotspot detection to JUST WORK so that this never happens, and they ALWAYS get a notification to sign in. MikroTik sucks at this

The captive portal detection not working for you is almost certainly caused by some kind of hotspot walled garden entry that shouldn't be there. The hotspot manager you are using I believe is HSNM hotspot manager. I would go through the walled garden with a fine-tooth comb and clean up anything from there that might be causing a problem. Something I see on the software page that is a bit alarming is the fact that they display YouTube videos when you are not authenticated, as well as other resources. Those same Google servers are likely being used to host the captive portal detection sites, and if they are allowed through by a walled garden entry, then the captive portal detection will be broken. In essence, you may be giving up reliable captive portal detection in exchange for the ability to play youtube videos for unauthenticated users. I'm not sure the trade-off is worth it.

As written many times above, that issue cannot be solved!
However, the writers of software like Chrome and Android do know that, and they use requests that you do not enter yourself (some DNS and some HTTP requests) to detect this situation.
When they find that they are on a hotspot/portal network, they allow the user to enter the login info instead of going to the homepage site.

This problem has not been fixed with the updates that come from the company .. The biggest problem when you activate the hotspot service to impose in the hotel and the customer will connect his laptop and then the page did not appear Login and the customer does not know in the field of information technology What is the attitude of the hotel towards such problems in the page access!

As written many times above, that issue cannot be solved!
However, the writers of software like Chrome and Android do know that, and they use requests that you do not enter yourself (some DNS and some HTTP requests) to detect this situation.
When they find that they are on a hotspot/portal network, they allow the user to enter the login info instead of going to the homepage site.

This problem has not been fixed with the updates that come from the company ..

What company do you mean here?

The biggest problem when you activate the hotspot service to impose in the hotel and the customer will connect his laptop and then the page did not appear Login and the customer does not know in the field of information technology What is the attitude of the hotel towards such problems in the page access!

It is a problem that has been caused by the move of all websites to https.
That move wasn't very wise, but we will all have to live with it, no matter if we are using a hotspot service, operating a hotel, or visiting one.

hing I see on the software page that is a bit alarming is the fact that they display YouTube videos when you are not authenticated

Yes, this is very, very suspicious, at least.
Good indication would be problems with Android phones, as their CP-detection relies on google servers. IOS should not be concerned here.

As written many times above, that issue cannot be solved!
However, the writers of software like Chrome and Android do know that, and they use requests that you do not enter yourself (some DNS and some HTTP requests) to detect this situation.
When they find that they are on a hotspot/portal network, they allow the user to enter the login info instead of going to the homepage site.

This problem has not been fixed with the updates that come from the company ..

What company do you mean here?

The biggest problem when you activate the hotspot service to impose in the hotel and the customer will connect his laptop and then the page did not appear Login and the customer does not know in the field of information technology What is the attitude of the hotel towards such problems in the page access!

It is a problem that has been caused by the move of all websites to https.
That move wasn't very wise, but we will all have to live with it, no matter if we are using a hotspot service, operating a hotel, or visiting one.

The company i mean "The mikrotik company", (The manufacture company),
Can solve the issue when the all website move to https! By new update to all routerborad's,
If I buy a certificate for https to upload in routerborad do you think the problem solved?

@loveman: I think you don't get it, MikroTik can't solve it. The redirection done by captive portals was always dirty trick, an abuse of technology. That's one reason why https became so popular, because too many people liked to tamper with someone else's unprotected traffic. Https prevents that, so the redirection no longer works, and it's by design.

You can buy certificate for your hotspot, but it won't help you with redirecting others websites. You can imagine certificate like house key, you can have your own, but it won't unlock other houses.

@loveman: I think you don't get it, MikroTik can't solve it. The redirection done by captive portals was always dirty trick, an abuse of technology. That's one reason why https became so popular, because too many people liked to tamper with someone else's unprotected traffic. Https prevents that, so the redirection no longer works, and it's by design.

You can buy certificate for your hotspot, but it won't help you with redirecting others websites. You can imagine certificate like house key, you can have your own, but it won't unlock other houses.

According to your advice at the moment not to buy a certificate so do not solve the problem, in the case of buying a certificate for the same subject What are the costs if i buy for hotspot?
What is the total prices?

You can get certificates for free. But only for your own site. So you cannot get a certificate for Google.com and so you CANNOT SOLVE the redirection problem.
And neither can MikroTik.
It is just a case of 'sorry but that is no longer possible, forget about it'.
That is why you should focus on getting the portal detection in the browser working correctly.
That is, you must make sure that the probes that the browser makes (resolving certain DNS names, fetching certain http pages) do NOT get treated special by the portal.
They must get the same treatment as any other DNS lookup or http page lookup. I.e. be disallowed.
When that is properly setup, it will work OK.

You can get certificates for free. But only for your own site. So you cannot get a certificate for Google.com and so you CANNOT SOLVE the redirection problem.
And neither can MikroTik.
It is just a case of 'sorry but that is no longer possible, forget about it'.
That is why you should focus on getting the portal detection in the browser working correctly.
That is, you must make sure that the probes that the browser makes (resolving certain DNS names, fetching certain http pages) do NOT get treated special by the portal.
They must get the same treatment as any other DNS lookup or http page lookup. I.e. be disallowed.
When that is properly setup, it will work OK.

"You can get certificates for free" whats the steps for how to get certificate to hotspot after that i have the secure website login in my server?
In my case without automatic redirection when the user need to login, I go to labtop of user and open the browser and write in address (ip of hotspot server) after that the browser change direct to hotspot login page.

When buying a certificate for your website, buy from "trusted" authority.
Otherwise you might face inconvenient issues when using newer IOS clients.

When buying a certificate for your website, buy from "trusted" authority.
Otherwise you might face inconvenient issues when using newer IOS clients.

Can you tell me about trusted website to buy from it.

https://support.apple.com/en-jo/HT209144#trusted

Is the website can buy direct to hotspot server mikrotik?

https://support.apple.com/en-jo/HT209144#trusted

Price and Trust have nothing to do with each other!
E.g. Letsencrypt certificates are free and they are trusted, but paid certificates from some used-to-be-big-names like Symantec are NOT Trusted!

https://support.apple.com/en-jo/HT209144#trusted

Price and Trust have nothing to do with each other!
E.g. Letsencrypt certificates are free and they are trusted, but paid certificates from some used-to-be-big-names like Symantec are NOT Trusted!

If possible, please write down the basic steps in order to make the login page safe "secure" for hotspot server, and are they all change to secure for free or not?

No amount of money you spend on certificates will fix this issue. You cannot get a certificate that's valid for the entire internet.

Best things to do:

• Intercept ALL requests to internet (make sure gstatic.com, captive.apple.com, etc are NOT whitelisted as some misguided posts suggest)
• Make sure interception is redirecting to a local IP or DNS name (eg http://hotspot.local/) and only hosted on private IP range.
• Ideally have a FQDN with split horizon DNS so you can use Let's Encrypt and redirect to https instead to protect your logins (eg https://myhotspot.example.com/)

The rest is up to the client device to detect the portal and redirect you. You have no control over this.

Here's a list of domains that devices check. I have no idea if this list is totally accurate and if its missing any (can't see a post date) but its a start
https://success.tanaza.com/s/article/Ho ... rtal-works

It's all well and good saying what should or shouldn't be done but we need information on WHAT to do. For instance

Best things to do:

• Intercept ALL requests to internet (make sure gstatic.com, captive.apple.com, etc are NOT whitelisted as some misguided posts suggest)
• Make sure interception is redirecting to a local IP or DNS name (eg http://hotspot.local/) and only hosted on private IP range.
• Ideally have a FQDN with split horizon DNS so you can use Let's Encrypt and redirect to https instead to protect your logins (eg https://myhotspot.example.com/)

Fine, but does the default MikroTik hotspot do the first 2 in which case nothing else needs to be done? Or are additional steps required?
I'm of the opinion yes its handled that way by default, but need to make sure

What I did do is scour all the walled garden section (there's a bunch) and found *gstatic.com and *akamai* was infact in the wallen garden list. We are using HSNM (lets see if that now fixes the issue and reduces complaints)
These were not in the configurable parameters but actually came from a php script. I found the file in /sitiweb/hotspot/functions/walledgarden.php and simply commented out mentions of these 2

Despite having these 2 entries in there, hotspot detection did work ~98% of the time but doing packet captures in a lab I can see our android devices do infact check gstatic but also check other DNS names so just blocking gstatic on its own didn't have an issue in my lab testing, not to say it wouldnt' with some specific versions of android perhaps
I do however have a Win7 test PC that isn't detecting hotspot and displaying any notification. Hence when using it and trying to open i.e. google, facebook, etc (all the usual sites people would likely have as their homepage that are now only HTTPS) it just fails the connection with no notification.

Now here's another question, i'm not typing https://www.google.com
I'm simply typing www.google.com or google.com and it fails. Why? isn't the default request http://google.com which then either doesn't exist or says 'use https' so then it tries https://google.com
That fails, heck even specifically typing 'http://google.com' to try and force a http connection fails, but typing 'news.com.au' ALWAYS works (https://news.com.au does not, but nobody manually types https) and successfully redirects to the hotspot (it's actually what we tell people to try first as its an easy address to say over the phone) yet that is also a 'HTTPS only' site as well
What's going on in this scenario?
Let's ignore 'HTTPS' interception for a second and look at the steps leading up to that, because i'm NOT typing 'https://' as the prefix, i'm just typing the domain name. My understanding is it'll always try http first. That part should be interceptable surely?

Now here's another question, i'm not typing https://www.google.com
I'm simply typing www.google.com or google.com and it fails. Why?

Google uses HSTS. Once you visited an HSTS enabled site your browser will force you into using HTTPS any time you access that domain. HSTS expiration period for google.com is currently set to 1 year.

Correct.
But thats not all, as firefox, for example, has a pre-loaded list of HSTS sites. I guess, same for Chrome.

Like said above, you should not be typing anything. Computers and phones have hotspot detection, unless you somehow messed up your access list (I hope it's empty!) then the device will detect, that it isn't seeing "SUCCESS" in the (for example) Apple portal, so it will pop open a login page.

Before talking about https redirection, try to resolve why this doesn't happen.

What I did do is scour all the walled garden section (there's a bunch) and found *gstatic.com and *akamai* was infact in the wallen garden list. We are using HSNM (lets see if that now fixes the issue and reduces complaints)
These were not in the configurable parameters but actually came from a php script. I found the file in /sitiweb/hotspot/functions/walledgarden.php and simply commented out mentions of these 2

Very bad idea to put gstatic.com into walled garden. Which could mean, there are more such harmful entries.

Best to do, just to remove everything from walled garden, and then only insert those domains, you really need.
In detail, to be found during test of CP then, of course, with different OS (even diferent versions of Android), and different browsers.
Note, that the removal of gstatic.com _might_ be offset by any other google-domain in walled garden, as its DNS could resolve to gstatic.coms DNS, by chance.
Which means, even the usage of some of googles domains, serving some fancy fonts, _MIGHT_ do harm, when in walled garden, just to allow to fetch the fonts.
Better to host these fonts on your own server.

Removing akam-domains will help for same reason, as some of Apples domains, used for CP-testing, are hosted there.
So, as a rule of thumb: Walled garden as small as possible.
As Normis wrote: Empty walled garden is best

Like said above, you should not be typing anything. Computers and phones have hotspot detection

But as I keep saying, I want some actual information on this. Not just 'it should work'
HOW does it work? I would like information on how all devices detect hotspot in the first place. Not just a brief overview of "they try and connect to a site and if it fails it'll show you the login page" that doesn't tell me anything. I want to know what sites are attempted to be accessed, does it do a http request? DNS? ping?
And how does does it know the hotspot login page to send the user to?
Lets say the hotspot is hosted at 1.1.1.1, how does it know to send the user to 1.1.1.1? This is handled behind the scenes by the MikroTik router but I want to know this information to properly troubleshoot and possibly improve it as much as possible

For instance that site I posted says Windows attempts to fetch a URL from....
ipv6.msftncsi.com
ipv6.msftncsi.com.edgesuite.net
www.msftncsi.com
www.msftncsi.com.edgesuite.net
teredo.ipv6.microsoft.com
teredo.ipv6.microsoft.com.nsatc.net

So what if IPv6 is also enabled but hotspot isn't handling IPv6? Will that mean no hotspot notification, but sites that are on IPv6 work fine but when a user tries a IPv4 site it silently fails (never got a notification to login)

And how does does it know the hotspot login page to send the user to?
Lets say the hotspot is hosted at 1.1.1.1, how does it know to send the user to 1.1.1.1?

Back to one of my previous posts: Using openwrt for hotspots will show you, because open source. I.e. check the docs and source code for coova-chilli,
the most advanced CP for openwrt, also used in other higher end commercial hotspot products.
_MAY_ be , mikrotik uses it "under the hood", too, which I suspect, because of "walled garden" and radius interface.
May be, normis can (willing to ?) to confirm.

But as I keep saying, I want some actual information on this. Not just 'it should work'
HOW does it work? I would like information on how all devices detect hotspot in the first place.

This question has nothing to do with Mikrotik, all vendors do it in their own unique way (and also change those ways from time to time). This is in no way standardized. You should direct such questions to Google, Apple, Microsoft, etc. I doubt anyone on this forum will be able to answer this particular question fully.

What the above poster said, yes.

But in essence, every OS will try to open some webpage in the background, and get the content from that webpage. If content doesn't match with what's expected, it will open up that URL in a popup, to show you what is there. Most likely a login form.

1) This question has nothing to do with Mikrotik, all vendors do it in their own unique way (and also change those ways from time to time).
2) I doubt anyone of this forum will be able to answer this particular question fully.

1) Correct.
2) Wrong. However, because of 1), no further details here.
Quite a lot of info about it to be found using goggles search; although quite often incomplete or outdated, so to be taken with a grain of salt.

But as I keep saying, I want some actual information on this. Not just 'it should work'
HOW does it work? I would like information on how all devices detect hotspot in the first place. Not just a brief overview of "they try and connect to a site and if it fails it'll show you the login page" that doesn't tell me anything. I want to know what sites are attempted to be accessed, does it do a http request? DNS? ping?

When the device connects to the network, or the browser starts, it does an HTTP request (not HTTPS) to download a specific file from the vendor website. The file is a special file only used for detecting captive portal. Different vendors use different files (URLs) in different versions sometimes. If it gets the expected file from the vendor website, it knows there is no captive portal and doesn't bother to present a login page. If the file from the vendor website is not as expected (different contents) or the HTTP response doesn't match (generally it expects 204 I believe) then the device knows there is a captive portal.

And how does does it know the hotspot login page to send the user to?
Lets say the hotspot is hosted at 1.1.1.1, how does it know to send the user to 1.1.1.1? This is handled behind the scenes by the MikroTik router but I want to know this information to properly troubleshoot and possibly improve it as much as possible

The MikroTik hotspot takes any HTTP request from an unauthenticated user destined for the Internet and replies instead with a HTTP 302 (redirect) to go to http://1.1.1.1/login. Since the device is using HTTP (and not HTTPS) to request the captive portal detection page from the browser or device vendor site (ex. www.msftncsi.com or clients3.google.com), when it makes this request to www.msftncsi.com, it not only receives the wrong data and wrong code, so that it knows there is a captive portal, but it also receives the MikroTik's inserted reply of HTTP 302 ordering it to redirect to http://1.1.1.1/login. The device knows since it got this 302 redirect it was not expecting that that must be the captive portal address and pops up a browser window and goes to the URL it received from the HTTP 302 request for the user to complete the captive portal login process.

So what if IPv6 is also enabled but hotspot isn't handling IPv6? Will that mean no hotspot notification, but sites that are on IPv6 work fine but when a user tries a IPv4 site it silently fails (never got a notification to login)

Yes, that is correct. If the hotspot has an IPv6 address and the customer gets IPv6 from the hotspot device, and firewall rules allow them to browse, they will be able to get online to any IPv6 sites but the captive portal detection will not work so they will not be prompted to log in and any v4 sites would be broken. If a user cannot get an IPv6 address connected to the hotspot you are safe - the device itself can have IPv6 as long as the interface your hotspot users are on does not have v6. If you really need IPv6 on your hotspot for some reason, it must be blocked until the user authenticates, and only then should they be allowed through IPv6. Hotspot doesn't do anything with IPv6 out of the box, so you would need to script this somehow with the IPv6 firewall and lookup their v6 address through neighbor discovery or something. If it even works it is a lot of work and so it would not be worth doing unless you really really need IPv6.

With the default hotspot configuration, the captive portal detection works properly for all devices, all vendors. What breaks it is if you start to change the configuration to poke holes through (with walled garden entries). For instance, suppose you want your captive portal login to be all flashy and professional looking and get fonts from google or something. So you create a walled garden entry to allow fonts.gstatic.com - great, your login page now has nice fonts. But if fonts.gstatic.com runs on the same IP address as clients3.google.com, now by allowing the fonts through you have broken the captive portal detection for all devices that use clients3.google.com. Even worse is that you can't just rely on the fact that for you fonts.gstatic.com always resolves to a different IP than clients3.google.com it doesn't mean it will be the case for all users, since there may be DNS round robin involved and content distribution networks that sometimes change the results, so those two sites may be on the same IP or on different IPs at different times of the day or for different people in different places.

In order to have reliable captive portal detection that just works and works all the time, suppose you have Windows users. You need to make sure that your captive portal does not have walled garden entries for any Microsoft-owned IP addresses, and also that it doesn't have walled garden entries for any CDN-delivered hosts that may possibly also hold Microsoft content and may hold a copy of the captive portal detection page. Any IP that is owned by Microsoft or related in any way to Microsoft could also be used to serve up a copy of the Microsoft captive portal detection page and if you have a walled garden entry to allow that through, whether it is a video or some fonts or a map or the weather, the walled garden entry will also allow the captive portal detection page through without modification and the end user device will incorrectly detect there is no captive portal and fail to display the login.

The same goes for Android based systems or Chrome browser or Chromium and Google. If you have any rule to allow Google maps through to display a map on the login page, or a rule to use Google fonts on the login page, or a rule to allow displaying the weather on a login page, or a rule to display a YouTube video on the login page, that rule is at risk to also allow through the unmodified captive portal detection page for Android or Chrome or Chromium users causing their captive portal detection to be broken and resulting in a failure to be redirected to the login screen.

Sometimes it is something seemingly harmless you are trying to allow through. For instance maybe some developer creating a nice flashy login page for you is using external javascript libraries like bootstrap or jquery or things like that. You create walled garden entries for those javascript libraries to allow them to download, everything seems great. But now captive portal detection is potentially broken, if one of your browser or device vendors uses the same CDN to host their captive portal detection page.

So what can you do to pull external content in a safe way that will not break captive portal detection? The best way is to store all such content on a server controlled by you, and let that server and only that server through with a walled garden entry. If you are pulling the content from your own server, you should know that Microsoft is not also storing things on there and that Google is not storing things on there, and so you can be sure this will not break captive portal detection for any possible browsers or operating systems. You can even potentially use that as a reverse proxy to pull content from vendors like Google in a way that doesn't break the captive portal detection. Then you have only a single address allowed via walled garden.

For some people this may be too restrictive. If you would rather open some things (ex. display google fonts, youtube videos, etc.) for your login users, a workaround is to give them a website to go to that runs HTTP when they connect. For instance, the hotel desk clerk can tell the customer "connect to wifi and go to hotelconnect.com" or something, and you either register that domain or set up an internal DNS entry and set up a server there. The server itself can be almost empty, it really doesn't matter, but you would make sure it only runs http and not https and responds with some kind of page on http port 80 and then users hitting it would get redirected to the login page successfully even if captive portal detection is broken. You might even be able to create a local dns record pointing at the mikrotik itself with the name you want (ex. hotelconnect.com), I have not tried this.

Awesome response. Thank you for taking the time to clear it up
Before I created this thread I had no knowledge of how the actual 'hotspot detection' works outside of a HTTP redirect when the user themselves opens a page and try and browse. But that's really a last ditch effort and in an ideal world shouldn't even happen. Ideally the device should be aware of this before the user ever opens a web browser themselves, it's only when that fails that the redirection of the site they want to access is necessary

This is very helpful information because if I start to notice a pattern of specific devices not playing ball, and I have the time to investigate properly. I can run packet captures of that device and use this information to figure out whats going on, or whats not happening properly. Without understanding the underlying technology and how it works, you can't properly troubleshoot. You can make educated guesses at best, and thats not good enough for me when things seldom work and seldom don't

My understanding is IPv6, Proxy/cache servers (or clients proxy settings) or DNS can cause it to break and should be looked at. As well as ensuring walled garden does not contain any of the known URL's or IP's. The fonts are a prime example, i'm sure there is some content in the pages that relied on a google server, hence gstatic was in there by HSNM. I'll have to check all the pages and make sure they work fine after removing it

I don't run IPv6 anywhere yet but knowing about this helps greatly because there may be a server which has had DHCPv4 turned off, but DHCPv6 was forgotten and left on. Or a router got installed that had IPv6 package installed for testing and again same problem. Again this just comes to theoretical scenario's that may happen and knowing what to look for, rather than assuming the world is perfect and should work accordingly

No problem.

Related to this, I just found this page here has a MikroTik DNS "fix" for captive portal issues:

https://socifi-doc.atlassian.net/wiki/s ... ion+active

Why do they need a DNS "fix"? Their SOCIFI system must use a Google owned IP that is sometimes also used for captive portal detection. They are allowing that through with a walled garden entry and that is unwittingly also allowing Google captive portal detection traffic through.

They explain to set those static DNS entries: clients1.google.com, clients3.google.com, connectivitycheck.android.com, connectivitycheck.gstatic.com

Doing so overrides the automatic resolution and may make the problem go away but I don't feel this is the best solution. The underlying problem here is that they are allowing something they shouldn't through a walled garden list and instead of fixing it the right way (by no longer allowing that through the walled garden) they are using the workaround of forcing the DNS responses for any hostnames used for captive portal detection. This will only work as long as somebody at google doesn't have the idea to make a new address for captive portal detection for some new device or browser version and then you suddenly need to add that to everything.

I don't run IPv6 anywhere yet but knowing about this helps greatly because there may be a server which has had DHCPv4 turned off, but DHCPv6 was forgotten and left on. Or a router got installed that had IPv6 package installed for testing and again same problem.

A router with the IPv6 package simply installed is not a concern. The interface needs a valid v6 global address on a /64 subnet with advertise=yes in order for end users to get a v6 global address that could cause a problem. If the interface only has an fe80:: link local IPv6 there is no issue, since the customer could potentially reach that interface with IPv6 but that doesn't allow them to get online and doesn't interfere with captive portal detection.

In order for IPv6 to work for users on the hotspot network someone would have to add a global IPv6 address on the same interface used for hotspot users, either static or pulled from a pool, and it would have to be valid. So at the present time it is generally not a concern unless someone has set it up correctly and successfully on purpose. It may be different in the future, I don't know if MikroTik plans to have defconf (default configuration) include a DHCPv6 client etc. I hope they do for other reasons, but in the event they do that, the address should be removed on a device being used for hotspot. Even better, they could add v6 support into hotspot.

A router with the IPv6 package simply installed is not a concern. The interface needs a valid v6 global address on a /64 subnet with advertise=yes in order for end users to get a v6 global address that could cause a problem.

There is some bug in the RouterOS DHCPv6 that I still need to investigate further but that could make that statement partly invalid.
I am running a CCR (RouterOS 6.44.5) with several internal networks (no Hotspot, but I think it does not matter).
There are 3 networks that have IPv6 addresses and advertise=yes, plus one network that does not have an IPv6 address.

Recently I added 3 DHCPv6 servers for the 3 networks with IPv6 to advertise the CCR as the DNS resolver on that network (I do not advertise that in RA because then I cannot control which address is advertised). These 3 servers are tied to the 3 networks that have IPv6. It works, clients on those networks still get their address from RA and they get a DNS resolver when they attempt to get that using DHCPv6.

But recently, I added a Windows 10 machine on the network which does NOT have IPv6, no address, no RA, no DHCPv6 server, and it had connectivity problems.
Looking in "ipconfig /all" I saw that it had obtained 3 IPv6 addresses belonging to the 3 networks it isn't connected to!
For now I disabled the IPv6 protocol but I plan to look further into it and attempt to make a trace what is going on there.

But recently, I added a Windows 10 machine on the network which does NOT have IPv6, no address, no RA, no DHCPv6 server, and it had connectivity problems.
Looking in "ipconfig /all" I saw that it had obtained 3 IPv6 addresses belonging to the 3 networks it isn't connected to!
For now I disabled the IPv6 protocol but I plan to look further into it and attempt to make a trace what is going on there.

This doesn't sound like an IPv6 problem to me. There is probably a switching loop or switching problem somewhere where the three VLANs are being untagged and bridged to the fourth VLAN. The result would be that the router sending out regular "broadcast" ICMPv6 RA packets would get looped, bridged back to the other network, received by the computer and it would add the addresses, but it would not be possible to send responses back because the FIB on the switch probably is using a different path to send back to the MAC of the router where those VLANs exist.

I encountered a problem like this before, not with MikroTik, but with a Polycom IP phone on a Cisco switch with Voice VLAN configured. The phone was incorrectly leaking the ICMPv6 RA's from the voice VLAN to the PC port on the phone and when the computer was plugged into the PC port on the phone it would get an unusable IPv6 address on the Voice VLAN along with the working IPs on the data VLAN.

DHCPv6 on RouterOS does not have the ability to assign addresses, so your problem is ICMPv6 leaking, not DHCPv6 issues. Check your network carefully for misconfigurations where you may be bridging together those three VLANs with the other in untagged form on some spoke switch.

The networks are on different physical ports of the CCR router...
As I mentioned, I plan to investigate this further.

Testing on the network without IPsec I cannot reproduce the issue. Likely the PC has temporarily been connected to one of the other networks and has remembered the address.
(I don't know how it got all 3 addresses, nor do I know why Microsoft would remember the IPv6 address for so long even when it is connected to a different IPv4 network)

I found that the cause of this problem is a Windows bug...
The PC was connected to a switchport which had tagged VLANs on the other networks, and it turns out that Windows just strips all VLAN tags on incoming traffic and treats everything as "the network", so it of course receives the RA from the other networks, and it cannot communicate with them because it does not add the VLAN tags back on.
When the PC was moved to another spot and on a normal switchport (without the tagged traffic), everything went back to normal.
So not a MikroTik bug!

IS POSIBLE..

THIS VIDEO: https://www.youtube.com/watch?v=XcohMjJ ... x=10&t=12s

HOTSPOT HTTPS

And what exactly is that video supposed to prove? The part where redirection to login page works automatically, clearly shows that originally requested url used http, not https. So yeah, that works. But when requested page uses https, there's no redirection, but page saying (according to Google Translate) "the wi-fi used may require that you visit the entry page", which looks like standard hotspot detection. Where is anything different from RouterOS?

I still consider this resolved, because modern operating systems check a predefined HTTP link in the background, if that page is not what is expected, the operating system opens a login popup automatically.

Nothing special has to be done in the router.

I don't think theres anything more MikroTik could do (maybe better handling of IPv6 with this? But we don't use IPv6 so it doesn't affect me right now)
The industry should move to a better system in general, such as a new DHCP option number that includes a URL. So when a client connects and gets an IP lease it sees this extra DHCP information and then can immediately prompt the user that this is a captive portal system and they should be directed to that URL. Rather than relying on various HTTP request checks and needing the complicated DNS interception etc etc
Then it would be very simple for any device that recognises that DHCP option to immediately prompt the user when opening their browser, before even trying to go to a webpage that "NOTE: Captive portal system is in use, you may be required to login before you have connectivity, click here". Would never have to intercept the users request to go to a page, and they would have information on why its not working
Optional additional DHCP options to provide information on certificate information etc to mitigate MITM and spoofing attacks

As still to this day we have devices that do not seem to detect the captive portal system and give no notification to the user, we still get the occasional support call (though less of them as I found our system had gstatic* in whitelist)

The industry should move to a better system in general, such as a new DHCP option number that includes a URL.

That is RFC7710. Nobody uses it. The usual chicken-egg situation: nobody queries the option, so nobody bothers to support it in their DHCP server, so noboty bothers to query it because it will not be there anyway.
In some systems (including RouterOS) you could put the data in the DHCP server as a user-defined option and wait for someone to retrieve it (when you are patient).
Of course when it would get some real usage, MikroTik should offer more friendly UI to set it.