Is there a way to create a mangle rule that matches the first packet (from client to server) of a https connection?
It doesn't need to 100%. What I want is a rule that can separate actual https initial packets from most other (i.e. port scanners) initial packets.

Hey

First packets of https connection are the TCP handshake, then followed by ssl handshake.
The first handshake is ip / port based.

What are your criteria?

Can you describe in use case format or words that do not discuss solutions or configuration setup. What is that you want to be able to do or to prevent regarding usage of the network?
I want office users to be able to............
I want to ensure guest using our wirelss config do not..................

my OpenVPN only accept the connection if the first packet is 60bytes. Connection like telnet won't get through and will be thrown to DROP rule.

Not great but works for me.

my OpenVPN only accept the connection if the first packet is 60bytes. Connection like telnet won't get through and will be thrown to DROP rule.

Not great but works for me.

Something like that would be nice, assumed first packet from a browser always has the same size.

What I had in mind was a https server on a non-standard port.
And have a filter rule that allows only ip's listed in a address list to access this server.
When a client wants to connect it will send a initial packet towards the server. If this packet can be matched by a mangle rule, the ip could be added to the address list and thereby be allowed access to the server.

Why would anyone want a rule that allows anyone to connect? Well, I'm thinking that if the first packet of a https connection differs from any packet from a port scanner, the port will look closed to the portscanners, but still be open for https connections. This way I hope to make the server a little bit harder to find.

First packet in https session will be IP/TCP packet sent from client towards server ... without payload ... so only IP and TCP headers. Special features: it will have SYN flag set.

Second packet in https session will be IP/TCP packet sent from server towards client ... without payload ... so only IP and TCP headers. Special features: it will have ACK and SYN flags set.

Third packet in https session will be IP/TCP packet sent from client towards server ... without payload ... so only IP and TCP headers. Special features: it will have ACK flag set.

Fourth packet in https session will be IP/TCP packet sent from client towards server ... with start of TLS handshake.

So it will be only the fourth (third sent by client) packet with some payload which would differentiate legitimate https client from port scanner.

And the same story is happening for all TCP services ... because that kind of behaviour is inherent to TCP sessions. No way around it.

With UDP it's different because UDP doesn't do the three-step initial handshake and already first packet carries payload (in TCP only the fourth does). OpenVPN can work over UDP, therefore it's possible to do what @pegasus123 described.

Thanks. Then the answer to my question is 'no'.

The old age conundrum. How to have folks access your servers when you dont have a list of their WANIPs (not static) and thus one cannot at a minimum have a source address list in the mix. An address list in of itself does not provide security for 'sensitive information; and that is why two factor authentication and VPN tunnels are often used for secure communications.
For most of us, that perhaps dont have anything anybody wants, using a non-standard port, with a server with TLS password protection and a source address list is probably acceptable or used to be - and perhaps add in some method to stop scanners that are probing other more common ports may help mask your server, and add a rule that blocks those with too many repeated attempts at your port (if possible) - in addition to isolating that server on its own vlan. I personally will not connect to my network and especially router without a VPN tunnel.
;

Well,, maybe you can use "TLS-host" option is some way

Well,, maybe you can use "TLS-host" option is some way

To do it, somebody (https server or router or IDS or ...) has to perform TLS handshake. Even willingness to perform it (by sending SYN-ACK reply to initial SYN packet) reveals to portscanner that the port is open and active ... which is what @petterg wanted to avoid.