I think I have solved the mistery: Bind (or openssl, who ever does the validation) requires Subject Alternative Name. Regardless if using the hostname or the IP address, they must be in the subject alternative name. When using self-signed certificates, it is probably best to put both in the SAN. Using the following certificate on the server, the validation in dig works fine, regardless if using the hostname or IP address.
The DNS-over-TLS specification insists on this behavior. See https://datatracker.ietf.org/doc/html/rfc8310.html#section-8.1
Quote: A compliant DNS client MUST only inspect the certificate's subjectAltName extension for the reference identifier. In particular, it MUST NOT inspect the Subject field itself. -- Petr Špaček Internet Systems Consortium -- Visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe from this list ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
