Use DNS validation to allow these internal services to pull ACME certs. There's so much less headache, long-term.
Split-horizon DNS (and the tedious make-work it can create when you start needing to mirror public-accessibly records in the private DNS) has always been something to aspire to move away from in my experience.
https://community.letsencrypt.org/t/dns-persist-01-deploymen...
I wonder if the interim version has been rolled out to some CAs.
You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward.
To prevent having to include DNSimple authentication on the client's internal server I have a small API server on the web which does the Acme work.
https://github.com/acme-dns/acme-dns
CAUTION, though, the last time I downloaded a binary release, ClamAV triggered on it, so I kept my old version which worked. I was using the 1.0 series (without any problems!), and now it seems the project has picked up development again with a 2.0 series.
Services themselves are constrained to the server, bound to listen on 127.0.0.1 only.
Key is only available to the reverse proxy.
- Don't use split DNS. Don't use any special internal or dev domain. Leave it to your infrastructure to route/NAT those public IPs to your internal network.
- Don't use the HTTP-01 challenge. Use DNS-01.
- Don't run your own internal CA. Use Let's Encrypt. If you care about name leakage (CT Logs), use wildcard certs. Use a central reverse proxy/load balancer for termination.
If you do that anyways, you could also use something like oidc authN/authZ on the reverse proxy level and just expose it to the internet.
You dont even need to self host the oidc idp, you can use Google/Github or even something like ATProto
This means that I can always use public DNS servers like 1.1.1.1, 8.8.8.8, nextDNS etc
This is not "done right" by any stretch but it's extremely low effort to set up and has never once failed me, unlike countless complex meshy things.
I use the form of hostname.int.example.com for everything inside my home network. None of which is accessible to the outside world. I use LetsEncrypt with DNS validation to get the certificates.
Removing attack surface is better than trying to hide it.
I'd prefer this over split DNS, any day.
* with some notable root certs that I have… questionable… trust and confidence are not simply controlled by certain state actors.
Or even a more extreme example: https://crt.sh/?id=27555237869 (sorry for any possible crt.sh downtime) - the domain name in question never existed in public or private DNS by itself. It is used only for a WPA3-Enterprise network, as the CN that WiFi clients expect to be present in the RADIUS server certificate, but never resolve. In the public DNS, only the "_acme-challenge" TXT record exists.
The point was that you can obtain a certificate for a domain name without creating any records other than the _acme-challenge TXT record. I.e., that the domain might be completely empty all the time except for this record.
I also use Tailscale so I configure my DNS to use my Tailscale IP addresses. If you don’t want to expose them on a public DNS server you can add them only to an internal DNS server.
See also perhaps DNS aliasing in case you are not able to dynamically update your 'primary' domain, but can update a secondary or sub-domain:
* https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...
So if "example.com" is control by Corporate IT, and they don't want 'random' folks fiddling with it, then you can create a "dnsauth.example.com" and point the dns-1 challenge record from "…foo.example.com" to "foo.dnsauth.example.com" (or a completely different domain, like "…example.net").
There are DNS servers written strictly focused on this use case:
* https://github.com/acme-dns/acme-dns
Also code that handles a bunch of DNS provider APIs so you don't have to roll your own for ACME client hooks:
https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
I get it, I could just do *.mydomain.com and slap that wildcard cert everywhere, but it's still in the public logs..
Specifically grafana is nice to be able to see on the phone, and split horizon DNS and corp VPN is a hassle, to say the least, on phones.
I bet you can do it with HA-Proxy, but I use https://github.com/ThomasHabets/sni-router
But on hosts you control, you should absolutely provision them with an identity and join the local CA. You're going to need it for a multitude of other reasons.
In that case there's no need to validate anything as names, dns records, certificates and anything else should already be in place.
Unless you enjoy that sort of thing.
Most browsers support trust on first use for leaf certs
And later if something changes, then they can do the whole DOING SOMETHING NASTY! thing, which is effectively the experience today
Using a browser in an air gapped environment is so much more pain than it should be.
* "internal services" = on a single server that is publicly routable
You now also have to build infrastructure to distribute the wildcard from (presumably) central place where you generate it to all the different places where it is desired.
And hope the wildcard's private key does not leak from one of myriad of places it now lives.
Leaking is an issue but we're talking about internal services too.
1. Register a domain ("server.com") and put it on some public DNS that can do DNS validation with acme.sh.
2. Use DNS validation to get a certificate on your domain from Let's Encrypt. You can just grab a wildcard one ("*.server.com").
3. CNAME all of your services on a public DNS to an internal address ("email.server.com" → "server.internal", "plex.server.com" → "server.internal").
4. Resolve your internal address on a local DNS server with an A record ("server.internal" → 192.168.0.123). This can often just be done on your router.