Hopefully you all can help!
I’ve been to hundreds of threads over the last few days trying to puzzle this out, with no luck.
The problem:
- Caddy v2 with acme HTTP-1 ACME challenge (Changed from TLS-ALPN challenge)
- Cloudflair DNS with proxy ON
- All cloudflair https is off
- This is a .co domain
Any attempt to get certificates fails with an invalid challenge response. If I try and navigate (or curl) to the challenge directly I always get SSL validation errors as if all the requests are trying to upgrade to HTTPS.
I’m kind of at my wit’s end here and am running out of things to try.
If I turn Cloud flare proxy off and go back to TLS-ALPN challenge, everything works as expected. However I do not wish to expose myself directly and want to use the proxy.
What should I be doing?
I have now solved this by using Cloudflair DNS ACME challenge. Cloudflair SSL turned back on. Everything works as expected now, I can have external clients terminate SSL at cloudflair, cloudflair communicate with my proxy through HTTPS, and have internal clients terminate SSL at caddy.
It is easier to think of the SSL termination in legs.
- Client to Cloudflare; if you’re behind orange cloud, you get this for free, don’t turn orange cloud off unless you want to have direct exposure.
- Cloudflare to your sever; use their origin cert, this is easiest and secure. You can even get one made specific so your subdomains, or wildcard of your subdomain. Unless you have specific compliance needs, you shouldn’t need to turn this off, and you don’t need to roll your own cert.
- Your reverse proxy to your apps; honestly, it’s already on your machine, you can do self signed cert if it really bothers you, but at the end of the day, probably not worth the hassle.
If, however, you want to directly expose your service without orange cloud (running a game server on the same subdomain for example), then you’d disable the orange cloud and do Let’s Encrypt or deploy your own certificate on your reverse proxy.
Having an HTTPS enabled server behind a proxy means the server is connecting with the proxy endpoint. That’s not how HTTPS works. If you want HTTPS enabled for a server BEHIND a proxy, you would do it at the original hand off (forward proxy), in this case meaning the Cloudflare proxy that clients connect to.
You’re seeing the errors because the proxy backend is being told to speak HTTPS with Caddy, and it doesn’t work like that.
I am doing SSL termination at the handoff which is the caddy proxy. My internal servers have their SSL terminated at caddy, my traffic does not go to the internet… It loops back from my router to my internal Network.
However DNS still needs to have subdomains in order to get those certificates, this cloudflair DNS. I do not want my IP to be associated with the subdomains, thus exposing it, therefore cloudflair proxy.
You’re seeing the errors because the proxy backend is being told to speak HTTPS with Caddy, and it doesn’t work like that.
You can have SSL termination at multiple points. Cloudflare can do SSL termination and Cloudflair can also connect to your proxy which also has SSL termination. This is allowed, this works, I have services that do this already. You can have SSL termination at every hop if you want, with different certificates.
That said, I have cloudflair SSL off, as stated in the OP. Cloudflare is not providing a cert, nor is it trying to communicate with my proxy via HTTPS.
Contrary to your statement about this not working that way, cloudflair has no issues proxying to my proxy where I already have valid certs. Or even self signed ones, or even no certs. The only thing that doesn’t work is the ACME challenge…
Edit: I have now solved this by using Cloudflair DNS ACME challenge. Cloudflair SSL turned back on. Everything works as expected now, I can have external clients terminate SSL at cloudflair, cloudflair communicate with my proxy through HTTPS, and have internal clients terminate SSL at caddy.
If you’re behind Cloudflare, don’t. Just get an origin certificate from CF, it’s a cert that CF trust between itself and your server. By using Cloudflare you’re making Cloudflare responsible for your cert.
I stated in the OP that cloudflair HTTPS is off :/
I’m not using cloudflare for the certificate. I also can’t use the cloud for certificate anyways for internal services through a loopback.
Similarly you can have SSL termination at multiple layers. That’s works I have services that proxy through multiple SSL terminations. The issue that I’m having is that the ACME challenge seems to be having issues, these issues are documented and explained in various GitHub threads, however the set of solutions are seemingly different and convoluted for different environments.
This is why I’m asking this question here after having done a reasonable amount of research and trial and error.
I would have a look at your encryption mode on cloudflare, it could be set so that its trying to force https.
https://developers.cloudflare.com/ssl/origin-configuration/ssl-modes/
I stated in the OP that this is off :/
Caddy has been kicking my ass also!
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters CF CloudFlare DNS Domain Name Service/System HTTP Hypertext Transfer Protocol, the Web HTTPS HTTP over SSL IP Internet Protocol SSL Secure Sockets Layer, for transparent encryption
[Thread #977 for this sub, first seen 18th Sep 2024, 03:15] [FAQ] [Full list] [Contact] [Source code]
In this episode of “Doing everything the hardest way possible.”
Cloudflare has SSL, y u need mo?
Because the majority of my traffic and services are internal with internal DNS? And I want valid HTTPS certs for them, without exposing my IP in the DNS for those A records.
If I don’t care about leaking my IP in my a records then this is pretty easy. However I don’t want to do this for various reasons. One of those being that I engage in security related activities and have no desire to put myself at risk by leaking.
Even services that I exposed to the internet I still don’t want to have my local network traffic go to the internet and back when there is no need for that. SSL termination at my own internal proxy solves that problem.
I now have this working by using the cloudflare DNS ACME challenge. Those services which I exposed to the internet cloudflare is providing https termination for, cloudflare is then communicating with my proxy which also provides https termination. My internal communication with those services is terminated at my proxy.
Can’t speak for OP but I was also attempting this and couldn’t get it working. My use case is that CF tunnels make multiple of my self hosted services available on the Internet via HTTPS and without directly exposing my home IP.
It does however mean that even when I use a service on my home network, everything is being proxied through CF which makes things much slower than they need to be 90% of the time. So my idea is to use caddy in parallel to CF and have a local DNS server point to my homelab, thereby circumventing the proxy whenever I’m on my home network.
But like I said I could not get this working just yet.
You’ll need to disable proxy, run certbot, then re enable proxy.
LE won’t sign a site already cf encrypted, or behind cf (even with cf SSL disabled.)
You could try a DNS challenge or other method.
I run the setup you’re aiming for, and as the other guy said, DNS challenge is the way to go. That’s what I do, and it works beautifully. It even works with Caddy auto-https, you just need to build Caddy with the cloudflare-dns plugin.
