Selfhosted

Self signed for this use case is fine. you know and trust both ends of your connection, and no one else needs to know or trust either end of the connection.

You don't need a cert authority to self-sign. You can do it once and be good to go.

You can just use openssl to generate x509 certificates locally. If you only need to do this for a few local connections, the simplest thing to do is create them manually and then manually place them in the certificate stores for the services that need them. You might get warnings about self-signed certificates/unrecognized CA, but obviously you know why that's the case.

This method becomes a problem when:

1. You need to scale - manually transferring certs is fine maybe half a dozen times, after that it gets real tedious and you start to lose track of where they are and why.
2. You need other people to access your encrypted services - self-signed certs won't work for public access to an HTTPS website because every visitor will get a warning that you're signing your own encryption certs, and most will avoid it. For friends and family you might be able to convince them that your personal cert is safe, but you'll have to have that conversation every time.
3. You need to implement expiration - the purpose of cert expiration is to mitigate the damage if the cert private key leaks, which happens a lot with big companies that have public-facing infrastructure and bad internal security practices (looking at you, Microsoft). As an individual, it is still worthwhile to update your certs every so often (e.g. every year) if for no other reason than to remind yourself how your SSL infrastructure is connected. It's up to you whether or not it's worth the effort to automate the cert distribution.

I've used Letsencrypt to get certs for the proxy, but the traffic between the proxy and the backend is plain HTTP still. Do I need to worry about securing that traffic considering its behind a VPN?

In spite of things you may have read, and the marketing of VPN services, a VPN is NOT a security tool. It is a privacy tool, as long as the encryption key for it is private.

I'm not clear on what you mean by "between the proxy and the backend". Is this referring to the VPS side, or your local network side, or both?

Ultimately the question is, do you trust the other devices/services that might have access to the data before it enters the VPN tunnel? Are you certain that nothing else on the server might be able to read your traffic before it goes into the VPN?

If you're talking about a rented VPS from a public web host, the answer should be no. You have no idea what else might be running on that server, nor do you have control over the hypervisor or the host system.

is there an easier way to do self-signed certs besides spinning up your own certificate authority?

Letsencrypt works fine, just use a "real" domain and DNS challenge.

Your service will need to be on the "real" domain, but it won't need to be accessible externally and you won't need a public DNS entry for it (of course your VPS will still need to be able to resolve the backend's name).

As continued from my answer for ypur previous post I suggest you route pure TCP traffic all the way to your backend and terminate TLS (with a Let's Encrypt cert) there. In fact, I prefer not to mount any certs on the VPS. This does not involve separate certs nor internal domains.

For a domain you own, you can use Let's Encrypt. If it's a custom TLD (.lan, etc.) then you need to do self-signed. Most systems can install certificates.

If that traffic is going through an encrypted Wireguard tunnel, I don't see a reason to encrypt it a second time. Judging by your description, it's already encrypted on transport between the router and VPS. HTTPS would add nothing there. It will however add encryption within your DMZ, if you expect something nefarious going on within your DMZ.

If your DNS host has an API, you can get any certificate you like for the host.

e.g. a cert for server.example.com

Even though that host doesn't exist in public DNS

@early_riser @jwildeboer has a blog post about using step-ca for something like this.

https://jan.wildeboer.net/2025/07/letsencrypt-homelab-stepca/

Self signed certs are usually created with OpenSSL. Find an example online. If you own a domain create your cert against that name.

The better option is to get your backend also using let’s encrypt and change to https. The whole point of lets encrypt is “encrypt all the things”

You should be able to fix your browser cert error messages by adding the cert to your trusted root store. Easy to do on desktops, mobile devices might be harder to do without an MDM.

What I've usually seen is that the VPS does TLS termination and then comms between the VPS and the LAN are sent http, but still secure due to traveling through the VPN. This is the easiest way if you don't require full e2ee and trust your LAN

frp has an option to encrypt the tunnel