Renewing Let’s encrypt certificates with Traefik

Let’s encrypt, the biggest free provider of TLS certificates recently issued a warning to renew all TLS based certificates. Here is how to force a certificate renewal if you are using Traefik as a reverse proxy.

Traefik and let’s encrypt

I already wrote about my home server setup using Traefik as a reverse proxy in the past. This works really well as I can just run my services via Docker and docker-compose and then add a couple of lines to that file to make any service available from the Internet via my own domain. A couple of labels like this

labels:
      - traefik.enable=true
      - traefik.docker.network=traefik_proxy
      - traefik.web.frontend.rule=Host:servicename.${DOMAINNAME}
      - traefik.web.port=80

tell Traefik (I am still using version 1.x) to make the Docker service that runs a software on port 80 available via servicename.mydomain.de. Super simple as Traefik takes care about all the details of routing traffic and of getting and updating a certificate in the background.

The provider for these certificates is the free let’s encrypt service. There are different methods Traefik can use to get these certificates, I decided to use TLS because it works on port 443 (port 80 is used otherwise). It can be enabled in the traefik.toml file via

[entryPoints]
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]

[acme]
  email = "my@email.de"
  storage = "/etc/traefik/acme/acme.json"
  entryPoint = "https"

[acme.tlsChallenge]

The full config and explanation can be found here.

The problem

This process has worked well for years now but today I, as well as many others got this email:

Hello,

Please immediately renew your TLS certificate(s) that were issued from 
Let's Encrypt using the TLS-ALPN-01 validation method and the following 
ACME registration (account) ID(s):

 123456

We've determined that an error made it possible for TLS-ALPN-01 
challenges, completed before today, to not comply with certificate 
issuance requirements. We have remediated this problem and will revoke 
all unexpired certificates that used this validation method at 16:00 UTC 
on 28 January 2022. Please renew your certificates now to ensure an 
uninterrupted experience for your site visitors.

We apologize for any inconvenience this may cause. If you need support 
in the renewal process, please comment on our forum post. Our staff and 
community members are available to help:

https://community.letsencrypt.org/t/170449

Thank you,

The Let's Encrypt Team

So I had to figure out how to manually renew the certificates. A quick check via my browser told me that the certificates would only expire in a month otherwise, so no automatic renewal. Fortunately the fix was easy and with only a very short downtime.

Forcing certificate renewal with Traefik

Forcing a renewal is easy. You need access to the acme.json file that Treafik uses to store the certificate information. The path of that is defined in the traefik.toml file (see the code above). First you should make a copy of this file somewhere else. Then open the file, you should see a list like this:

"Certificates": [
    {
      "Domain": {
        "Main": "some.service.com",
        "SANs": null
      },
      "Certificate": "someVeryLongString>
      "Key": "anotherVeryLongString>
    },
...

Now find and delete all the lines starting with "Certificate". Finally restart Traefik. If you run a setup with docker-compose like I do a simple docker-compose -f traefik.yaml restart traefik should work. Afterwards you will see in the logs that Traefik gets new certificates for all the domains you are using. This took only a couple of seconds in my case, all my other services kept working.

PS there is an official post about this from the Traefik team now, from what I can tell they propose the same solution.