How to get SSL/TLS certificates at UCC for that sweet HTTPS (and other) goodness.

Let's Encrypt

Since the launch of the free SSL/TLS certificate authority Let's Encrypt, UCC has been saving money and SSLising everything by using it.

LE was (is?) revolutionary in that it is free and completely automated. The exact method you want to use depends on which machine and which service you are getting a certificate for. We use the official Certbot client (UCC systems were previously set up with acmetool, but this is no longer maintained and the official client is fine now). The certificates are automatically renewed every two months as they are only good for three months.

On new machines, install certbot by using apt install certbot.

HTTP Challenges

If the host is reachable by HTTP/HTTPS after TheCloudflarening, Letsencrypt HTTP challenges can be used.

You will probably also want one of the following:

  • python3-certbot-apache (for automatic verification and setup with Apache)

  • python3-certbot-nginx (for automatic verification and setup with Nginx)

New VMs

If you don't have an existing webserver setup that you are worried about breaking, you can use the Fully Automatic method (apache shown below, can also use nginx): 

# apt install certbot python3-certbot-apache
# Create an account - use your personal email here if this is not an official UCC server
certbot register --agree-tos -m [email protected]
# Using the apache installer, get certificates for somehost in both usual domains
certbot run --apache --domains somehost.ucc.asn.au,somehost.ucc.gu.uwa.edu.au

That's it! You're done. The Debian packages set up automatic renewal and reinstallation of the certificates.

Existing web servers (eg Mussel)

Although we have good backups and version control, nobody has ever tested whether the core webservers handle the automagic configuration method. If it breaks, you get to put the pieces back together.

If you aren't willing to be the first, or you need the certificate for another service (eg mail server) you can use the certonly command with the preconfigured webroot, like this: 

certbot certonly --webroot --webroot-path /services/ssl/acme-challenge/ -d cooluserhostingatucc.id.au,www.cooluserhostingatucc.id.au

/services/ssl/acme-challenge is set up on Mussel and Mooneye so that the various redirects and firewalls don't make it too hard to get the right certificates. Otherwise, use a webroot that you have configured in the server software such as: 

# cat /etc/apache2/conf-available/letsencrypt.conf
Alias "/.well-known/acme-challenge/" "/var/www/.well-known/acme-challenge/"
<Directory "/var/www/.well-known/acme-challenge/">
  AllowOverride None
  Options None
  Require all granted
</Directory>
# a2enconf letsencrypt

certbot certificates will show you the paths you need for the Apache config.

While the automatic issuers know how to restart the services you are interested in, certbot certonly does not. You may need to create a restart hook, which lives in /etc/letsencrypt/renewal-hooks/deploy/. It can be as simple as systemctl reload apache2, or much more complicated. Ask for help if you need it.

DNS challenges and wildcard certificates

/!\ Danger Will Robinson /!\

Improperly configured DNS or disclosure of dynamic DNS update keys can lead to domain compromise. Don't hesitate to ask for help if you are not sure.

Most of the time, Let's Encrypt/certbot will use a Web server to prove that the machine owns the domain it is using. Sometimes this is impossible (due to firewalling, wildcard certificates, multiple machines sharing a hostname). If necessary, DNS verification can be used.

Previously UCC used RFC2136 DNS updates to assist with DNS verification. This no longer works post TheCloudflarening. We have to either use a Cloudflare API key, or delegate the DNS challenges and use a helper to update the DNS appropriately.

Wildcard certs using Cloudflare API key

These are currently being used on mailfish, mussel, and motsugo. Primarily, hosts with large numbers of virtual hosts. As this uses the Cloudflare API, this should be used on trusted hosts only where absolutely necessarily. It is better to use the alternatively below in most other circumstances.

These hosts have been configured using the https://gitlab.ucc.asn.au/ucc-systems/ucc-ansible-soe UCC Ansible SOE. This contains a ucc_wildcard_cert role.

The role does several things

  • If the version of certbot is older than 1.6, build a more recent version
  • Installs python3-certbot-dns-cloudflare plugin
  • Grabs the Cloudflare API key out of uccpass
  • Places the credentials on the host
  • Runs certbot to generate the cert 

    certbot certonly \
    --dns-cloudflare \
    --dns-cloudflare-credentials /etc/letsencrypt/dns-cloudflare-credentials.ini \
    --email [email protected] \
    --server https://acme-v02.api.letsencrypt.org/directory \
    --agree-tos \
    -d ucc.asn.au \
    -d ucc.gu.uwa.edu.au \
    -d *.ucc.asn.au \
    -d *.ucc.gu.uwa.edu.au

Non-Wildcard certs using Quovadis API helper

As an alternative to using the Cloudflare API key, a REST-like API helper has been created to update the delegated DNS text records to perform the ACME challenge.

The quovadis tool can be accessed at https://quovadis.ucc.asn.au/quovadis/

  • Provides an API to update _acme_challenge DNS entries (delegated via a CNAME in the ucc.machines / zonemake.py setup)
  • Uses desec.io to host a zone that can be updated via an API for the challenges
  • Has an example certbot helper script

The code for the Quovadis tool itself can be found in the UCC gitlab, https://gitlab.ucc.asn.au/UCC/quovadis/. You shouldn't have to touch it in normal circumstances.

Quovadis has a relatively minimal API. The username used by the API is your UCC username.

Setup

curl -X POST -d username=accmurphy https://quovadis.ucc.asn.au/quovadis/register

This will cause your API key to be sent to your UCC email address. For UCC core systems, use "root". The API key for "root" is stored in uccpass.

It is possible to update the API key via an API call. 

curl -X POST -d username=accmurphy -d api_key=foo -d new_api_key=bar https://quovadis.ucc.asn.au/quovadis/update-api-key

Use

curl -X POST -d username=accmurphy -d api_key=foo -d challenge=thingy -d value=123456abcdef https://quovadis.ucc.asn.au/quovadis/update-challenge

The easiest way to use the API is by a certbot hook script. 

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory \
        --manual \
        --preferred-challenges=dns \
        --manual-auth-hook /etc/letsencrypt/ucc-hooks/certbot-hook-quovadis.sh \
        -d thehostname.ucc.asn.au

The hook script, and an example config file, can be downloaded from:

To install 

mkdir /etc/letsencrypt/ucc-hooks
cp certbot-hook-quovadis.sh /etc/letsencrypt/ucc-hooks

To configure, create a config file config.sh in the same directory as the script containing 

QV_API_KEY=<your API key>
UCC_USERNAME=<your UCC username>

A DNS delegation must be setup for the ACME challenge. In ucc.machines, this looks like: 

[ipsec]
A: 1
zones: ucc asn

[_acme-challenge.ipsec]
CNAME: root-ipsec.quovadis-challenges.ucc.asn.au.
zones: ucc asn

In this example ipsec is the relevant hostname (corresponding to the challenge argument in the API), root is the username

To use the script, runs it as a manual auth hook with certbot. An example command line is: 

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory \
                 --manual \
                 --preferred-challenges=dns \
                 --manual-auth-hook /etc/letsencrypt/ucc-hooks/certbot-hook-quovadis.sh \
                 -d example.ucc.asn.au -d example.ucc.gu.uwa.edu.au

Troubleshooting

- Automatic certbot renew renewals may fail until they have been manually run once with: certbot renew --manual-public-ip-logging-ok

  • - This setting is recorded in /etc/letsencrypt/renewal/mudkip.ucc.asn.au.conf

    • - manual_public_ip_logging_ok = True

Post certificate generation things

Run certbot certificates to see it again.

Don't forget to set up a reload hook. Here is one for strongSwan in /etc/letsencrypt/renewal-hooks/deploy/reload-strongswan: 

[ -f /run/starter.charon.pid ] && kill -s USR1 $(cat /run/starter.charon.pid) && /usr/sbin/ipsec rereadall

For a wildcard certificate, just add or change the --domains argument above (eg --domains secure.ucc.asn.au,\*.secure.ucc.asn.au). Be very careful with the key material that is generated, as wildcard certificates pose extra security risks.

-- CategorySystemAdministration

uccwiki: SSLCertificates (last edited 2023-12-04 19:25:34 by JamesArcus)

MoinMoin Powered | Python Powered
Written to be specifications compliant.

UCC

Wiki content is publicly editable, and as such does not necessarily reflect
the views of the University Computer Club, the Guild or the University of
Western Australia.

Contact the webmasters.