How to get SSL/TLS certificates at UCC for that sweet HTTPS (and other) goodness.

= Let's Encrypt =
Since the launch of the free SSL/TLS certificate authority [[https://letsencrypt.org/|Let's Encrypt]], UCC has been saving money and SSLising everything by using it.

LE was (is?) revolutionary in that it is free and completely automated. The exact method you want to use depends on which machine and which service you are getting a certificate for. We use the official Certbot client (UCC systems were previously set up with [[https://github.com/hlandau/acme/|acmetool]], but this is no longer maintained and the official client is fine now). The certificates are automatically renewed every two months as they are only good for three months.

On new machines, install certbot by using `apt install certbot`. 

== HTTP Challenges ==

If the host is reachable by HTTP/HTTPS after [[TheCloudflarening]], Letsencrypt HTTP challenges can be used.

You will probably also want one of the following:
 * `python3-certbot-apache` (for automatic verification and setup with Apache)
 * `python3-certbot-nginx` (for automatic verification and setup with Nginx)


=== New VMs ===
If you don't have an existing webserver setup that you are worried about breaking, you can use the Fully Automatic method (apache shown below, can also use nginx):

{{{
# apt install certbot python3-certbot-apache
# Create an account - use your personal email here if this is not an official UCC server
certbot register --agree-tos -m wheel@ucc.gu.uwa.edu.au
# Using the apache installer, get certificates for somehost in both usual domains
certbot run --apache --domains somehost.ucc.asn.au,somehost.ucc.gu.uwa.edu.au
}}}

That's it! You're done. The Debian packages set up automatic renewal and reinstallation of the certificates.

=== Existing web servers (eg Mussel) ===

Although we have good backups and version control, nobody has ever tested whether the core webservers handle the automagic configuration method. If it breaks, you get to put the pieces back together.

If you aren't willing to be the first, or you need the certificate for another service (eg mail server) you can use the `certonly` command with the preconfigured webroot, like this:

{{{
certbot certonly --webroot --webroot-path /services/ssl/acme-challenge/ -d cooluserhostingatucc.id.au,www.cooluserhostingatucc.id.au
}}}

`/services/ssl/acme-challenge` is set up on Mussel and Mooneye so that the various redirects and firewalls don't make it too hard to get the right certificates. Otherwise, use a webroot that you have configured in the server software such as:

{{{
# cat /etc/apache2/conf-available/letsencrypt.conf
Alias "/.well-known/acme-challenge/" "/var/www/.well-known/acme-challenge/"
<Directory "/var/www/.well-known/acme-challenge/">
  AllowOverride None
  Options None
  Require all granted
</Directory>
# a2enconf letsencrypt
}}}

`certbot certificates` will show you the paths you need for the Apache config.

While the automatic issuers know how to restart the services you are interested in, `certbot certonly` does not. You may need to create a restart hook, which lives in `/etc/letsencrypt/renewal-hooks/deploy/`. It can be as simple as `systemctl reload apache2`, or much more complicated. Ask for help if you need it.

== DNS challenges and wildcard certificates ==



{{{#!wiki warning
/!\ '''Danger Will Robinson''' /!\

Improperly configured DNS or disclosure of dynamic DNS update keys can lead to domain compromise. Don't hesitate to ask for help if you are not sure. 
}}}

Most of the time, Let's Encrypt/certbot will use a Web server to prove that the machine owns the domain it is using. Sometimes this is impossible (due to firewalling, wildcard certificates, multiple machines sharing a hostname). If necessary, DNS verification can be used.

Previously UCC used RFC2136 DNS updates to assist with DNS verification.  This no longer works post [[TheCloudflarening]].   We have to either use a Cloudflare API key, or delegate the DNS challenges and use a helper to update the DNS appropriately.

=== Wildcard certs using Cloudflare API key ===

These are currently being used on mailfish, mussel, and motsugo.   Primarily, hosts with large numbers of virtual hosts.   As this uses the Cloudflare API, this should be used on trusted hosts only where absolutely necessarily.   It is better to use the alternatively below in most other circumstances.

These hosts have been configured using the [[https://gitlab.ucc.asn.au/ucc-systems/ucc-ansible-soe UCC Ansible SOE]].  This contains a ucc_wildcard_cert role.  

The role does several things
 * If the version of certbot is older than 1.6, build a more recent version
 * Installs python3-certbot-dns-cloudflare plugin
 * Grabs the Cloudflare API key out of uccpass
 * Places the credentials on the host
 * Runs certbot to generate the cert

{{{
    certbot certonly \
    --dns-cloudflare \
    --dns-cloudflare-credentials /etc/letsencrypt/dns-cloudflare-credentials.ini \
    --email wheel@ucc.asn.au \
    --server https://acme-v02.api.letsencrypt.org/directory \
    --agree-tos \
    -d ucc.asn.au \
    -d ucc.gu.uwa.edu.au \
    -d *.ucc.asn.au \
    -d *.ucc.gu.uwa.edu.au
}}}


=== Non-Wildcard certs using Quovadis API helper ===

As an alternative to using the Cloudflare API key, a REST-like API helper has been created to update the delegated DNS text records to perform the ACME challenge.

The quovadis tool can be accessed at [[ https://quovadis.ucc.asn.au/quovadis/ ]]

 * Provides an API to update _acme_challenge DNS entries (delegated via a CNAME in the ucc.machines / zonemake.py setup)
 * Uses desec.io to host a zone that can be updated via an API for the challenges
 * Has an example certbot helper script

The code for the Quovadis tool itself can be found in the UCC gitlab, [[ https://gitlab.ucc.asn.au/UCC/quovadis/ ]]. You shouldn't have to touch it in normal circumstances. 

Quovadis has a relatively minimal API.   The username used by the API is your UCC username.

==== Setup ====

{{{

curl -X POST -d username=accmurphy https://quovadis.ucc.asn.au/quovadis/register

}}}

This will cause your API key to be sent to your UCC email address.    For UCC core systems, use "root".  The API key for "root" is stored in uccpass.

It is possible to update the API key via an API call.

{{{

curl -X POST -d username=accmurphy -d api_key=foo -d new_api_key=bar https://quovadis.ucc.asn.au/quovadis/update-api-key

}}}

==== Use ====

{{{

curl -X POST -d username=accmurphy -d api_key=foo -d challenge=thingy -d value=123456abcdef https://quovadis.ucc.asn.au/quovadis/update-challenge

}}}

The easiest way to use the API is by a certbot hook script.

{{{


certbot certonly --server https://acme-v02.api.letsencrypt.org/directory \
        --manual \
	--preferred-challenges=dns \
        --manual-auth-hook /etc/letsencrypt/ucc-hooks/certbot-hook-quovadis.sh \
        -d thehostname.ucc.asn.au

}}}

The hook script, and an example config file, can be downloaded from:

 * [[ https://gitlab.ucc.asn.au/UCC/quovadis/-/blob/master/scripts/certbot-hook-quovadis.sh ]]
 * [[ https://gitlab.ucc.asn.au/UCC/quovadis/-/blob/master/scripts/config.sh ]]

To install

{{{

mkdir /etc/letsencrypt/ucc-hooks
cp certbot-hook-quovadis.sh /etc/letsencrypt/ucc-hooks

}}}

To configure, create a config file config.sh in the same directory as the script containing

{{{
QV_API_KEY=<your API key>
UCC_USERNAME=<your UCC username>
}}}

A DNS delegation must be setup for the ACME challenge.  In ucc.machines, this looks like:

{{{

[ipsec]
A: 1
zones: ucc asn

[_acme-challenge.ipsec]
CNAME: root-ipsec.quovadis-challenges.ucc.asn.au.
zones: ucc asn

}}}

In this example ipsec is the relevant hostname (corresponding to the challenge argument in the API), root is the username


To use the script,  runs it as a manual auth hook with certbot.  An example command line is:

{{{

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory \
                 --manual \
                 --preferred-challenges=dns \
                 --manual-auth-hook /etc/letsencrypt/ucc-hooks/certbot-hook-quovadis.sh \
                 -d example.ucc.asn.au -d example.ucc.gu.uwa.edu.au

}}}

== Troubleshooting ==

- Automatic `certbot renew` renewals may fail until they have been manually run once with: `certbot renew --manual-public-ip-logging-ok`
  - This setting is recorded in `/etc/letsencrypt/renewal/mudkip.ucc.asn.au.conf`
    - `manual_public_ip_logging_ok = True`

== Post certificate generation things ==

Run `certbot certificates` to see it again.

Don't forget to set up a reload hook. Here is one for strongSwan in `/etc/letsencrypt/renewal-hooks/deploy/reload-strongswan`:

{{{
#! /bin/sh
[ -f /run/starter.charon.pid ] && kill -s USR1 $(cat /run/starter.charon.pid) && /usr/sbin/ipsec rereadall
}}}

For a wildcard certificate, just add or change the `--domains` argument above (eg `--domains secure.ucc.asn.au,\*.secure.ucc.asn.au`). Be very careful with the key material that is generated, as wildcard certificates pose extra security risks.

--
CategorySystemAdministration