OpenSSH Certificates: Configuration and Usage (Part 2)

For those who haven’t read part 1 of this series, here’s a short summary:

OpenSSH key-based authentication is divided into two parts; public key authentication and certificate authentication. Certificate authentication has a central CA, trusted by the hosts, which can issue certificates based on a users public key to provide short-lived or long-lived access to a server. There are some commercial and open-source tools available for implementation, all have their pros and cons.

Setting up your CA

The first step of setting a server up for certificate authentication is creating your SSH CA. With ssh-keygen, this can simply be done like this:

# Create an ECDSA key with a size of 521, a comment of "CA" and the filename ssh_ca
ssh-keygen -t ecdsa -b 521 -C CA -f ssh_ca

After this command, you have a file called ssh_ca in your current folder which contains more or less a standard SSH key. This can now be used as a certificate authority!

Trust the CA

To be able to use the CA to sign keys for authentication, the server that you’re going to connect to needs a copy of the ssh_ca public key and a couple of rows of configuration in the OpenSSH configuration file.

Start by copying the ssh_ca.pub file to your server, placing it (for example, you can put this basically anywhere) at /etc/ssh/ssh_ca.pub (Or C:\ProgramData\ssh\ssh_ca.pub on Windows)

Then, you’re going to specify to the ssh service on the machine that it can trust the certificates issued by this authority. You do this by editing the /etc/ssh/sshd_config (C:\ProgramData\ssh\sshd_config on Windows) file and adding the following section:

# Linux
TrustedUserCAKeys /etc/ssh/ssh_ca.pub

# Windows
TrustedUserCAKeys C:\ProgramData\ssh\ssh_ca.pub

Then, save the file and restart the SSH service:

# Linux (systemd)
systemctl restart sshd

# Linux (other)
/etc/init.d/ssh restart

# Windows
net stop "OpenSSH SSH Server"
net start "OpenSSH SSH Server"

You’ve now established what’s called a chain of trust. The server will trust that you, the CA, only verifies keys that are actually allowed to access the server. This also means that the ssh_ca private key is now as valuable as the passwords, secrets or private keys that can be used directly to access the server as long as the public key is trusted by a server.

Issue a certificate from the CA

Now, you just need to certify a user to be able to connect to the server. The prerequisites for this is that there is a private and public key for the user, the contents of which do not need to be on the server in the authorized_keys file. This means you could also generate this keypair on-demand, increasing the rotation of the key and minimizing the damage in case it’s compromised.

The command for this is the same as creating the CA. You can use the same key type as the CA (ECDSA), or you can use one of the other key types available (RSA, DSA or ED25519):

# Create an ECDSA key with a size of 521, a comment of "username" and the filename ssh_user
ssh-keygen -t ecdsa -b 521 -C "Username" -f ssh_user

If you at this time try to use this key to connect to the server, the authentication should fail and it should either prompt you for the password or deny the login if you have password authentication completely disabled.

Now, you can issue a certificate from the CA certifying that this public/private keypair is allowed to access the server. You do this by issuing the following command:

# Sign the public key with the following options:
# -s ssh_ca: Specify which private key should be used for the signing
# -I username@domain: This is the Key ID, meant to specify which user the certificate is issued to for auditing
# -z 1: This is the serial number of the key, used to keep track of certificates on the CA side
# -V +12h: The validity period for the key, this specifies it to 12 hours from now
# ssh_user.pub: Which public key should be permitted access to the server

ssh-keygen \
-s ssh_ca \
-I username@domain \
-z 1 \
-V +12h \
ssh_user.pub

This will save the certificate to the file ssh_user-cert.pub in the current directory. You can verify the contents of the certificate with the following command:

$ ssh-keygen -Lf ssh_user-cert.pub

  Type: [email protected] user certificate
  Public key: ECDSA-CERT ...
  Signing CA: ECDSA ...
  Key ID: "username@domain"
  Serial: 1
  Valid: from 2022-01-01T12:00:00 to 2022-01-01T23:59:59
  Principals: (none)
  Critical Options: (none)
  Extensions:
    permit-X11-forwarding
    permit-agent-forwarding
    permit-port-forwarding
    permit-pty
    permit-user-rc

Now, you should be able to connect to your server with the same private key that failed just a couple of steps ago. If everything worked as intended, you should be able to access the server!

Now that you’ve completed your first certificate authentication, let’s have a look at what other options ssh-keygen and the configuration provide.

Additional Options – Principals

The Principals-part of the certificate is used for access control to certain users and/or servers. For example, if you were to integrate a solution for certificate issuance with your Active Directory, one of the Allowed Principals on the server might be an active directory group, which gets put in the principals-part of the certificate if a user is a member of it.

If that group name is specified server-side as allowed any user of that group can now access the server, but if it’s not included in the list permission will be denied.

An important note on this is that if you leave the principals-field empty during the issuing process, it is interpreted as an all-access pass instead of denying access to all servers, meaning you should always put a value in this field (for example “no-servers”) even if the user shouldn’t have access to any servers.

To specify the principals on the host, you can configure this in a couple of different ways:

# You can specify a file that contains a list of principals allowed to access all accounts on the server
# Create the file with the principals "principal1" and "server-group-prod"
$ echo -e "principal1\nserver-group-prod" > /etc/ssh/ssh_principals

# Allow these principals access to all accounts
AllowedPrincipals /etc/ssh/ssh_principals

# Use a different file for each user account on the server
$ echo -e "principal1\nserver-group-prod" > /etc/ssh/ssh_principals.d/root
$ echo -e "principal2\nserver-group-prod" > /etc/ssh/ssh_principals.d/anotheruser

# Check the respective file for each user
AllowedPrincipals /etc/ssh/ssh_principals.d/%u

# Fetch a principal file from another server (Fetched on each login)
AllowedPrincipalsCommand curl -sb https://server.com/get_principals

# Fetch a different principals file for each user (Fetched on each login)
AllowedPrincipalsCommand curl -sb https://server.com/get_principals/user/%u

# Specify which user should run the AllowedPrincipalsCommand
AllowedPrincipalsCommandUser different-user

On signing, principals are added to the command as specified below:

# Add a single principal for the user with the following attribute
# -n principal1
#
# Add multiple principals separated by a comma
# -n user-abcdefg,server-group-prod,server-group-development
ssh-keygen \
-s ssh_ca \
-I username@domain \
-z 1 \
-V +12h \
-n user-abcdefg,server-group-prod,server-group-development \
ssh_user.pub

Additional Options – Extensions

A list of optional extensions that the user is allowed to utilize. This can also be used together with for example PAM-USSH to specify if a user is allowed to use sudo on the machine. The usable options for this part are:

• no-touch-required (Signatures made with the certificate requirer no additional verification)
• permit-X11-forwarding (Indicates that X11 forwarding will be permitted, denied if absent)
• permit-agent-forwarding (Indicates that agent forwarding should be allowed, denied if absent)
• permit-port-forwarding (Indicates that port forwarding should be permitted, denied if absent)
• permit-pty (Indicates PTY allocation should be permitted, denied if absent)
• permit-user-rc (Indicating that the execution of ~/.ssh/rc should be permitted, denied if absent)

These are specified together with the certificate signing command as follows:

# Add permit-port-forwarding to extensions
# -O extension:permit-port-forwarding
#
# Add multiple options to extensions
# -O extension:permit-port-forwarding -O extension:permit-user-rc -O extension:permit-pty
#
ssh-keygen \
-s ssh_ca \
-I username@domain \
-z 1 \
-V +12h \
-O extension:permit-port-forwarding \
-O extension:permit-user-rc \
-O extension:permit-user-rc \
ssh_user.pub

Additional Options – Critical Options

Like extensions, this can be used for extra attributes to specify permissions on the server. The meaning of “Critical” in this context means that an implementation must refuse to authorize a key that has unrecognized options specified here. The recognized options at present are:

• force-command (Lock the user into a specific piece of software, e.g. internal-sftp or a specific shell, and ignore any commands that are sent along with the ssh command)
• source-address (Specify the source addresses from which the user can connect with this certificate)
• verify-required (Asserts that additional verification is required before usage, such as touching a yubikey or other hardware key to verify user presence)

These are specified together with the certificate signing command as follows:

# Add force-command option for sftp users
# -O critical:force-command=internal-sftp
#
# Add multiple critical options
# -O critical:force-command=internal-sftp -O critical:source-address=10.10.10.0/24
#
ssh-keygen \
-s ssh_ca \
-I username@domain \
-z 1 \
-V +12h \
-O critical:force-command=internal-sftp \
-O critical:source-address=10.10.10.0/24 \
ssh_user.pub

Using PKCS#11 storage for the private key (e.g. Yubikey)

When issuing certificates, you can also retrieve the key from a hardware storage or other PKCS#11-type token by specifying the public key with the -s option and the token library for retrieving the private key with the -D option:

ssh-keygen \
-s ssh_ca.pub \
-D libpkcs11.so \
-I username@domain \
-z 1 \
-V +12h \
-O critical:force-command=internal-sftp \
-O critical:source-address=10.10.10.0/24 \
ssh_user.pub

Certificate Revocation Lists

You can also implement certificate revocation lists in a similar way to how SSL Certificates handle this, meaning the CA or another server publishes a list explicitly stating the serial numbers of the certificates that have been revoked.

This mitigates access to servers because the clock is offset, and gives an extra layer of security to be able to withdraw a certificate before its expiration date for any reason, especially important if certificates are valid longer than a couple of hours.

This part won’t be covered at length in this series, see this excellent write–up by Mr. Gupta at DBSentry for more information at this point, but I might dig into it further in the future.

End of Part 2

This has been a short summary of how to use SSH Certificates with ssh-keygen for authentication towards a server. I hope you learned something, if you’re interested in following the rabbit hole even deeper you can check out Part 3 of the series, A deep-dive into the OpenSSH Certificate format once it becomes available.