In the previous post we ran ssh-keygen once and then proceeded to dig inside the key files it produces. In this follow-up, we’ll discuss all its capabilities, which go far beyond key generation.

ssh-keygen

If we look at its manual page ssh-keygen(1), we’ll find that it lists 20 invocations of the command, with various options and purposes. Keygen is not just a key generator - it’s a complete tool for managing keys (both user and host keys), revocation lists and known-hosts files.

Generating keys

If ran with no options, it will generate an RSA key and ask where to store it. Adding -f path/basename tells it to save this new key under that filename, together with its public counterpart, suffixed with .pub. When you need something else than RSA, use -t key_type (one of: rsa, dsa, ecdsa, ed25519, the latter two used for elliptic curve cryptography) to specify key type. With -m format, which was used often in the previous part, the keys will be generated in another container format, which may be useful to other tools. For some algorithms you can also pass -b length to generate keys of a given bit length. Larger is generally safer but slower. You probably don’t need this option because ssh comes with reasonable defaults.

The keys may also be encrypted with a passphrase, and keygen will prompt for it. Skip that prompt with -N passphrase (possibly passing a blank passphrase to keep it unencrypted). The key also contains a comment field, by default filled with the current username@hostname, but you can alter it with -C comment. Once generated, change the passphrase with -p -f keyfile -P old_passphrase -N new-passphrase, and the comment with -c -C new_comment.

Inspecting key files

View a fingerprint of your key file with ssh-keygen -l -f keyfile. This will output a SHA256 fingerprint, which can be changed with -E algorithm. A useful choice here is MD5, because that’s how GitHub displays key fingerprints in its management interface. If they match, you have a correct key to authenticate with them. AWS uses yet another format. Another way to display a fingerprint is with -B -f keyfile, which outputs bubblebabble. That format is easier to remember or compare non-visually, as it resembles a string of short nonsense words. One more format is the visual digest, output when adding -v to this command – it’s a small ASCII-art picture.

Importing, converting and exporting keys

To regenerate the public key, invoke keygen with -y -f private_key_file, and to export it in another format use -e -f private_key -m format. Choose between RFC4716 (default), PEM or PKCS#8. You can also do this while changing the passphrase, just append -m format to that command. Private keys can be converted with -i -f private_key_file -m format (which also works for public keys like -e).

If you have a PKCS#11 smartcard plugged in (or a YubiKey, which works like one) with a private key loaded, output its corresponding public key with ssh-keygen -D pkcs_library.so (ykcs11.so or opensc-pkcs11.so, wherever they are installed). This will most likely ask for a PIN first.

Host keys

Invoking with -A generates a set of host keys (which are regular keys stored in a system-wide location, normally unreadable by users) either in the default location, or wherever -f /path/prefix specifies. There is likely no need for this, as it’s done automatically on system setup/provisioning. You can then inspect and convert these files with all the other commands listed above, except that you need to be root to read them.

Known-hosts files

These are an inverse of your .authorized_keys file, listing public host keys of remotes you trust to connect. Usually ssh adds entries to this file automatically and keeps it hashed, so it’s not very human-readable. Search it with -F hostname -f known_hosts_file. The hostname must be fully qualified - partial matches or wildcards won’t work. If you have an old, un-hashed file, convert it with -H -f known_hosts_file. If a remote changes its host key, ssh will warn you (sternly) and advise to remove the mismatched key file, by using -R hostname -f known_hosts_file.

Key fingerprint listing also works for known hosts files: ssh-keygen -l -f known_hosts_file will show fingerprints for all entries (which may still be hashed). All the format options -B -v -E algorithm continue to work here as well.

Certificates and signing

Public key files may optionally be signed with another private key. This is different from HTTPS and should not be confused with it. This process produces certificates, which contain the original public key with some extra data appended and signed with a CA’s private key. There are two flavors of these: user and host certificates, both can be used to set up an alternative style of public key authentication.

User certificates

Signing user keys with a CA key, and then trusting that CA key (by listing its public part in the remote sshd’s TrustedUserCAKeys file) replaces configuring authorized_keys on the remote. Specifically, instead of each target user account having an individual authorized keys file, users bearing a trusted certificate may log in as any of the user accounts stored in the certificate. This requires configuration on the server, and is not enabled by default.

To sign, invoke with -s ca_key -I identity -n username1,username2,... private_key_file. Here, ca_key is the private key used for signing, identity is any string pertaining to the key’s use, e.g. the user’s email address or full name. Option -n specifies principals, which in this case are user names that this user is allowed to authenticate as, and the last argument is that user’s public (or private) key.

Add -V to restrict validity: it takes one or two colon-separated datetimes. If only one given, it is the expiry date; if two then they are start and expiry date respectively. Each can be given as a YYYYMMDD date, YYYYMMDDHHMMSS time, or a time offset starting with + or - followed by numbers and units (no spaces). Units are w for weeks, d for days, m for months. Expiry time can also be given as forever (which is the default, actually) to specify that the certificate does not expire. Even more fine-grained control is available with -O option, which can permit or restrict individual ssh features (e.g. port forwarding), limit connections to a list of source hosts, and even force-replace shell with a different command. Optionally, set the serial number, an additional numeric identifier (a 64-bit integer) with -z.

If you don’t have the CA private key file but it is loaded in the agent, add -U and pass its public key filename into -s. If it’s loaded on a PKCS#11 device (smartcard or authentication key), also supply the public part and add -D pkcs_library.so, as described in the Import/Export section.

This will create a new file, named after your private key file but suffixed with -cert.pub. It’s a companion to your regular private key, and ssh will use it automatically if found. Inspect a certificate with -L -f cert_file.

Host certificates

Signing is done almost the same way, except that the key you sign should be the target machine’s host key, and -h needs to be added to options to tell keygen it’s a host key. Principals are now hostnames (and there should generally be exactly one of them). Afterwards, the certificate needs to be stored on the target machine and configured with sshd’s HostCertificate option.

Next, the known_hosts file on a connecting machine needs to have an entry starting with @cert-authority, followed by the target’s hostname or wildcard pattern, and the public part of CA key used for signing. This is a security and convenience improvement when provisioned correctly, because now the user does not need to trust a remote’s host key (especially on the first connection). If the certificate present matches CA public key stored in known hosts file, ssh trusts it automatically.

DigitalOcean’s tutorial covers setting up both of these variants. Don’t worry about the deprecation notice, it’s good for any Linux version. Smallstep has an in-depth article on user certificates, and offers a convenient CA tool, bridging SSH with your Single Sign-On solution.

Group exchange moduli

RFC4253, the standard for SSH transport protocol, defines establishing the shared symmetric key with Diffie-Hellman Group Key Exchange, but with a fixed modulus (generating number). D-H security is based on difficulty of finding discrete logarithms, i.e. inverses of exponentiation in finite groups (exponentiation modulo large primes). This has proven to be safe enough, but there is still an attack possibility by a sufficiently advanced adversary.

To mitigate that possibility, we can generate our own set of moduli, which must be then distributed between connecting clients and servers. This is a fast, but memory-intensive process. First we generate candidate moduli with ssh-keygen -G moduli_file, optionally adding -b length to specify their bit length, and possibly -M memory (in megabytes) to limit the amount of RAM used.

Then the numbers need to be tested for correctness: that they are actually prime (more specifically a safe prime). This is a CPU-intensive process, started by invoking keygen with -T output_file -f moduli_file. Add -a number to increase number of primality testing rounds, which is 100 by default. Options -j start and -J numlines may be used to process only a part of the candidates file by specifying, respectively, the first line and number of lines processed. Adding -K checkpoint_file allows restarting this process, by recording progress into the specified file, and continuing from an appropriate point.

SSHFP

Host keys, or rather their fingerprints, can be also stored in DNS records (RFC4255). This is another mechanism, besides host certificates, to reduce problems with trusting yet-unknown hosts, especially on first connection. If ssh is invoked with the VerifyHostKeyDNS option set to yes, it will immediately trust hosts which have SSHFP DNS records matching the host key they present. If set to ask, the user is prompted first, but notified about the matching fingerprints anyway. On mismatch or absence, user confirmation is always needed.

Keygen has a mode to assist with setting this up: run it as root with -r hostname. It will output a set of DNS records, two per each host key (different signature algorithms) that you need to add into your DNS zone. Don’t run it as non-root as it will just output signatures for your own keys, not host keys. However, to actually make this work requires more steps, such as properly setting up DNSSEC for your zone.

Key Revocation Lists

The final feature from this exhaustive list of Things SSH-Keygen Does is Key Revocation List Management. KRLs are most useful when doing certificate-based authentication, but can be also used to revoke regular public keys. Remote ssh servers must be configured to use such a list, with RevokedKeys option, and it should be shared among all hosts in a company or domain. Public key files in users’ directories can be forgotten, but a centralised list takes priority over these, allowing “global” revocation with one stroke.

Creating a KRL is done with -k -f revocation_list_file item1 item2.... Items are either public key files, certificate files, or KRL specification files. KRL specs list revoked items one per line, and those can be either public key content, fingerprints, certificate serial numbers (assigned with -z) or key IDs (with -I). If we add -u, the revocation list is updated and not recreated from scratch. Adding -z version` allows versioning of the revocation file. With -Q -f revocation_list_file public_key we can query if a given key is contained in the revocation list.

This is the basis of any CA system for public key authentication. One such open-source system is cashier. Another one, with a narrower purpose, is Netflix’s BLESS.

Final remarks

Now that we know (almost) everything about ssh-keygen, we can use it to its full extent. Go wild!

SSH ships with many tools (or as Python developers say: batteries included), and we’ll look at more of them in upcoming posts.

(not very serious) Bonus

In this article we used keygen and ssh-keygen interchangeably. However, the short form was, for a time, reserved for small programs that aided users of illegal software and games by producing illicit registration keys. These were accepted by said software, unlocking full versions and replacing proper keys as sold by an authorized vendor. Keygens were typically Windows apps, popping up a small window with a cracker group’s logo, and blaring electronic music through the speakers while running.

If you miss these times, I’ve got a solution for you! Install figlet and vlc-bin, then save this as keygen.sh somewhere on your $PATH:

#! /bin/bash
cvlc -q -A alsa "http://c64.com:8000" &
figlet "ssh-keygen $(ssh -V 2>&1 | cut -d' ' -f1)"
ssh-keygen $*
echo "Press Return to exit..."
read
kill %1

This will do the blaring, sourcing music from c64.com’s live audio stream that plays mp3-rendered music from games for that platform. The logo is provided by figlet, and shows “ssh-keygen 8.1p5” or whatever ssh version you have installed. All options will be passed to the actual keygen command. When it finishes, you still need to hit Enter to exit and stop the music.