In the last post we decoded the internals of an encrypted private key, basing our work on previous efforts in reading unencrypted ones. We ended up with a still unsolved question on how to conveniently use these keys, which require a password to be operational. In this post, we’ll dig further into the SSH ecosystem to find answers.

Finding a secure place

Since the filesystem didn’t meet our security needs, we need a better protected place to store those unencrypted keys. In Unix systems, that next level of security is process isolation. Basically, one process may not read or write into another’s memory, and they are only allowed to interact via well-defined channels. These are collectively called IPC, for inter-process communication, and include things such as message queues, pipes or sockets, explicitly shared memory or shared file mappings.

So if we had a process that kept unencrypted keys in its memory, handed them out over one of these, would this be a solution to our problem? Well, no. Without a way to verify that the other process can be trusted with them, this is as bad as just storing unencrypted data on disk. It should not expose keys directly, but instead be able to fulfill signing and encryption requests, accepting data on input and producing signed or encrypted data on output. But it should never just hand out the keys directly.

This is exactly what ssh-agent does. It listens on a local socket, accepting requests from any process that has permissions to connect to it, and handles a couple of commands, none of which is give me the private key modulus and exponent.

The Agent, or Agents

On Linux systems, ssh-agent is usually the one from the OpenSSH package, and it’s started by default with your session. However, many distros include gnome-keyring, which also has an agent component. KDE-based distros ship with KDE Wallet, which can be used for passphrases, but is not an agent replacement itself, like gnome-keyring. Windows systems, starting with Windows 10 release 1803, can install OpenSSH as an optional component, including the agent. Additional configuration may be required to launch it on session start. WSL also comes with ssh and agent out of the box. Other implementations also ship with an agent: git-bash, PuTTY’s pageant, cygwin. On macOS, Keychain implements the agent protocol, and there are additional configuration directives and options integrating it into the OpenSSH ecosystem.

Some password managers can also be used as agents, with a plugin: KeeAgent for KeePass 2.x. Storing keys in it may be convenient, but it’s not recommended: it makes device loss mitigation harder. With the standard pubkey model, you’d just revoke or delete keys from devices you lost or no longer trust. But when they are stored in a password manager, it’s only the master password stopping the attacker from gaining access, and you can’t just remove it without losing access (or needing to secure alternative access, such as new keys) yourself. People have also devised scripts to use their password managers just for passphrases, which is slightly better: Knut Ahlers has one such script for LastPass, and Justin Dray has a more opinionated one which generates random passphrases and stores them into LastPass.

Usage

Check if an agent is running:

$ echo $SSH_AUTH_SOCK
/run/user/1000/keyring/ssh
$ file $SSH_AUTH_SOCK
/run/user/1000/keyring/ssh: socket

This environment variable tells compliant programs where to find the agent. On my system, that particular socket is handled by gnome-keyring-daemon, and another one is handled by ssh-agent itself. Alternatively, using IdentityAgent in ssh_config always overrides that value. Regardless, we can launch a new agent in the current shell. The backticks are important: ssh-agent outputs commands that need to be parsed by current shell.

$ eval `ssh-agent -s`
Agent pid 130987
$ echo $SSH_AGENT_PID
130987
$ echo $SSH_AUTH_SOCK
/tmp/ssh-SXxm4chDLuan/agent.130986

To stop it, either send a signal to that PID, or use ssh-agent -k. The primary tool that communicates with the agent is ssh-add. Unlike its name says, it can both add and remove identities (private keys) from the agent. When started, the agent has no identities. List them with -l, and add by passing in the private key filename. If that key is encrypted, you will be prompted for a password. Now, please note that all commands below apply to OpenSSH’s agent. Other agents may not operate in the exact same way: for example, gnome-keyring refuses to delete “standard” keys (id_rsa etc), but when using its GUI (Seahorse, a.k.a. Gnome’s Passwords and Keys), it wants to delete the key files permanently, instead of just removing identities from the session.

$ ssh-add -l
The agent has no identities.
$ ssh-add example_key
Identity added: example_key (user@hostname).
$ ssh-add -l # key size, fingerprint algorithm:fingerprint comment (key-type)
3072 SHA256:fFZJB0LcAyH4gAr0KYSrCjrrbj+O1LCZ/F3+jOiaJu0 user@hostname (RSA)
$ ssh-add -l -E md5 # specify fingerprint algorithm: md5, sha1, sha256, sha512
3072 MD5:87:b8:d7:2b:be:63:12:b0:ec:52:e7:60:36:56:cc:45 user@hostname (RSA)
$ ssh-add encrypted_key
Enter passphrase for encrypted_key:
Identity added: encrypted_key (user@hostname)
# Force a graphical password prompt
$ ssh-add encrypted_key < /dev/null

Forcing graphical prompt works because ssh-add (and ssh, when necessary) prefer stdin for passphrase input, if it’s available and a terminal. By redirecting /dev/null, we disable stdin, which forces it to find another way. Which is to examine SSH_ASKPASS and DISPLAY environment variables, and try launching whatever executable or script the former one points to. There are also some defaults if it’s unset. Whatever that program outputs is used as a passphrase (which is how the LastPass integrations listed earlier work).

Running with -L outputs public keys for all loaded identities. This output is suitable for putting in an authorized_keys file somewhere.

$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrX8JYebb<truncated> user@host
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIPAtYe2j<truncated> user@host

An important option when adding is -c, which will pop a confirmation window every time the key is used. This can be useful when diagnosing authentication problems, especially key loading order. You may need to have SSH_ASKPASS set correctly, and a requisite package installed (all of them have askpass in the name somewhere).

Remove a specific identity with -d identity_file, or clear out the agent with -D. Add and remove keys from a PKCS#11 device (like a YubiKey) with -s pkcs_library.so and -e pkcs_library.so respectively. You’ll likely need OpenSC and pcscd running for that to work.

Once added, and with SSH_AUTH_SOCK (or IdentityAgent) present, ssh will prefer using keys from the agent. This means that it will offer them for authentication first, followed by keys from disk as matched by rules in ssh_config, then default keys in standard locations. The same applies for scp and sftp, two other programs from the OpenSSH bundle, both used for file transfers.

The agent can also be put in a locked state, during which it will claim to have no identities loaded, and refuse signing operations. Be careful however, this will only prevent ssh from using keys that are loaded in the agent. Keys stored in files, matched with rules in ssh_config are still available. Lock with -x (lowercase X), which will prompt for a password twice; unlock with -X (uppercase). Forcing a graphical prompt by redirecting also works here, as it uses the same input mechanism. It’s a convenient way to temporarily disable it: killing the agent works just as well, but you need to enter the passphrases again after restarting it. Locking and unlocking keeps the unencrypted keys, so there are no extra password prompts.

One final important option is -t timeout, where timeout is either number of seconds, or a duration specification, same as used for various sshd timeout options. It is a sequence of numbers and units, concatenated with no spaces, and units are s for seconds, m for minutes, then h, d, w for hours, days, weeks; lowercase or uppercase. So 1m30s is same as 90s or just 90. This will only allow ssh-agent to keep the key for so long, and it will completely forget it once that time expires (requiring adding it again with ssh-add).

Security

No process is safe from reading its memory when running with elevated priviledges. This includes ssh-agent. NetSPI has a gdb-based script to dump the agent’s memory (run this as root), and another one to extract keys from that. Additionally, while the agent will never hand out keys (other than public ones) but only sign/encrypt as requested, these operations can still be performed by any process, at any time they are available. Usually this is only ssh (and scp, sftp) or tools using it as transport (rsync, git). Unless keys are added with the explicit-confirmation flag (-c), you will never know if they were used and when. A malicious program running under your user account could scan your shell history and ssh config, then silently try connecting to hosts found there. On success, it could send these vulnerable hosts along with your key files to an attacker. Or connect to these hosts, try gaining root priviledges, and then install a backdoor. Both have been done - see Careto and Windigo. Securing your keys with passphrases mitigates that first risk, since the keyfiles are unusable without a passphrase.

Therefore, keep your low-risk keys in the agent, and the high-risk keys either somewhere unreachable (on PIV keys, external media plugged only as needed), with low timeouts, or at least load them with the explicit flag to detect usage.

Forwarding

There are many scenarios where one would need to connect from machine A to host C via host B: network isolation, VPN, security policies, auditing logs. But if we want to keep using public key authentication, it is host B that needs to hold valid keys for connecting to host C (and not our machine A). This shifts the burden of trust onto machine B, which may or may not be what you want. A solution to that problem is Agent Forwarding.

Connect with -A (or set ForwardAgent yes in ssh_config) to set SSH_AUTH_SOCK on the remote shell to a newly created socket, which uses your original agent forwarded over a secure connection. This lets you authenticate from host B using the agent running on machine A, so there is no need to setup public keys on host B. To explicitly stop forwarding (even if configured), use -a when connecting. Keys signed with certificates may also explicitly forbid or permit port forwarding, regardless of options used.

However, this still has some issues similar to before: while the remote host cannot dump memory from your agent, it can still use it to sign things and connect behind your back (unless explicit confirmation is enabled). Always consider that when using forwarding.

Conclusions

In this post we learned about encrypted private keys, how to use ssh-agent to handle them conveniently, and the associated risks and caveats. Together with the previous posts on key internals and key generation, this concludes a series on SSH keys. Watch this space for more posts on SSH usage in the future!

Sources and extra reading

Overdue footnote

Let’s go back to this part from the previous post:

class PrivateKey < BinData::Record
  endian :big

  uint64 :checksum # padding or checksum, not important

Turns out, this checksum field is actually important and not useless at all! This is a shortcut that ssh uses to determine if the passphrase used is valid. The correct definition is rather something like:

class PrivateKey < BinData::Record
  endian :big

  uint32 :check_a 
  uint32 :check_b

AES will happily decrypt data even with the wrong key + IV, and output garbage. Therefore, these two values are seeded with an identical, random value. If, after decrypting, they are still equal - we know the decryption key (passphrase) was correct, and that all other following data is also correct. Otherwise, we throw a bad passphrase error. See the implementation in openssh-portable: seeding and validation.