In the last post we decoded the internals of an encrypted private key, basing our work on previous efforts in reading unencrypted ones. We ended up with a still unsolved question on how to conveniently use these keys, which require a password to be operational. In this post, we’ll dig further into the SSH ecosystem to find answers.
Finding a secure place
Since the filesystem didn’t meet our security needs, we need a better protected place to store those unencrypted keys. In Unix systems, that next level of security is process isolation. Basically, one process may not read or write into another’s memory, and they are only allowed to interact via well-defined channels. These are collectively called IPC, for inter-process communication, and include things such as message queues, pipes or sockets, explicitly shared memory or shared file mappings.
So if we had a process that kept unencrypted keys in its memory, handed them out over one of these, would this be a solution to our problem? Well, no. Without a way to verify that the other process can be trusted with them, this is as bad as just storing unencrypted data on disk. It should not expose keys directly, but instead be able to fulfill signing and encryption requests, accepting data on input and producing signed or encrypted data on output. But it should never just hand out the keys directly.
This is exactly what
ssh-agent does. It listens on a local socket, accepting requests from any process that has permissions to connect to it, and handles a couple of commands, none of which is give me the private key modulus and exponent.
The Agent, or Agents
On Linux systems,
ssh-agent is usually the one from the OpenSSH package, and it’s started by default with your session. However, many distros include gnome-keyring, which also has an agent component. KDE-based distros ship with KDE Wallet, which can be used for passphrases, but is not an agent replacement itself, like gnome-keyring. Windows systems, starting with Windows 10 release 1803, can install OpenSSH as an optional component, including the agent. Additional configuration may be required to launch it on session start. WSL also comes with ssh and agent out of the box. Other implementations also ship with an agent:
git-bash, PuTTY’s pageant, cygwin. On macOS, Keychain implements the agent protocol, and there are additional configuration directives and options integrating it into the OpenSSH ecosystem.
Some password managers can also be used as agents, with a plugin: KeeAgent for KeePass 2.x. Storing keys in it may be convenient, but it’s not recommended: it makes device loss mitigation harder. With the standard pubkey model, you’d just revoke or delete keys from devices you lost or no longer trust. But when they are stored in a password manager, it’s only the master password stopping the attacker from gaining access, and you can’t just remove it without losing access (or needing to secure alternative access, such as new keys) yourself. People have also devised scripts to use their password managers just for passphrases, which is slightly better: Knut Ahlers has one such script for LastPass, and Justin Dray has a more opinionated one which generates random passphrases and stores them into LastPass.
Check if an agent is running:
This environment variable tells compliant programs where to find the agent. On my system, that particular socket is handled by gnome-keyring-daemon, and another one is handled by ssh-agent itself. Alternatively, using
ssh_config always overrides that value. Regardless, we can launch a new agent in the current shell. The backticks are important: ssh-agent outputs commands that need to be parsed by current shell.
To stop it, either send a signal to that PID, or use
ssh-agent -k. The primary tool that communicates with the agent is
ssh-add. Unlike its name says, it can both add and remove identities (private keys) from the agent. When started, the agent has no identities. List them with
-l, and add by passing in the private key filename. If that key is encrypted, you will be prompted for a password. Now, please note that all commands below apply to OpenSSH’s agent. Other agents may not operate in the exact same way: for example, gnome-keyring refuses to delete “standard” keys (
id_rsa etc), but when using its GUI (Seahorse, a.k.a. Gnome’s Passwords and Keys), it wants to delete the key files permanently, instead of just removing identities from the session.
Forcing graphical prompt works because
ssh, when necessary) prefer stdin for passphrase input, if it’s available and a terminal. By redirecting
/dev/null, we disable stdin, which forces it to find another way. Which is to examine
DISPLAY environment variables, and try launching whatever executable or script the former one points to. There are also some defaults if it’s unset. Whatever that program outputs is used as a passphrase (which is how the LastPass integrations listed earlier work).
-L outputs public keys for all loaded identities. This output is suitable for putting in an
authorized_keys file somewhere.
An important option when adding is
-c, which will pop a confirmation window every time the key is used. This can be useful when diagnosing authentication problems, especially key loading order. You may need to have
SSH_ASKPASS set correctly, and a requisite package installed (all of them have
askpass in the name somewhere).
Remove a specific identity with
-d identity_file, or clear out the agent with
-D. Add and remove keys from a PKCS#11 device (like a YubiKey) with
-s pkcs_library.so and
-e pkcs_library.so respectively. You’ll likely need OpenSC and
pcscd running for that to work.
Once added, and with
IdentityAgent) present, ssh will prefer using keys from the agent. This means that it will offer them for authentication first, followed by keys from disk as matched by rules in
ssh_config, then default keys in standard locations. The same applies for
sftp, two other programs from the OpenSSH bundle, both used for file transfers.
The agent can also be put in a locked state, during which it will claim to have no identities loaded, and refuse signing operations. Be careful however, this will only prevent ssh from using keys that are loaded in the agent. Keys stored in files, matched with rules in
ssh_config are still available. Lock with
-x (lowercase X), which will prompt for a password twice; unlock with
-X (uppercase). Forcing a graphical prompt by redirecting also works here, as it uses the same input mechanism. It’s a convenient way to temporarily disable it: killing the agent works just as well, but you need to enter the passphrases again after restarting it. Locking and unlocking keeps the unencrypted keys, so there are no extra password prompts.
One final important option is
-t timeout, where timeout is either number of seconds, or a duration specification, same as used for various
sshd timeout options. It is a sequence of numbers and units, concatenated with no spaces, and units are
s for seconds,
m for minutes, then
w for hours, days, weeks; lowercase or uppercase. So
1m30s is same as
90s or just
90. This will only allow ssh-agent to keep the key for so long, and it will completely forget it once that time expires (requiring adding it again with
No process is safe from reading its memory when running with elevated priviledges. This includes ssh-agent. NetSPI has a gdb-based script to dump the agent’s memory (run this as root), and another one to extract keys from that. Additionally, while the agent will never hand out keys (other than public ones) but only sign/encrypt as requested, these operations can still be performed by any process, at any time they are available. Usually this is only ssh (and scp, sftp) or tools using it as transport (rsync, git). Unless keys are added with the explicit-confirmation flag (
-c), you will never know if they were used and when. A malicious program running under your user account could scan your shell history and ssh config, then silently try connecting to hosts found there. On success, it could send these vulnerable hosts along with your key files to an attacker. Or connect to these hosts, try gaining root priviledges, and then install a backdoor. Both have been done - see Careto and Windigo. Securing your keys with passphrases mitigates that first risk, since the keyfiles are unusable without a passphrase.
Therefore, keep your low-risk keys in the agent, and the high-risk keys either somewhere unreachable (on PIV keys, external media plugged only as needed), with low timeouts, or at least load them with the explicit flag to detect usage.
There are many scenarios where one would need to connect from machine A to host C via host B: network isolation, VPN, security policies, auditing logs. But if we want to keep using public key authentication, it is host B that needs to hold valid keys for connecting to host C (and not our machine A). This shifts the burden of trust onto machine B, which may or may not be what you want. A solution to that problem is Agent Forwarding.
-A (or set
ForwardAgent yes in
ssh_config) to set
SSH_AUTH_SOCK on the remote shell to a newly created socket, which uses your original agent forwarded over a secure connection. This lets you authenticate from host B using the agent running on machine A, so there is no need to setup public keys on host B. To explicitly stop forwarding (even if configured), use
-a when connecting. Keys signed with certificates may also explicitly forbid or permit port forwarding, regardless of options used.
However, this still has some issues similar to before: while the remote host cannot dump memory from your agent, it can still use it to sign things and connect behind your back (unless explicit confirmation is enabled). Always consider that when using forwarding.
In this post we learned about encrypted private keys, how to use
ssh-agent to handle them conveniently, and the associated risks and caveats. Together with the previous posts on key internals and key generation, this concludes a series on SSH keys. Watch this space for more posts on SSH usage in the future!
Sources and extra reading
- SSH Agent Hijacking
- Tradeoffs of ssh-agent holding all your keys
- Identities offered to remote server
- Problems with screen
- Stealing ssh keys from memory
- ssh-agent protocol client in net/ssh
Let’s go back to this part from the previous post:
Turns out, this
checksum field is actually important and not useless at all! This is a shortcut that ssh uses to determine if the passphrase used is valid. The correct definition is rather something like:
AES will happily decrypt data even with the wrong key + IV, and output garbage. Therefore, these two values are seeded with an identical, random value. If, after decrypting, they are still equal - we know the decryption key (passphrase) was correct, and that all other following data is also correct. Otherwise, we throw a bad passphrase error. See the implementation in openssh-portable: seeding and validation.