Previously we learned about the bazillion things that ssh-keygen can do; and before that we dissected SSH keys themselves. Now it’s time to consider: how do we store them securely?

Private keys are sensitive

While public keys are designed to be shared, private keys are very different. Their confidentiality is key to public-key authentication security: authenticating with a ssh server without having a private key relies on either cracking the Discrete Logarithm Problem, or, well, stealing the key. Therefore we need to think how to prevent this theft.

Filesystem security (Unix/POSIX model)

The first, most obvious choice is to rely on filesystem features: the concepts of file ownership and permission bits. This has always been enforced by ssh: it will ignore key files whose permissions are too open. When explicitly specified with -i keyfile, when loading keys specified with IdentityFile directives, and when trying default key file names like id_rsa. With a small catch: only if they are actually needed.

This is because of how key negotiation works: your client first asks the server I have this public key, can I use it for authentication?. The server may then reject it, or respond with yes, now prove that you have the corresponding private key. Only then is the private key actually loaded, either from a file or the agent.

All that, however, cannot protect you from malicious software running on your machine, which can just read and send your key files to a remote attacker. Nor does it protect from an elevated privilege attack, i.e. gaining root and then being able to bypass file permissions. Therefore, the next step is to make the key files unusable by third parties. Enter: encryption.

Key security

Storing your keys in an encrypted format makes them useless to the attacker: unless decrypted with the proper passphrase, they are just random garbage. But that applies to you too - to use them, you are required to provide that passphrase. This way you are trading security for convenience. If the passwords are to be secure, they need to be long. Retyping a long passphrase multiple times gets annoying quickly. Even moreso when you start to make mistakes while annoyed, thus having to retype again.

To solve that problem, we need to find a way to securely store either that passphrase, or a decrypted key. We’ll get back to that later, but first let’s take a look at an encrypted key file, as previously we only dealt with unencrypted ones.

Encrypted key internals

# strip padding | decode | hexdump | show first 5 lines
$ grep -v -- '---' encrypted_key | base64 -d | hexdump -C | head -5
00000000  6f 70 65 6e 73 73 68 2d  6b 65 79 2d 76 31 00 00  |openssh-key-v1..|
00000010  00 00 0a 61 65 73 32 35  36 2d 63 74 72 00 00 00  |...aes256-ctr...|
00000020  06 62 63 72 79 70 74 00  00 00 18 00 00 00 10 bd  |.bcrypt.........|
00000030  15 eb 45 fd 70 67 c6 7a  6e 73 b7 1f 42 c1 62 00  |..E.pg.zns..B.b.|
00000040  00 00 10 00 00 00 01 00  00 01 97 00 00 00 07 73  |...............s|

We can spot the difference already: in an unencrypted key, both the cipher and kdf fields were set to none, here we have aes256-ctr and bcrypt respectively. Keygen has no options to change that upon generation (it did have -Z ciphername up to somewhere around version 6.9, released in 2015). The default cipher occassionally changes with new OpenSSH versions, as bugs and vulnerabilities are fixed. Older releases used 3DES up to about 2010, then briefly aes128-cbc, before moving to aes256-cbc and the current aes256-ctr. Upgrade your old key files by setting a new passphrase: ssh-keygen -p -f key_file -P old_passphrase -N new_passphrase, which will re-encrypt them.

Running our Ruby decoder program from a previous post fails: we’re unable to read past the public key section. When trying to read length for the private key_type field, we get garbage. However, the public key part is identical - which means it’s not encrypted. No surprises here, as there’s no benefit of it being encrypted. Let’s step up our game and update the program to handle encrypted keys as well.

Extending our key parser

We’ll need OpenSSL for decryption, and bcrypt-pbkdf for an implementation of the key derivation function that SSH uses. The encryption algorithm also needs some parameters, which we’ll store in a struct; this is a bit overkill as we’ll be supporting only aes256-ctr + bcrypt.

require 'openssl'
require 'bcrypt_pbkdf'

CIPHERS = {
  'aes256-ctr' => OpenStruct.new(block_size: 16, key_len: 32,
                                 iv_len: 16, name: 'AES-256-CTR')
}

Next, let’s split the single large struct we used into smaller pieces.

class SSHKeyHeader < BinData::Record
  endian :big

  stringz :format # "openssh-key-v1"
  string32 :cipher # "none" or "aes256-ctr" in recent OpenSSH
  string32 :kdf # "none" or "bcrypt"
end

class PrivateKey < BinData::Record
  endian :big

  uint64 :checksum # padding or checksum, not important
  string32 :key_type # "ssh-rsa"
  bn32 :n # modulus
  bn32 :e # public exponent
  bn32 :d # private exponent
  bn32 :coeff # CRT helper value: q^(-1) mod p
  bn32 :p # prime 1
  bn32 :q # prime 2
  string32 :comment # user@host
  # And some irrelevant padding
end

class PlainSSHKey < BinData::Record
  endian :big

  bn32 # ignore kdf_data
  uint32 :num_keys # hardcoded to 1
  uint32 :pubkeys_len # length of following pubkey block
  array :pubkeys, initial_length: :num_keys do
    struct :pubkey do
      string32 :key_type # "ssh-rsa"
      bn32 :e # public exponent
      bn32 :n # modulus
    end
  end

  uint32 :privkey_len
  private_key :privkey
end

By reading the header first, we can already support unencrypted keys:

header = SSHKeyHeader.read(STDIN)
case header.cipher.to_s
when 'none'
  key = PlainSSHKey.read(STDIN)
else
  puts "Cannot read encrypted keys yet"
  exit
end

Encrypted keys differ from unencrypted ones in two places: the KDF data block, and the private key block. The latter one is obviously encrypted, and the former one contains key data that we need to combine with the passphrase to decrypt it. Let’s create an alternative to PlainSSHKey that can read it.

class BcryptData < BinData::Record
  endian :big
  uint32 # bcdata_len, not useful
  string32 :salt
  uint32 :rounds
end

class EncryptedSSHKey < BinData::Record
  endian :big

  bcrypt_data :kdf_data
  uint32 :num_keys # hardcoded to 1
  uint32 :pubkeys_len # length of following pubkey block
  array :pubkeys, initial_length: :num_keys do
    struct :pubkey do
      string32 :key_type # "ssh-rsa"
      bn32 :e # public exponent
      bn32 :n # modulus
    end
  end
  uint32 :enc_len # length of following privkey block
  string :enc_privkey, read_length: :enc_len
end

So far the only differences are kdf_data and enc_privkey, which is not a struct, but a string of bytes. BinData doesn’t offer a way to extend structures, only to embed them, and this small duplication is a price we can accept.

# class EncryptedSSHKey
  def privkey
    # OpenSSL cipher names use a slightly different format
    # Also, AES is a symmetric cipher, we could just as well call
    # .encrypt here
    decoder = OpenSSL::Cipher.new(cipher_params.name).decrypt
    # Most ciphers, including AES-256-CTR need two parameters:
    # key and Initialization Vector
    decoder.key, decoder.iv = key_and_iv
    # Feed cryptext into decoder, returns decrypted data
    data = decoder.update(enc_privkey.to_s)
    PrivateKey.read(data)
  end

Key and IV are both stored in that KDF data block. But to recover them, we need to run bcrypt_pbkdf, supplied by a gem of the same name. It will mix the stored salt with a passphrase, and the result contains both key and IV.

# class EncryptedSSHKey
  def salt
    kdf_data.salt.to_s
  end

  def key_and_iv
    key_len = cipher_params.key_len
    key_with_iv = BCryptPbkdf.key(passphrase, salt,
                                  key_len + cipher_params.iv_len,
                                  kdf_data.rounds)
    # The two values are concatenated, but we know their lengths
    # Note the fat ellipsis
    shield_key = key_with_iv[0...key_len]
    iv = key_with_iv[key_len..]

    [shield_key, iv]
  end

As this program is designed to be used in a pipeline, the passphrase must be supplied by commandline.

# class EncryptedSSHKey
  def passphrase
    ARGV[0] or raise 'Must provide passphrase on commandline'
  end

The last missing piece is cipher_params, which obviously comes from the hash we defined at the beginning. To get them, our EncryptedSSHKey record must be parameterized, which BinData neatly provides:

class EncryptedSSHKey < BinData::Record
  mandatory_parameter :cipher

  def cipher_params
    CIPHERS[get_parameter(:cipher)]
  end
end

Now, we can redo the case block to support encrypted keys:

header = SSHKeyHeader.read(STDIN)
cipher = header.cipher.to_s
key = case cipher
when 'none'
  PlainSSHKey.read(STDIN)
else
  # Not passing kdf, as we only support bcrypt anyway
  EncryptedSSHKey.read(STDIN, cipher: cipher)
end

# running the program
$ grep -v -- '---' encrypted_key | base64 -d | ruby privkey.rb 123456

And that’s it! Because privkey is now a method on both structures, and not a record field, the validation block at the end needs a tiny update, which isn’t worth discussing here. Download the full program to see the omitted details.

Now that we understand how encrypted keys work, we need to consider another security/convenience tradeoff: storing either the unencrypted key or its passphrase securely. More on that in the next article!