Secrets and private keys#
Where are secrets stored#
Most secrets are stored in one of two locations:
Secrets that are shared across all of our hub and cluster deployments, such as Auth0 secrets.
Secrets that are specific to each cluster / hub that we run - for example, cloud provider secrets to control the cluster programmatically - are stored in that cluster’s directory under
For example, see the 2i2c cluster directory.
All our secrets are encrypted with
For information about how to set up
sops, see the team compass documentation
Naming conventions for secret files#
All our secrets should contain the word
secret somewhere in their filename since this will ensure they will be git-ignored in their unencrypted format.
For unencrypted secret files, we follow the convention:
When we encrypt this file using
sops, our convention is to add the prefix
This will tell
git that the file is encrypted and therefore safe to be checked into version control.
You can change the name of the file during encryption like so:
sops --output enc-descriptive-name.secret.yaml --encrypt descriptive-name.secret.yaml
Similarly, we remove the
enc- prefix when decrypting a file so it can no longer be checked into version control, like so:
sops --output descriptive-name.secret.yaml --decrypt enc-descriptive-name.secret.yaml
How to rotate / change secrets#
Sometimes we need to rotate the secret keys used in our repository. For example, if a service we use has become compromised, and we need to generate new keys in order to protect the infrastructure.
To rotate our secrets, take these steps:
Determine which configuration file you’d like to update. See Where are secrets stored.
Unencrypt the configuration file. See the team compass documentation for instructions on unencrypting.
Generate a new key with
openssl rand -hex 32
This will return a random hash that looks something like this:
Find the key you’d like to replace, and replace its value with the hash that you’ve generated above.
If you wish to change the secret keys for the hub proxies, you would update the value of
secret_keyin the configuration file with proxy secrets.
Re-encrypt the file with
Commit the file to the repository and push.
You have now rotated the secret for this key!
Cleaning up decrypted files#
The naming conventions we outlined above allow us to clean the repository of unencrypted secrets using our
.gitignore config and the
git clean command.
To clean up unencrypted secrets (and other ignored files) you can use
git clean -Xfd which will delete untracked files (
-X), with required confirmation (
-f), recursively (