Managing Secrets in GitLab / Git

Let's say that you have to log in via ssh into an instance, and you work with GitLab, so you want to keep the private key in GitLab somewhere. Is it secure? Let's see!

Custom environment variables

You can use custom environment variables. Here you can read more about them (Developers cannot change them, only Maintainers and Owners can). There are two types of variables:

• Variable (the runner creates an environment variable that uses the key for the name and the value for the value)
  
• File (the runner creates an environment variable that uses the key for the name. For the value, the runner writes the variable value to a temporary file and uses this path)
  

It seems that we can use File type for our purpose. We can set up it via API or UI. So, let's do that! Go to project's Settings > CI/CD. There will be Variables section (btw, you can specify variables also per group and even for all projects (in admin panel)). Click Add Variable button and add a variable:

• Key:  PRIV_KEY
• Value: content of our private key
• Type: File
  

There are also three other choices: Environment scope, Protect variable and Mask variable. With Environment scope you can decide that this variable will be available only on a specific environment (by default each environment has access to this variable). With Protect variable you can export the variable to pipelines running only on protected branches or tags. With Mask variable our variable will be masked in job logs (the variable is hidden in job logs). But, not all variables can be masked, and sadly our private key cannot be. Here is more info about it. And, even then... masking your variables is not so secure as you think.

Here is an example. I have $TEST_SECRET variable, and the content is a secret for you! If you try to "get into" this variable, GitLab will prevent you for this, and it is great:

 

As you can see, you cannot use echo or cat command and get the content of this variable. I even tried to encode the content into base64, save this to a file, and then try to read the file; still without a success 🤔. But, we are able to echo the variable and pipe into base64 program. And if you copy the output (c3VwZXItc2VjcmV0LWl0LXNob3VsZC1iZS1pbnZpc2libGUK) to your local machine and type echo "c3VwZXItc2VjcmV0LWl0LXNob3VsZC1iZS1pbnZpc2libGUK" | base64 -d, you get: super-secret-it-should-be-invisible. It was supposed to be super secret!

I do not know how it works under the hood, but it seems that GitLab try to mask the content everywhere in the log file, so if you just echo the variable, you will not see the content, but when you also encode it into base64 for example, GitLab does not know anything about this new string (I mean, it is different than the masked content, right?), so it cannot be masked.

Anyway. If someone has access to your project and can edit jobs (like me, above), they can do what they want, so giving appropriate permissions is a must. Spawning such variables only on protected branches/tags is also a good idea. This way, you can perform such jobs only on master and staging branches for example and do code review before, and block the suspicious code 👮. You can also use rules keyword and give access to run some jobs to only a small subset of people.

Here is an example, how you can use custom environment variable to keep a private key:

It is a quite good solution. But, it has a flaw. Your variables are not versioned. If there are more than two maintainers/owners, it might be a problem. Someone can change a variable and you will not be notified about it. Maybe, we can keep our secrets in git repository? Then, we will have versioning by default. Such approach would be useful for storing passwords for example.

SOPS: Secrets OPerationS

We can use sops for this:

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP.

Using sops, we can keep our "secrets" secret 🤫. We are going to use AWS KMS for encrypting and decrypting, but please, keep in mind that it is only for demo purpose. If you want to use it to keep some secret data, it makes sense to create more than one master key. According to sops documentation:

Multiple master keys allow for sharing encrypted files without sharing master keys, and provide a disaster recovery solution. The recommended way to use sops is to have two KMS master keys in different regions and one PGP public key with the private key stored offline. If, by any chance, both KMS master keys are lost, you can always recover the encrypted data using the PGP private key.

By default, sops encrypts the data key with each of the master keys. So, the file can be decrypted by each master key. You need only one master key for decrypting.

We need to create a symmetric key in AWS KMS. Then we have to set up sops by creating a file called .sops.yaml and putting the info about our ARN(s) (Amazon Resource Name(s)). Here is an example:

After that we have to create a file where we want to keep our secrets. Let's say that we want to keep there our private key and some passwords. And, we are going to keep all of this in YAML file. So, type sops secrets.yaml, you will see an example in your default editor, delete it and put there this (no, it is not my private key, do not worry):

When you close the editor, sops will save the file with encoded values:

Such file without master key is useless for others, so we can keep this guy in git repository without worry. We can also renew the data key on regular basis. You can also use key groups. This way you can tell sops that some data must be decrypted by providing two (or more; there also might be some thresholds like, at least 3 of 5 keys for instance) keys (AWS KMS and PGP for example).

If someone wants to get the content of our file without master key, they will get error like:

Now, we want to get secrets from our file in a job. If we use AWS KMS, we need to provide somehow credentials for it (AWS Access Key ID, AWS Secret Access Key). In most cases we can keep them as custom environment variables (like in the first example). But, if you think that someone evil can do very bad things, you can provide these environment variables during spawning pipeline or job:

You should also remember about enabling Protect variable option. It is very important. Without this, everyone who can create a new branch, will get an access to these environment variables; then they can modify a job, and echo some passwords for example. We have seen that we can mask our variable, but you can still unmask it via base64 for instance (if someone does such things in your team, I think you have other problems than that 😅).

How to get values from this file then? You can just type sops -d secrets.yaml, but this will give you all the data. What if we want to get only a password for redis? You can do that easily by sops -d --extract '["dev"]["redis"]' secrets.yaml:

So, how would look like our gitlab job if we wanted to use sops for getting a private key and logging into our instance? It might be something like that:

Can I use it with Helm?

Yes! There is a plugin for Helm called helm-secrets. Instead of deploying your service by helm upgrade ..., you must use helm secrets upgrade -f secrets.yaml .... Let's look for an example!

Let's say that we have two environments: dev and prod. On dev environment we are fine with storing passwords in repository, but on prod we do not want to do that. We want to deploy grafana both on dev and prod environments. Here is our root catalogue:

And here is a code:

How it works then? We have two jobs: dev and prod. We install sops and helm-secrets plugin, add bitnami repository to helm and check if there is a secrets.yaml file. If it exists, then we want to use it, so we must change the command from helm upgrade to helm secrets upgrade and use both values.yaml and secrets.yaml files. Here is a content of these files:

Here is a link to the repository, so you can see the whole picture how it might work. I hope it was useful! Of course, there are many other tools for storing secret data (Vault by HashiCorp for instance), but I think that sops and custom environment variables are enough in most cases.

Here is a video if you don't want to read: