Secure a CI/CD Chain on Exoscale with Vault

The safety and proper operation of a CI/CD pipeline is one of the foundations enabling a company to test and deploy their applications in good conditions.

One of the conundrums of setting up such pipelines is authentication. In order to manage infrastructure from inside a CI/CD pipeline, two main approaches are possible:

1. using long-lived API credentials stored somehow in the CI/CD control plane
2. generating short-lived API credentials when needed as part of the job execution

Although the former method is the more common as it is simpler to set up, it can be a security liability in case of a leak or security breach: those credentials, often granted with powerful management permissions, can end up in the hands of malicious actors and open them the gates of your infrastructure.

Following the official launch of our IAM service and the release of the Vault backend plugin, we will demonstrate how you can leverage GitLab CI and the Exoscale platform features to safely implement a CI/CD pipeline: in our scenario, we will show you how to use short-lived API credentials restricted to strictly necessary API permissions to reduce the security risks of API credentials leak.

Vault Setup

After having installed and loaded the Exoscale secrets backend plugin into the Vault server according to the documentation, we must configure the backend with the credentials required to interact with the Exoscale IAM API:

%> vault write exoscale/config/root root_api_key=${EXOSCALE_API_KEY} root_api_secret=${EXOSCALE_API_SECRET}

We then proceed to define an Exoscale backend role, gitlab-ci-runner, that will apply to the GitLab CI runners when they request apikey secrets: this role will restrict generated API keys to the specific set of API operations that they need in order to successfully perform their job-related tasks, and nothing more.

Besides, we set a secrets lease duration of 15 minutes (renewable up to 30 minutes), which is enough for the standard duration of our CI jobs.

%> vault write exoscale/role/gitlab-ci-runner operations="compute/createSSHKeyPair,compute/registerCustomTemplate,compute/deployVirtualMachine,compute/destroyVirtualMachine,listInstancePools,listSecurityGroups,compute/listServiceOfferings,compute/listTemplates,compute/listVirtualMachines,compute/listVolumes,compute/listZones,compute/updateInstancePool,compute/getInstancePool,compute/queryAsyncJobResult,compute/scaleInstancePool,sos/putObject,sos/putObjectAcl,sos/getObject,sos/getObjectAcl,sos/deleteObject" ttl=15m max_ttl=30m

Next, we create a Vault policy that only allows the associated identities to create Exoscale backend apikey secrets for the role gitlab-ci-runner:

%> echo 'path "exoscale/apikey/gitlab-ci-runner" { capabilities = ["read"] }' | vault policy write exoscale-create-apikey -

Finally, we create the AppRole to be used by our GitLab CI runners, associated with the policy created during the previous step.

This authentication method is particularly suited for automated workflows in which no human operators are involved: in our case, GitLab CI runners authenticate to the Vault server at the beginning of a job, in order to retrieve short-lived restricted Exoscale API credentials required to manipulate infrastructure resources, for example to upload and register a custom template.

# Enable the AppRole authentication method
%> vault auth enable approle

# Create the AppRole
%> vault write auth/approle/role/gitlab-ci-runner \
    policies="exoscale-create-apikey" \
    token_ttl=15m \
    token_max_ttl=30m

To use the AppRole, we have to retrieve its role ID and secret ID that will serve as Vault credentials for the GitLab CI runners:

%> vault read auth/approle/role/gitlab-ci-runner/role-id
Key        Value
---        -----
role_id    30677f06-4ce6-828f-76a4-d3fb066b5842

%> vault write -f auth/approle/role/gitlab-ci-runner/secret-id
Key                   Value
---                   -----
secret_id             a68af9fd-3de1-1cd0-8d9f-5ef47fc70fbf

At this point, our Vault server is ready to issue Exoscale API credentials to our GitLab CI runners once they’re authenticated. To ensure our setup works as expected, we try to retrieve an apikey secret using the gitlab-ci-runner AppRole role ID/secret ID:

# Authenticate to the Vault server
%> export VAULT_TOKEN=$(vault write -field=token auth/approle/login role_id=30677f06-4ce6-828f-76a4-d3fb066b5842 secret_id=a68af9fd-3de1-1cd0-8d9f-5ef47fc70fbf)

# Request an apikey secret
%> vault read exoscale/apikey/gitlab-ci-runner
Key                Value
---                -----
lease_id           exoscale/apikey/gitlab-ci-runner/U6z5ofd9fCf2iYaZiS1wBhcp
lease_duration     15m
lease_renewable    true
api_key            EXOdfc1ef75850d0536a9bd9d1d
api_secret         QxXMwROIsWYjFasBcJ2lWgjdqYfo1fDDzSAWyN8YhFA
name               vault-gitlab-ci-runner-approle-1585661290792561655

Looking at the Exoscale IAM, we can confirm that the API key created by the Vault server as a result of our secrets request is conform to our configuration:

%> exo iam apikey show vault-gitlab-ci-runner-approle-1585661290792561655
┼────────────┼────────────────────────────────────────────────────┼
│ Name       │ vault-gitlab-ci-runner-approle-1585661290792561655 │
│ Key        │ EXOdfc1ef75850d0536a9bd9d1d                        │
│ Operations │ compute/createSSHKeyPair                           │
│            │ compute/deployVirtualMachine                       │
│            │ compute/destroyVirtualMachine                      │
│            │ compute/getInstancePool                            │
│            │ compute/listInstancePools                          │
│            │ compute/listSecurityGroups                         │
│            │ compute/listServiceOfferings                       │
│            │ compute/listTemplates                              │
│            │ compute/listVirtualMachines                        │
│            │ compute/listVolumes                                │
│            │ compute/listZones                                  │
│            │ compute/queryAsyncJobResult                        │
│            │ compute/registerCustomTemplate                     │
│            │ compute/scaleInstancePool                          │
│            │ compute/updateInstancePool                         │
│            │ sos/deleteObject                                   │
│            │ sos/getObject                                      │
│            │ sos/getObjectAcl                                   │
│            │ sos/putObject                                      │
│            │ sos/putObjectAcl                                   │
│ Resources  │ n/a                                                │
│ Type       │ restricted                                         │
┼────────────┼────────────────────────────────────────────────────┼

However, the perceptive reader might have noticed a design flaw: As put by HashiCorp in this blog post presenting the AppRole authentication method, an AppRole’s role ID and secret ID can be viewed as a username and a password for logging into a Vault server, and if we stop there we’d have effectively traded one form of long-lasting credentials for another.

AppRole credentials are actually meant to be retrieved (and stored) separately until the very moment they are supposed to be used by the intended application. We need to account for this in our setup to avoid exposing ourselves to the original security risk in case the gitlab-ci-runner AppRole credentials were to be leaked.

To address the issue, we will store the AppRole’s role ID inside the GitLab CI runners’ configuration, and instead of storing the AppRole secret ID value as-is along it, we will replace it with a Vault token granting its bearer access to the AppRole endpoint to request a one-time short-lived secret ID.

This way, if the role ID leaks it is useless without a matching secret ID, and if the Vault token allowing the creation of the gitlab-ci-runner AppRole secret IDs leaks those would be useless without the matching role ID.

First, let’s update the gitlab-ci-runner AppRole properties to enforce the secret ID duration:

%> vault write auth/approle/role/gitlab-ci-runner \
    policies="exoscale-create-apikey" \
    token_ttl=15m \
    token_max_ttl=30m \
    secret_id_num_uses=1 \
    secret_id_ttl=1m

Then we create a Vault policy granting the bearer of a token associated with it to only generate gitlab-ci-runner AppRole secret IDs:

%> echo 'path "auth/approle/role/gitlab-ci-runner/secret-id" { capabilities = ["update"] }' | vault policy write gitlab-ci-runner-secrets -

Finally, we issue a token to be stored in GitLab:

%> vault token create -policy=gitlab-ci-runner-secrets
Key                  Value
---                  -----
token                s.0LbBZSEU9OKUstEisqnSwclz
token_accessor       HePhJ3qOq0p7IfYdS3endPcy
token_duration       768h
token_renewable      true
token_policies       ["default" "gitlab-ci-runner-secrets"]
identity_policies    []
policies             ["default" "gitlab-ci-runner-secrets"]

When a GitLab CI runner receives a job, it’ll find this token in the metadata and use it to retrieve a one-time secret ID: it is the only moment both role ID and secret ID are both known, enabling the runner to authenticate with the role that generates restricted Exoscale credentials.

Note that the gitlab-ci-runner-secrets token obtained, as for any Vault token, has a limited duration and has to be periodically rotated; fortunately this can be done easily via GitLab’s API.

In the event of a token leak, it is possible to revoke it easily and replace it with a new one.

In the second part of the article, we’ll implement the CI/CD pipeline taking advantage of the Vault setup we’ve built, demonstrating how to setup GitLab CI to automatically and securely deploy an application on Exoscale.