---
title: "Secure a CI/CD Chain on Exoscale with Vault - Part 2"
description: "How to take advantage of the Exoscale's IAM and Vault backend plugin to secure your CI/CD chain, part 2/2"
date: 2020-05-05
author: "Johnatan Illi"
url: https://www.exoscale.com/blog/secure-ci-cd-with-vault-2/
---
# Secure a CI/CD Chain on Exoscale with Vault - Part 2

In the [first part](https://www.exoscale.com/blog/secure-ci-cd-with-vault/) of this series we've laid the foundations of an infrastructure that can be used to secure a CI/CD pipeline. This infrastructure relies on Vault to issue short-lived Exoscale API credentials to the GitLab CI runners, so they are granted just the necessary permissions to perform their tasks within a limited time window.

Before moving on to the second part of the series, let's recap some important bits from our previous actions:

* We've defined a Vault [Exoscale secrets backend](https://github.com/exoscale/vault-plugin-secrets-exoscale) `gitlab-ci-runner` role issuing Exoscale API credentials valid up to 15 minutes restricted to the set of API operations required by the GitLab CI runners.
* Then, we've defined a Vault policy named `exoscale-create-apikey` granting access to the Exoscale secrets backend `gitlab-ci-runner` role, and associated this policy to a Vault `gitlab-ci-runner` [*AppRole*](https://www.vaultproject.io/docs/auth/approle): this is the AppRole that will be used by the GitLab CI runners in this part of the series.
* Finally, we've create another Vault policy (`gitlab-ci-runner-secrets`) which we associate to a Vault token allowing its bearer to retrieve a secret ID from the `gitlab-ci-runner` AppRole.

In this second part, we will proceed to setup the CI/CD pipeline of an example application to perform the following tasks securely:

1. Build and register a custom template containing the application
1. Test that the application meets our functional requirements (*smoke testing*)
1. Deploy the application

![Secure your CI/CD chain with Vault on Exoscale 2](https://www.exoscale.com/blog/secure-ci-cd-with-vault-2/ci-cd-vault-exoscale-2.jpg)

## Prerequisites

Our example CI/CD pipeline relies essentially on the following tools:

**Vault**

As we showcased in our previous article, Vault is the backbone of our secure CI/CD pipeline implementation, providing the workers restricted Exoscale API credentials for a limited time. You can get [an overview of it works and how it is integrated with Exoscale here](https://www.exoscale.com/blog/manage-your-exoscale-iam-with-vault/).

**Packer**

[Packer](https://www.packer.io/) allows us to create a virtual machine disk image of our application, that will then be registered to Exoscale as a Compute instance [custom templates][exoscale-blog-packer]. You can learn more on how to use Packer on Exoscale in this [previous blog article](https://www.exoscale.com/blog/creating-custom-templates-using-packer/).

**GitLab Runners**

To run our GitLab CI jobs on Exoscale, we have deployed [GitLab Runners](https://docs.gitlab.com/runner/) on an [Instance Pool](https://www.exoscale.com/blog/instance-pool-release/) as presented in the _[Build Highly Available Gitlab Runners Infrastructure Using Instance Pool](https://www.exoscale.com/blog/highly-available-gitlab-runners/)_ article.


## CI/CD Pipeline Implementation

### GitLab CI Setup

In order to focus on the relevant bits, this articles assumes the following:

- Our example application is deployed on an Exoscale [Instance Pool](https://community.exoscale.com/documentation/compute/instance-pools/) to ensure scalability.
- For the sake of simplicity and brevity, the application is a simple *Hello, World!* server similar to what has been showcased in our past article [*Creating Custom Templates Using Packer*](https://www.exoscale.com/blog/creating-custom-templates-using-packer/).
- The GitLab CI runners have all the tools required (`exo` CLI, Packer, Vault) installed.

As explained in the previous part, Vault *AppRole* is an authentication method composed of 2 pieces of information: a *role ID* (unique and static, similar to a *username*), and a *secret ID* (which can – and should – be dynamically generated multiple times for a given role ID, similar to a *password*).

These 2 parts should not be stored durably together, so that in case of a data leak or security breach the worst that happens is only half of the credentials are known and therefore cannot be used to gain access to granted permissions. 

As a result, we will store the `gitlab-ci-runner` AppRole role ID directly on the runners' Compute instance in the [GitLab CI runner configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section); in addition, the Vault token created at the end of the previous part – which is associated to the `gitlab-ci-runner-secrets` policy – into a GitLab CI [variable](https://docs.gitlab.com/ee/ci/variables/) that will be passed to the GitLab CI runners with the other job metadata.

![GitLab CI variables](https://www.exoscale.com/blog/secure-ci-cd-with-vault-2/gitlabci-variables.png)

The GitLab CI backplane configuration is minimal, as most of the pipeline configuration is done through a [`.gitlab-ci.yml` file](https://docs.gitlab.com/ee/ci/yaml/) versioned in the application Git repository.

Before proceeding to describe all the sequential steps of our CI/CD pipeline (corresponding to GitLab CI `stages`), we start by setting a few variables and configuration *helpers* that will be used throughout the pipeline:

```yaml
# This helper defines the sequence of commands that the GitLab CI runners have to execute in order to retrieve Exoscale API credentials
.vault_get_api_credentials: &_vault_get_api_credentials >
  export VAULT_SECRET_ID=$(vault write -field=secret_id -f auth/approle/role/gitlab-ci-runner/secret-id) &&
  test -n "${VAULT_SECRET_ID}" || (echo "ERROR: failed to retrieve Vault gitlab-ci-runner AppRole secret ID" ; exit 1) &&
  export VAULT_TOKEN=$(vault write -field=token auth/approle/login role_id=${VAULT_ROLE_ID} secret_id=${VAULT_SECRET_ID}) &&
  credentials=$(vault read -format=JSON exoscale/apikey/gitlab-ci-runner) &&
  export EXOSCALE_API_KEY=$(jq -r .data.api_key <<< "${credentials}") EXOSCALE_API_SECRET=$(jq -r .data.api_secret <<< "${credentials}")

# This helper is a convenience macro that will define a variable containing the name of the Compute instance custom template resulting of the build, which will be suffixed with the tag associated to the Git commit or with the first characters Git commit hash if no tags are specified
.get_template_name: &_get_template_name >
  template_name=hello-$(test -n "${CI_COMMIT_TAG}" && echo ${CI_COMMIT_TAG} || echo ${CI_COMMIT_SHA:0:7})

# Global variables used across all stages
variables:
  EXOSCALE_ZONE: de-muc-1
  APP_INSTANCE_POOL: hello-app

# We apply the following tags to all the jobs that will be defined in the pipeline. The "vault" tag ensures our CI jobs will be dispatched only to our custom GitLab CI runners that are equipped with Vault client as well as the VAULT_ROLE_ID part of the AppRole.
default:
  tags:
    - vault
```

Our CI/CD pipeline consists of the following stages:

```yaml
stages:
  - build
  - test
  - deploy
```


#### Build Stage

The `build` stage builds the *hello* application custom template using Packer. Note: as illustrated [in our previous article showcasing Packer usage on Exoscale][exoscale-blog-packer], we use the `exoscale-import` post-processor to register the template automatically once the disk image build is complete.

```yaml
build:
  stage: build
  before_script:
    - *_vault_get_api_credentials
    - *_get_template_name
  script:
    - 'cd packer/'
    - 'ssh-keygen -q -t rsa -C "packer" -N "" -f id_rsa'
    - 'printf "#cloud-config\nssh_authorized_keys:\n- \"%s\"\n" "$(< id_rsa.pub)" > user-data'
    - 'cloud-localds seed.img user-data'
    - 'PACKER_SSH_PRIVATE_KEY=$(readlink -f id_rsa) packer build -var template_name=${template_name} hello.json'
```


#### Test Stage

During the `test` stage, we boot a Compute instance based on the custom template we've just registered, and run a few external tests to ensure that no regressions happen between changes (also called *smoke testing*):

```yaml
test:
  stage: test
  before_script:
    - *_vault_get_api_credentials
    - *_get_template_name
    - instance=hello-${CI_COMMIT_SHA:0:7}-test
    - 'exo -Q vm create ${instance} --zone ${EXOSCALE_ZONE} --template hello-${CI_COMMIT_SHA:0:7} --template-filter mine --security-group hello'
    - 'for i in {1..10}; do curl -s -o /dev/null http://$(exo -O text --output-template "{{.IPAddress}}" vm show ${instance}):8080 && break || sleep 10s; done'
  script:
    - 'curl -s http://$(exo -O text --output-template "{{.IPAddress}}" vm show ${instance}):8080 > hello.out'
    - 'exo -Q vm delete -f ${instance}'
    - 'grep -q "Hello, World!" hello.out || (echo "ERROR: unexpected application output: $(< hello.out)" ; exit 1)'
```


#### Deploy Stage

At this point our application is ready to be deployed to production, which in our scenario is running on an Instance Pool.

To deploy the new version of our application, we update the Instance Pool to point to the new custom template, then temporarily scale it up to double its original size to provision new members using the up-to-date template, wait a little while for them to boot up, and finally downsize the pool to its original size to discard the older members running the previous version of the application:

```yaml
deploy:
  stage: deploy
  before_script:
    - *_vault_get_api_credentials
    - *_get_template_name
  script:
    - 'pool_size=$(exo -O text --output-template "{{.Size}}" instancepool show ${APP_INSTANCE_POOL} -z ${EXOSCALE_ZONE})'
    - 'exo instancepool update ${APP_INSTANCE_POOL} --zone ${EXOSCALE_ZONE} --size $((${pool_size} * 2)) --template ${template_name} --template-filter mine'
    - 'sleep 60s'
    - 'exo -Q instancepool update ${APP_INSTANCE_POOL} --zone ${EXOSCALE_ZONE} --size ${pool_size}'
  only:
    - master
```

Please note that, although it is realistic and totally functional, this scenario cut a few corners for the sake of simplicity and brevity: a more bullet-proof implementation would actually ensure that the new Instance Pool members are ready to serve before discarding the older ones.

Also, as a precaution it would be safer to introduce an intermediate stage where the pipeline would deploy to a staging environment, and require manual confirmation before actually deploying to the production pool – e.g. using GitLab CI's [`when:manual`](https://docs.gitlab.com/ee/ci/yaml/#whenmanual) predicate.

<video width="100%" height="auto" controls>
  <source src="ci-cd-vault-exoscale.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>


## Closing Words

We hope that this series of articles has successfully showcased the power of the Exoscale platform our tooling in a practical setting. Cloud security has never been more important than it is now, and our team is dedicated to offer state-of-the-art solutions to our customers.

[exoscale-blog-packer]: https://community.exoscale.com/documentation/compute/custom-templates/