---
title: "Storage encryption with Exoscale Flexible Storage template, LUKS and HashiCorp Vault"
date: 2021-07-22
author: "Cédric Dufour"
tags: ["storage","tutorial"]
url: https://www.exoscale.com/blog/flexible-storage-luks-vault/
---
# Storage encryption with Exoscale Flexible Storage template, LUKS and HashiCorp Vault

[Exoscale Flexible Storage template][exoscale-template-flexible-storage] empowers users to resize
and/or create disk partitions as they deem fit, thanks to the flexibility provided by the [Linux
Logical Volume Manager (LVM)][wikipedia-lvm].

[exoscale-template-flexible-storage]: https://www.exoscale.com/blog/flexible-storage-template/
[wikipedia-lvm]: https://en.wikipedia.org/wiki/Logical_Volume_Manager_%28Linux%29

In this article, we will leverage this flexibility to create a new partition, encrypted using
[Linux Unified Key Setup (LUKS)][wikipedia-luks], which passphrase shall be secured using [HashiCorp
Vault][hashicorp-vault].

[wikipedia-luks]: https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup
[hashicorp-vault]: https://www.vaultproject.io/

To achieve this objective, we will, in the following sections:

* configure HashiCorp Vault to:
  + provide Encryption-as-a-Service to secure the LUKS passphrase
  + perform seamless authentication of compute instances, using Exoscale Vault Authentication Plugin

* create and encrypt a new LVM Logical Volume (LV) / partition using LUKS

* setup the bits'-n-pieces (script, systemd unit) required to automatically - yet securely - enable
  the LUKS encrypted partition at boot time and disable it on shutdown

All of it using [Exoscale CLI][exoscale-doc-cli] to interact with Exoscale cloud platform.

[exoscale-doc-cli]: https://community.exoscale.com/tools/command-line-interface/


## Configure your Vault server

We will assume that you already have a running Vault server available (which setup is out of the
scope of this article).

### Vault Encryption-as-a-Service

In order to secure the LUKS passphrase, we will use [Vault Transit Secrets Engine][hashicorp-vault-secret-transit],
enabling so-called Encryption-as-a-Service (EaaS):

```
# Enable Vault Transit Secrets Engine
$ vault secrets enable transit
# [output]
Success! Enabled the transit secrets engine at: transit/
```

[hashicorp-vault-secret-transit]: https://www.vaultproject.io/docs/secrets/transit

Along a LUKS (passphrase) dedicated encryption/decryption endpoint/key and ad-hoc Policy:

```
# Create LUKS-dedicated encryption/decryption endpoint/key
$ vault write -f transit/keys/luks
# [output]
Success! Data written to: transit/keys/luks

# Create ad-hoc Policy
$ vault policy write 'transit-luks' - <<EOF
# Allow Policy holder to encrypt LUKS passphrase
path "transit/encrypt/luks"
{
    capabilities = ["update"]
}

# Allow Policy holder to decrypt LUKS passphrase
path "transit/decrypt/luks"
{
    capabilities = ["update"]
}
EOF
# [output]
Success! Uploaded policy: transit-luks
```

### Vault Exoscale Authentication Plugin

[Vault Exoscale Authentication Plugin][vault-plugin-auth-exoscale] allows to authenticate Exoscale
[Compute Instances](https://www.exoscale.com/compute/) based on properties of each instance, whether intrinsic (set by the Exoscale cloud platform; e.g. the instance UUID) or assigned by the users (such as instance tags).

[vault-plugin-auth-exoscale]: https://github.com/exoscale/vault-plugin-auth-exoscale

#### Dedicated Exoscale IAM credentials

In order to access Exoscale API and verify/match compute instances properties, the Vault Exoscale
Authentication Plugin must be configured with [Exoscale IAM credentials][exoscale-doc-iam], ideally
scoped specifically for the purpose at hand:

```
# Create Vault-specific Exoscale IAM credentials
$ exo iam apikey create vault-plugin-auth-exoscale \
  --operation 'compute/listZones,compute/listVirtualMachines'
# [output (partial)]
Name    vault-plugin-auth-exoscale
Key     EXO...
Secret  ...
```

Mark the output `Key` and `Secret` as we will need them in the next section.

[exoscale-doc-iam]: https://community.exoscale.com/product/iam/quick-start/

#### Setup the Vault Exoscale Authentication Plugin

As any other Vault plugin, the Vault Exoscale Authentication Plugin must be registered before it can
be used. Make sure to add the `plugin_directory = "/usr/lib/vault/server/plugins"` to your
Vault server's configuration. Then:

```
# Register the Exoscale Authentication Plugin
$ vault plugin register \
  -command="vault-plugin-auth-exoscale" \
  -sha256="$(sha256sum /usr/lib/vault/server/plugins/vault-plugin-auth-exoscale | cut -d " " -f 1)" \
  auth exoscale
# [output]
Success! Registered plugin: exoscale
```

You may then enable and configure it with the Exoscale IAM credentials we created above:

```
# Enable the Exoscale Authentication Plugin
$ vault auth enable exoscale
# [output]
Success! Enabled exoscale auth method at: exoscale/

# Configure the Exoscale Authentication Plugin
$ vault write auth/exoscale/config api_key=EXO... api_secret=...
$ vault read auth/exoscale/config
# [output]
Key             Value
---             -----
api_endpoint    https://api.exoscale.com/v1
api_key         EXO...
api_secret      ...
```

#### Configure a Vault Exoscale Authentication Role

In order to grant our storage instance the proper permissions, we shall:
- authenticate it using the `validator` parameter and a [CEL expression][google-cel] that matches
  any property you may deem fit; e.g. the instance _name_
- authorize it by granting it the `transit-luks` Policy (`token_policies`)

```
$ vault write auth/exoscale/role/storage \
  validator='instance_name.startsWith("storage")' \
  token_period='24h' \
  token_policies='transit-luks'
# [output]
Success! Data written to: auth/exoscale/role/storage
```

Mark the Token being a [Periodic Token][hashicorp-vault-token-periodic], which will come handy for
the automation part (see section further below).

[google-cel]: https://github.com/google/cel-spec/blob/master/doc/langdef.md
[hashicorp-vault-token-periodic]: https://www.vaultproject.io/docs/concepts/tokens#periodic-tokens


## Create and encrypt a new LVM Logical Volume / partition

Please refer to the [Exoscale Flexible Storage template documentation][exoscale-doc-flexible-storage]
to create an Exoscale Flexible Storage instance and get acquainted with its (re)partitioning.

[exoscale-doc-flexible-storage]: https://community.exoscale.com/product/compute/instances/how-to/flexible-storage-management/

### Authenticate with the Vault server

Let's start by authenticating with the Vault server:

```
# Retrieve a Vault Token ("login")
# (NB: at the time of writing 'vault login -method=exoscale' is not yet supported)
$ umask 077
$ vault write -field=token auth/exoscale/login \
  role=storage \
  instance=$(wget -qO- http://metadata.exoscale.com/1.0/meta-data/instance-id) \
  zone=$(wget -qO- http://metadata.exoscale.com/1.0/meta-data/availability-zone) \
> ~/.vault-token

# Verify login/permissions
$ vault token lookup
# [output (partial)]
Key                 Value
---                 -----
display_name        exoscale
path                auth/exoscale/login
policies            [default transit-luks]
```

### Create the LUKS passphrase

We may now create a _random_ LUKS passphrase and secure it with Vault:

```
# Create and Vault-encrypt the LUKS passphrase
$ openssl rand -base64 24 \
| vault write -field=ciphertext transit/encrypt/luks plaintext=- \
| tee /etc/luks.passphrase
# [output]
vault:v1:lX3HqaqNdKbB2uD4yfcK8CfWsv/ugRFrTzJSrRFiuowpluFJ9nyGKbPq+HiyJINaq0SOEw==

# Decrypt and backup (!) the LUKS passphrase
$ cat /etc/luks.passphrase \
| vault write -field=plaintext transit/decrypt/luks ciphertext=-
# [output]
kDTAmQFldH0J0yhu3M2zfvHTOZLJMtTY  # !!! SECRET !!!
```

Mark the `/etc/luks.passphrase` being encrypted by Vault and the actual LUKS passphrase
being decrypted on demand by Vault (provided a successful login and proper permissions).

### Create a new LVM Logical Volume for LUKS

Creating the LUKS-dedicated LVM Logical Volume (LV) / partition is straight-forward:

```
$ lvcreate -n lv.luks -L 10G vg.flex
# [output]
  Logical volume "lv.luks" created.
```

### Encrypt the LVM Logical Volume with LUKS

We may now encrypt the new Logical Volume using LUKS:

```
# Install LUKS utilities
$ apt-get install cryptsetup-bin

# Fill the new partition with random data
# (optional but recommended; takes time!)
$ dd if=/dev/urandom of=/dev/mapper/vg.flex-lv.luks bs=4k

# Initialize the LUKS partition
$ cat /etc/luks.passphrase \
| vault write -field=plaintext transit/decrypt/luks ciphertext=- \
| cryptsetup luksFormat --batch-mode --type luks2 --pbkdf argon2id --key-file - /dev/mapper/vg.flex-lv.luks

# Show the LUKS partition status
$ cryptsetup luksDump /dev/mapper/vg.flex-lv.luks
# [output (partial)]
LUKS header information
Version:       	2
Epoch:         	3
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
```

And "open" it:

```
# Open the LUKS partition
cat /etc/luks.passphrase \
| vault write -field=plaintext transit/decrypt/luks ciphertext=- \
| cryptsetup open --type luks --key-file - /dev/mapper/vg.flex-lv.luks luks
```

We can now format its filesystem (example given with `ext4`):

```
$ mkfs.ext4 -L LUKS /dev/mapper/luks
# [output (partial)]
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 2617344 4k blocks and 655360 inodes
```

And mount it:

```
# Create the encrypted data mountpoint
mkdir -p /luks

# Mount the encrypted partition
mount /dev/mapper/luks /luks

# Show the partition capacity/usage
df -h /luks
# [output]
Filesystem        Size  Used Avail Use% Mounted on
/dev/mapper/luks  9.8G   37M  9.3G   1% /luks
```

Mark `/dev/mapper/luks` being the (encrypted) partition to use (_not_ `/dev/mapper/vg.flex-lv.luks`).


## Boot and shutdown automation

In order to automate the various steps to open, mount, unmount and close the LUKS partition,
we will need a control script and systemd unit, which shall open and mount the LUKS partition
on `start`, as well as unmount and close it on `stop`.

Ideally, we would also use the [Vault Agent][hashicorp-vault-agent] to automatically authenticate
our compute instance using the [Vault Exoscale Authentication Plugin][vault-plugin-auth-exoscale]
and act as proxy for the required `vault` commands.

[hashicorp-vault-agent]: https://www.vaultproject.io/docs/agent

For the sake of simplicity, we'll stick to "manually" authenticating with the Vault server, within
the control script itself.

### Control script and systemd unit

Let's start with a small shell script to perform `start`, `stop` and `status` actions;
in `/usr/local/bin/luks-partition`:

```
#!/bin/sh
set -e

case "${1}" in

  'start')
    # Log into Vault
    if ! test -e ~/.vault-token && [ -z "${VAULT_TOKEN}" ] && [ -z "${VAULT_AGENT_ADDR}" ]; then
      VAULT_TOKEN="$(
        vault write -field=token auth/exoscale/login \
              role=storage \
              instance=$(wget -qO- http://169.254.169.254/1.0/meta-data/instance-id) \
              zone=$(wget -qO- http://169.254.169.254/1.0/meta-data/availability-zone)
      )"
      export VAULT_TOKEN
    fi

    # Open the LUKS partition
    if ! test -e /dev/mapper/luks; then
      cat /etc/luks.passphrase \
      | vault write -field=plaintext transit/decrypt/luks ciphertext=- \
      | cryptsetup open --type luks --key-file - /dev/mapper/vg.flex-lv.luks luks
    fi

    # Mount the LUKS partition
    if ! mountpoint -q /luks; then
      mkdir -p /luks
      mount /dev/mapper/luks /luks
    fi
    ;;

  'stop')
    # Unmount the LUKS partition
    if mountpoint -q /luks; then
      umount /luks
    fi

    # Close the LUKS partition
    if test -e /dev/mapper/luks; then
      cryptsetup close luks
    fi
    ;;

  'status')
    mountpoint /luks
    cryptsetup status luks
    ;;

esac
```

And the corresponding systemd unit; in `/etc/systemd/system/luks-partition.service`:

```
[Unit]
Description=Start/stop the LUKS-encrypted partition
After=local-fs.target vault.service

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=VAULT_ADDR=http://127.0.0.1:8200
ExecStart=/usr/local/bin/luks-partition start
ExecStop=/usr/local/bin/luks-partition stop

[Install]
WantedBy=multi-user.target
```

Which we can now enable and start:

```
# Reload systemd configuration
$ systemctl daemon-reload

# Enable the LUKS partition control unit
$ systemctl enable luks-partition

# Start the LUKS partition control unit
$ systemctl start luks-partition

# Query the LUKS partition control unit status
$ systemctl status luks-partition
# [output]
● luks-partition.service - Start/stop the LUKS-encrypted partition
   Loaded: loaded (/etc/systemd/system/luks-partition.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fri 2021-07-02 07:07:28 UTC; 3s ago
  Process: 21515 ExecStart=/usr/local/bin/luks-partition start (code=exited, status=0/SUCCESS)
 Main PID: 21515 (code=exited, status=0/SUCCESS)
```

Should you have a service - e.g. MariaDB database - that has its data on the LUKS partition, just
make sure to set the proper `After=luks-partition.service` dependency in its systemd unit or as
an override. Example given in `/etc/systemd/system/mariadb.service.d/after-luks-partition.conf`:

```
[Unit]
After=luks-partition.service
```