HashiCorp Vault

Install and configure a self-managed HashiCorp Vault server for development and testing purposes.

Last Updated: 26 May 2022 • Page Author: Jillur Quddus

Overview

HashiCorp Vault is used to secure, store and control access to tokens, passwords, certificates, API keys and other secrets. This page provides instructions on how to install and configure a self-managed instance of HashiCorp Vault server for development and testing purposes only (i.e. non-production).

This page provides instructions on how to install and configure a self-managed instance of HashiCorp Vault server for development and testing purposes only. To configure a secure production-ready instance of HashiCorp Vault, please refer to the HashiCorp Vault documentation.

The instructions below are for Ubuntu 20.04. Installation instructions for other Linux distributions including Fedora, RHEL and Amazon Linux may be found at https://www.vaultproject.io/docs/install.

Installation

Please run the commands below via your command line to install the self-managed open-source version of HashiCorp Vault.

# Add the HashiCorp repository
$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"

# Install HashiCorp Vault
sudo apt-get update && sudo apt-get install vault

Configuration

Integrated Storage

By default HashiCorp Vault will use system memory to manage secrets. In order to deploy Vault using integrated disk-based storage so that secrets are retained between Vault server restarts, we shall create a Vault configuration file and define the absolute path in which to store Vault data. We shall also define the hostname and port on which Vault server will listen for requests. Please create the following configuration file in a location of your choosing, and update the storage and listener configuration namespaces relevant to your environment, in addition to the api_addr and cluster_addr properties respectively.

# Create the Vault configuration file in a location of your choosing
$ touch ~/vault/config.hcl

# Populate the Vault configuration file as follows
# Update the storage.path, listener.address, api_addr and cluster_addr properties
# as per your environment
$ vi ~/vault/config.hcl

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "node1"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = "true"
}

disable_mlock = true
api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
ui = true

Starting Vault Server

Start the Server

To start Vault server using our custom configuration in config.hcl, please run the following command via the command line:

# Start Vault Server using our custom configuration
$ vault server -config=~/vault/config.hcl

==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Go Version: go1.16.2
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: true
           Recovery Mode: false
                 Storage: raft (HA available)
                 Version: Vault v1.7.0
             Version Sha: 4e222b85c40a810b74400ee3c54449479e32bb9f

==> Vault server started! Log data will stream in below:

Initialisation

We must initialise Vault before we can start using it. Initialisation only needs to be performed once for every new backend that has never been used with Vault before. We can initialise Vault either via the command line or via HTTP.

To initialise Vault via the command line, please run the following commands:

# Set the Vault listener address as an environment variable
$ export VAULT_ADDR='http://127.0.0.1:8200'

# Initialize Vault via the CLI
$ vault operator init

Unseal Key 1: 4jYbl2CBIv6SpkKj6Hos9iD32k5RfGkLzlosrrq/JgOm
Unseal Key 2: B05G1DRtfYckFV5BbdBvXq0wkK5HFqB9g2jcDmNfTQiS
Unseal Key 3: Arig0N9rN9ezkTRo7qTB7gsIZDaonOcc53EHo83F5chA
Unseal Key 4: 0cZE0C/gEk3YHaKjIWxhyyfs8REhqkRW/CSXTnmTilv+
Unseal Key 5: fYhZOseRgzxmJCmIqUdxEm9C3jB5Q27AowER9w4FC2Ck

Initial Root Token: s.KkNJYWF5g0pomcCLEmDdOVCW

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

To initialise Vault via HTTP, please run the following commands via your command line, which is dependent on the curl and jq packages respectively.

# Initialize Vault via HTTP
$ curl \
    --request POST \
    --data '{"secret_shares": 1, "secret_threshold": 1}' \
    http://127.0.0.1:8200/v1/sys/init | jq

Unsealing

Vault server starts in a sealed state which requires unsealing - a process by which Vault can decrypt the data stored in the backend. To unseal Vault, you must supply a certain number of unseal keys, where the number is defined by the key threshold (normally three - please see the output immediately after running the vault operator init command). To supply one unseal key via the command line, please run the following command:

# Begin unsealing Vault via the CLI
$ vault operator unseal

To supply one unseal key via HTTP, please run the following command:

# Begin unsealing Vault via HTTP
$ export VAULT_TOKEN="<Initial Root Token>"
$ curl \
    --request POST \
    --data '{"key": "Unseal Key"}' \
    http://127.0.0.1:8200/v1/sys/unseal | jq

Run the unseal command above (either via the CLI or HTTP) the required number of times to unseal the Vault. Once unsealed successfully, the Sealed property in the output will change to false.

KV Secrets Engine v2

POB expects the KV Secrets Engine Version 2 to be enabled. The KV secrets engine is used to store arbitrary secrets as key/value pairs. To enable the KV Secrets Engine Version 2 via the command line, please run the following command:

# Enable the KV Secrets Engine Version 2 via the CLI
$ vault secrets enable -version=2 kv

To enable the KV Secrets Engine Version 2 via HTTP, please run the following command:

# Enable the KV Secrets Engine Version 2 via HTTP
$ export VAULT_TOKEN="<Initial Root Token>"
$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{ "type":"kv-v2" }' \
    http://127.0.0.1:8200/v1/sys/mounts/secret

Access Policies

We shall define a new custom access policy (called pob-policy in our case, but you may name it anything you wish) to enable POB to create, read and update secrets at a custom path (in our case secret/data/pob/*), as follows:

# Create a new access policy for POB to manage secrets at a custom path
$ vault policy write pob-policy - << EOF
path "secret/data/pob/*" {
  capabilities = ["create", "read", "update"]
}
EOF

We shall then attach this newly created policy to a newly generated token (as follows). This token can then be used by POB (configured in the Spring bootstrap context) as part of token-based Vault authentication.

# Attach the policy to a new token
$ export VAULT_TOKEN="$(vault token create -field token -policy=pob-policy)"

# Validate that the token was exported properly and has the correct policies attached
$ vault token lookup | grep policies

Managing Secrets

Environment Variables

To read and write secrets stored in our custom path, ensure that the VAULT_TOKEN environment variable is set with the token that we attached our custom access policy to (see above). Also ensure that the VAULT_ADDR environment variable is set with the Vault listener address. To persist these environment variables across environment restarts, you may wish to define them in your ~/.profile file (or equivalent).

Writing Secrets

To write your very first secret to a custom path, such as secret/pob/development (note that /data is automatically inserted into the path i.e. secret/data/pob/development), please run the following command via the command line:

# Write the first secret to a custom path
$ vault kv put secret/pob/development spring-datasource-url="jdbc:hsqldb:hsql://localhost:9001/pob"

Note that the PUT verb is used to write the very first secret, as it overwrites all other secrets at that path. To add further secrets to the same path, please use the PATCH verb as follows:

# Write further secrets to the same custom path
$ vault kv patch secret/pob/development spring-datasource-username="sa"
$ vault kv patch secret/pob/development spring-datasource-password="password"
$ vault kv patch secret/pob/development spring-datasource-driver-class-name="org.hsqldb.jdbc.JDBCDriver"

Reading Secrets

To read secrets from a custom path, such as secret/pob/development, please use the GET verb as follows:

# Read secrets from a custom path
$ vault kv get secret/pob/development

To read a specific secret stored in a custom path given its key, please use the GET verb along with the -field property as follows:

# Read a specific secret stored in a custom path
$ vault kv get -field=spring-datasource-password secret/pob/development

Updating Secrets

To update a secret in a custom path, simply use the PATCH verb using the same key, as follows:

# Update a secret in a custom path
$ vault kv patch secret/pob/development spring-datasource-username="guest"

Deleting Secrets

To delete secrets stored in custom path, please use the DELETE verb as follows:

# Delete all secrets from a custom path
$ vault kv delete secret/pob/development

POB Context

To configure POB to load externalised variables defined in application.yml from HashiCorp Vault, please configure the spring.cloud.vault namespace in bootstrap.yml, as follows:

Property	Description	Example Value
enabled	Whether to enable externalised configuration from HashiCorp Vault.	true
host	HashiCorp Vault hostname. The host name will be used for SSL certificate validation if required.	localhost
port	HashiCorp Vault port.	8200
scheme	Whether to use http or https.	http
authentication	Authentication mechanism to authorise client requests.	TOKEN
token	If using token-based authentication, then the static token to use. Note that the token should be set as an environmental variable and NOT stored as plaintext in bootstrap.yml. As described above, please use a token that has a policy attached to it that enables POB to read and write secrets at the required custom path.	${VAULT_TOKEN}
kv.enabled	Whether to enable the Key-Value secrets backend, supporting storage of arbitrary values as key-value store.	true
kv.backend	The path of the secret mount to use.	secret
kv.default-context	The context name used by all applications.	pob/development

For further information regarding configuring Spring Cloud Vault with secret backends, please visit https://cloud.spring.io/spring-cloud-vault/multi/multi_vault.config.backends.html.