Codify cluster management with Terraform

10min
|
Vault
Terraform
Interactive

Personas

The scenario described in this tutorial introduces the following personas:

admin is the organization-level administrator
student is a user allowed to write data to a path in vault

Challenge

A manual system administration can become a challenge as the scale of infrastructure increases. Often, an organization must manage multiple Vault environments (development, testing, staging, production, etc.). Keeping up with the increasing management demand soon becomes a challenge without some sort of automation.

Solution

One of the pillars behind the Tao of Hashicorp is automation through codification.

HashiCorp Terraform is an infrastructure as code which enables the operation team to codify the Vault configuration tasks such as the creation of policies. Automation through codification allows operators to increase their productivity, move quicker, promote repeatable processes, and reduce human error.

This tutorial demonstrates techniques for creating Vault policies and configurations using Terraform Vault Provider.

Prerequisites

Terraform installed
A Vault environment to connect

Note

If you are running Vault Enterprise, see the Codify Management of Vault Enterprise Using Terraform tutorial.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Scenario introduction

Vault administrators must manage multiple Vault environments. The test servers get destroyed at the end of each test cycle and a new set of servers must be provisioned for the next test cycle. To automate the Vault server configuration, you are going to use Terraform to provision the following Vault resources.

Type	Name	Description
ACL Policy	admins	Sets policies for the admin team
ACL Policy	eaas-client	Sets policies for clients to encrypt/decrypt data through transit secrets engine
auth method	userpass	Enable and create a user, "student" with `admins` and `fpe-client` policies
secrets engine	kv-v2	Enable kv-v2 secrets engine at `kv-v2`
secrets engine	transit	Enable transit secrets engine at `transit`
encryption key	payment	Encryption key to encrypt/decrypt data

The following steps are demonstrated:

Examine the Terraform files

Clone or download the demo assets from the hashicorp/learn-vault-codify GitHub repository to perform the steps described in this tutorial.
```
$ git clone https://github.com/hashicorp/learn-vault-codify.git
```
This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found within a sub-directory.

Change the working directory to learn-vault-codify/community.

$ cd learn-vault-codify/community

The directory contains Terraform files to configure Vault.

$ tree
.
├── auth.tf
├── main.tf
├── policies
│   ├── admin-policy.hcl
│   └── eaas-client-policy.hcl
├── policies.tf
└── secrets.tf

Open the main.tf file in your preferred text editor to examine its content.
main.tf
```
123456789provider "vault" {
  # It is strongly recommended to configure this provider through the
  # environment variables:
  #    - VAULT_ADDR
  #    - VAULT_TOKEN
  #    - VAULT_CACERT
  #    - VAULT_CAPATH
  #    - etc.
}
```
Within the file is a vault provider block. You can provide the server connection details inside this block (Vault server address, client tokens, etc.); however, it is strongly recommended to configure those target server specific information using environment variables. And that's what you are going to do in this tutorial.

Open the policies.tf file and examine the vault_policy resources.

policies.tf

1 2 3 4 5 6 7 8 9 1011# Create admin policy in the root namespace
resource "vault_policy" "admin_policy" {
  name   = "admins"
  policy = file("policies/admin-policy.hcl")
}

# Create 'training' policy
resource "vault_policy" "eaas-client" {
  name   = "eaas-client"
  policy = file("policies/eaas-client-policy.hcl")
}

Open the auth.tf file and note that it enables userpass auth method and creates a user, "student" with admins and eaas-client policies attached. The password is set to "changeme".

auth.tf

1 2 3 4 5 6 7 8 9 1011121314151617resource "vault_auth_backend" "userpass" {
  type = "userpass"
}

# Create a user, 'student'
resource "vault_generic_endpoint" "student" {
  depends_on           = [vault_auth_backend.userpass]
  path                 = "auth/userpass/users/student"
  ignore_absent_fields = true

  data_json = <<EOT
{
  "policies": ["admins", "eaas-client"],
  "password": "changeme"
}
EOT
}

Open the secrets.tf file which enables kv-v2 secrets engine (line 2-5).

secrets.tf

1 2 3 4 5 6 7 8 9 10111213141516171819# Enable K/V v2 secrets engine at 'kv-v2'
resource "vault_mount" "kv-v2" {
  path = "kv-v2"
  type = "kv-v2"
}

# Enable Transit secrets engine at 'transit'
resource "vault_mount" "transit" {
  path = "transit"
  type = "transit"
}

# Creating an encryption key named 'payment'
resource "vault_transit_secret_backend_key" "key" {
  depends_on = [vault_mount.transit]
  backend    = "transit"
  name       = "payment"
  deletion_allowed = true
}

It enables transit secrets engine at transit (line 8-10), and create an encryption key named, "payment" (line 14-19).

Run Terraform to configure Vault

Optional: Start a Vault server in development mode with root as the root token if you don't have one running already.
```
$ vault server -dev -dev-root-token-id root
```
Set the client token in the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN="root"
```
If the token is different, be sure to set it to the correct token value that has permissions to create policies, enable secrets engines, and enable auth methods.
Set the target Vault server address in the VAULT_ADDR environment variable if it's not done so already.
```
$ export VAULT_ADDR="http://127.0.0.1:8200"
```
If you are connecting to a remote Vault server, be sure to set the VAULT_ADDR value to the correct target Vault API address.

Initialize Terraform to pull Vault provider plugin.

$ terraform init

Initializing the backend...
...snip...
Terraform has been successfully initialized!

This will download the Vault plugin.

Execute the apply command to configure Vault.

$ terraform apply

This will display the actions to be performed by Terraform.

When prompted, enter yes to accept the plan and proceed with Vault configuration.

Plan: 7 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Once completed, the output similar to the following displays.

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Verify the configuration

List existing policies to make sure that admins and eaas-client policies were created.
```
$ vault policy list

admins
default
eaas-client
root
```

List enabled secrets engines to verify that kv-v2 and transit secrets engines are enabled.

$ vault secrets list

Path          Type         Accessor              Description
----          ----         --------              -----------
...
kv-v2/        kv           kv_4ae6ced0           n/a
...
transit/      transit      transit_5a1b59df      n/a

List transit keys to make sure that payment key exists.
```
$ vault list transit/keys
Keys
----
payment
```
Unset the VAULT_TOKEN environment variable which you used to run Terraform.
```
$ unset VAULT_TOKEN
```

Now, verify that you can log in with userpass auth method using the username, "student".

$ vault login -method=userpass username=student
Password (will be hidden):

Enter changeme when prompted to enter the password.

Example output:

Key                    Value
---                    -----
token                  s.ac401MQbM0GMv6bS7UuZmxzi
token_accessor         FDnXsZmgCWggMVlsJOjFaCLr
token_duration         768h
token_renewable        true
token_policies         ["default" "eaas-client"]
identity_policies      []
policies               ["default" "eaas-client"]
token_meta_username    student

The generated token has eaas-client policy attached.

Review the eaas-client policy.

$ cat policies/eaas-client-policy.hcl

# Permits CRUD operation on kv-v2
path "kv-v2/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Encrypt data with 'payment' key
path "transit/encrypt/payment" {
  capabilities = ["update"]
}

# Decrypt data with 'payment' key
path "transit/decrypt/payment" {
  capabilities = ["update"]
}

# Read and list keys under transit secrets engine
path "transit/*" {
  capabilities = ["read", "list"]
}

# List enabled secrets engines
path "secret/metadata/*" {
   capabilities = ["list"]
}

Make sure that student can encrypt and decrypt data using the payment key.

$ vault write transit/encrypt/payment \
     plaintext=$(base64 <<< "1111-2222-3333-4444")

Example output:

Key            Value
---            -----
ciphertext     vault:v1:RYVesgy2eJe4LeJ7v38WYmBEDN6vjPnPfvZm1UCYTajIIlVWjFujNYlA6YZ06lRz
key_version    1

Now, decrypt the returned ciphertext.

$ vault write transit/decrypt/payment \
    ciphertext="vault:v1:RYVesgy2eJe4LeJ7v38WYmBEDN6vjPnPfvZm1UCYTajIIlVWjFujNYlA6YZ06lRz"

Example output:

Key          Value
---          -----
plaintext    MTExMS0yMjIyLTMzMzMtNDQ0NAo=

The returned value is base64-encoded and must be decoded to reveal the original input value.
```
$ base64 --decode <<< "MTExMS0yMjIyLTMzMzMtNDQ0NAo="

1111-2222-3333-4444
```

Open a web browser and launch the Vault UI (e.g. http://127.0.0.1:8200/ui). In the sign in page, select Username auth method, enter student in the Username text field, and changeme in the Password text field.
Under Secrets Engines, verify that kv-v2 and transit secrets engines are listed.
Select the Policies tab and verify that admins and eaas-client policies exist.
Return to the Secrets tab. Select View configuration of the transit secrets engine.
Select Keys tab and verify that payment key exists.
Select payment and then Encrypt.
Enter 1111-2222-3333-4444 in the Plaintext field, and then click Encrypt.
Click Copy & Close.
Now, select Decrypt tab and paste in the ciphertext you just copied.
Click Decrypt.
The returned plaintext is base64 encoded. Click Copy & Close.

Execute the base64 --decode command to decode the returned plaintext.

Example:

$ base64 --decode <<< "MTExMS0yMjIyLTMzMzMtNDQ0NAo="

1111-2222-3333-4444

Note

The details about how transit secrets engine works are out of scope for this tutorial. If you are not familiar with transit secrets engine, read the Encryption as a Service: Transit Secrets Engine tutorial.

Clean up

When you are done exploring, you can undo the configuration made by Terraform.

Before running Terraform, make sure that VAULT_TOKEN and VAULT_ADDR environment variables exist and their values are set as you have done so in the Run Terraform to configure Vault section.

Make sure that VAULT_TOKEN and VAULT_ADDR environment variables are set.
```
$ echo $VAULT_ADDR; echo $VAULT_TOKEN
```

Destroy the Vault resources created by Terraform.

$ terraform destroy -auto-approve

...snip...

Destroy complete! Resources: 7 destroyed.

Remove the terraform state files.
```
$ rm *.tfstate.*
```
Unset the VAULT_TOKEN and VAULT_ADDR environment variables.
```
$ unset VAULT_TOKEN VAULT_ADDR
```

Note

To learn more about Terraform, visit Learn Terraform.

Next steps

Treat your Terraform files like any other code and manage them through a version control system such as GitHub. You may integrate it with your favorite CI/CD tool (Jenkins, Travis CI, Circle CI, etc.), always review and test the configuration.

Travis CI example:

You can test your Terraform files against a development server that runs locally, or use a Docker image of Vault.

sudo: false

env:
  - VAULT_ADDR=<target_vault_address>  VAULT_TOKEN=<token>

before_install:
  - scripts/install.sh

before_script:
  - vault server -dev -dev-root-token-id="$VAULT_TOKEN"

script:
  - terraform init
  - terraform apply -auto-approve

Summary

In this guide you learned a technique for creating Vault policies and configurations using the Terraform Vault Provider. For more information, see the help and reference section.

Help and Reference

Terraform Vault Provider documentation page
Terraform Provider GitHub repository
Learn Terraform

Tip

Terraform users can leverage the Vault's dynamic secrets engine to generate short-live cloud credentials when provisioning cloud resources. Inject secrets into Terraform using the Vault provider tutorial demonstrates the use of AWS secrets engine to manage AWS IAM credentials used by Terraform.

Multi-cloud Kubernetes

Codify Vault Enterprise with Terraform

This tutorial also appears in:

13 tutorials

Operations fundamentals
Learn basic Vault operations that are common to both Vault Community Edition and Vault Enterprise users.
- Vault
8 tutorials

Interact with Security tools
Use Terraform to interact with security tooling like HashiCorp Vault and Boundary. Manage Vault and Vault Enterprise. Configure Vault to use AppRole with Chef. Deploy Consul-backed Vault on Kubernetes.
- Terraform
13 tutorials

HashiCorp Products - Better Together
Use Terraform with other Hashicorp products including Vault, Boundary, Consul, Packer, and Hashicorp Cloud Platform.
- Terraform