Testing Framework in Terraform 1.6: A deep-dive

In my earlier weblog submit A Complete Information to Testing in Terraform: Hold your exams, validations, checks, and insurance policies so as I went by all of the choices for testing and validation which are accessible to you once you write your Terraform configurations and modules. We noticed verify blocks, pre-conditions and post-conditions associated to a useful resource’s lifecycle, customized circumstances for enter variables and output values, and extra. The most recent and biggest matter I lined was the testing framework that arrived in Terraform 1.6.

On this submit I wish to focus extra on the testing framework to uncover the probabilities that it brings.

Background on testing with Terraform

If that is the primary time you hear concerning the new testing framework for Terraform I wish to offer you a brief introduction to what it’s.

Terraform 1.6 introduced a brand new testing framework to basic availability after it had been accessible as an experimental characteristic for a time period. The release notes for model 1.6 listed a single new characteristic¹:

terraform take a look at: The terraform take a look at command is now typically accessible. This comes with a major change to how exams are written and executed, based mostly on suggestions from the experimental section.

Terraform exams are written in .tftest.hcl information, containing a collection of run blocks. Every run block executes a Terraform plan and non-compulsory apply in opposition to the Terraform configuration underneath take a look at and might verify circumstances in opposition to the ensuing plan and state.

What we study from the discharge notes is that we will now write exams for our Terraform configurations by together with a number of .tftest.hcl information that every include a collection of run blocks. We additionally study {that a} run block runs terraform plan or terraform apply instructions. Which means that these exams may create actual infrastructure out of your configuration. For me it is a good factor. I’m a robust believer in in terms of testing infrastructure-as-code there is no such thing as a method to make certain it should work until you truly deploy it for actual. Why is that? There are simply too many issues that might go flawed. There could be hidden dependencies that you haven’t any concept of earlier than you truly attempt to create new infrastructure.

What shouldn’t be clear from the discharge notes is who this testing framework is meant for. Is it meant for all Terraform customers? Must you instantly bounce on the TDD-with-Terraform practice and begin writing exams for all Terraform configuration? This isn’t the case. A minimum of this isn’t the first supposed case. The testing framework is designed for module producers.

Are you answerable for creating infrastructure modules internally on your group or publicly for the Terraform group? Then you’re a module producer and this testing framework is for you.

Are you consuming modules so as to create the infrastructure on your software? Then this testing framework shouldn’t be primarily supposed for you. Nevertheless, there’s nothing stopping you from utilizing it when you assume it is smart on your state of affairs.

Module producers write code that different customers will devour. Customers of your modules rely on the contract that your module exposes. What’s a part of the module contract? Typically this consists of the next objects:

• The enter variables your module expects.
• The output values your module produces.
• Any externally seen assets your module creates. This might embrace configuration information, software gateways in Azure, community loadbalancers in AWS, a Cloud Bigtable desk in GCP, or it could possibly be just about anything.

That final level may make you surprise if there are any assets created in a module that’s not a part of the contract? There undoubtedly could possibly be. Some assets could possibly be inner implementation particulars which are required so as to assemble the remainder of the infrastructure. If a useful resource could possibly be swapped out with a unique useful resource with out module shoppers noticing then it’s an inner implementation element and never a part of the contact.

In case you are a module producer and also you make an replace of your module the place you unintentionally make a major change to the contract your module exposes, then this error may find yourself inflicting hassle on your module shoppers.

That is precisely the reasoning behind every other type of take a look at in software program growth, testing Terraform modules is not any completely different!

One final level to make concerning the new testing framework is that you simply write your exams in HashiCorp Configuration Language (HCL). This implies there is no such thing as a have to study a brand new language so as to write exams on your Terraform code. There isn’t any want to put in a further take a look at software that you could preserve monitor of and replace and scan for vulnerabilities and so forth. There isn’t any want to combine your Terraform configuration with a bunch of test-related framework information. Run your exams and deploy your infrastructure utilizing one and the identical Terraform binary.

With all that background out of the way in which, let’s transfer on to seeing all of the nitty-gritty particulars of what this testing framework can do.

Nomenclature

Generally I have to remind myself of the nomenclature of the Terraform HCL code. To ensure we’re all on the identical web page I introduce the nomenclature I exploit right here:

• A block are containers of different content material. A block have a block sort and nil or extra labels. The generic block seems like this:

  <BLOCK TYPE> "<BLOCK LABEL 1>" "<BLOCK LABEL 2>" ... "<BLOCK LABEL N>" {
    # block content material
  }
  

  In Terraform the variety of labels is zero, one, or two. A block can include different blocks. Some frequent block varieties are terraform, supplier, useful resource, information, and run.

• An expression represents a literal worth equivalent to a string or a quantity, or they could possibly be referencing different values such because the title of a useful resource. An expression is also extra complicated consisting of perform calls, string interpolations, and references. Some examples of expressions:

  "it is a string worth"
  12345
  azurerm_storage_account.my_account.title
  "rg-${var.title}-${var.location}"
  

• An argument is the project of a price (from an expression) to a reputation. Arguments seem inside blocks. Some examples of arguments:

  resource_group_name = "rg-my-resource-group"
  sort = record(string)
  depend = size(var.number_of_regions)
  

Even when the small print of the HCL had been acquainted from earlier than the nomenclature could be unfamiliar.

Check framework patterns

The run block is what executes a given take a look at once we run terraform take a look at. This block is central to the testing framework, so it is a block you could grow to be conversant in.

Within the following subsections I’ll undergo three testing patterns² that you simply may see in Terraform.

Earlier than we have a look at the patterns let’s briefly have a look at a typical listing construction for Terraform exams.

To start with, do not forget that your take a look at information ought to use the .tftest.hcl file ending. If not, the terraform take a look at command won’t execute the run blocks on your exams.

While you execute terraform take a look at you ought to be positioned within the module root listing. That is the listing the place your main foremost.tf file exists. The Terraform binary will search for your take a look at information within the root listing or in a listing named exams. I like to recommend you place your exams within the exams listing, and don’t place take a look at information within the module root listing.

A typical listing construction for a easy module with exams is that this:

$ tree .
.
├── foremost.tf
├── outputs.tf
├── suppliers.tf
├── exams
│   └── foremost.tftest.hcl
└── variables.tf

When you place your take a look at information elsewhere you could add -test-directory=path/to/exams to the terraform take a look at command. However as soon as once more I like to recommend that you simply preserve your take a look at information within the exams listing to keep away from complicated future contributors to your module.

What number of take a look at information ought to you have got? The straightforward reply is it relies upon! In case you are constructing a big module with many shifting components you’ll most likely have to have a number of take a look at information divided up into coherent and associated components that take a look at a sure a part of your module. In case your module have a small interface (variables and outputs) it would suffice with a single take a look at file. Use frequent sense right here, if it seems like a file has too many exams then it possible does.

Sample 1: Assertions

The primary sample is straightforward, it consists of a run block with a nested assert block:

run "file_contents_should_be_valid_json" {
  command = plan

  assert {
    situation     = strive(jsondecode(local_file.config.content material), null) != null
    error_message = "The file shouldn't be legitimate JSON"
  }
}

Let’s break down this run block:

• The run block has one label. This label is the title of the take a look at. You need to give your take a look at a self-explanatory title that describes what it does. If the take a look at fails you must instantly know why it failed. On this instance the title is file_contents_should_be_valid_json. If this take a look at fails I do know that the contents of a file was not legitimate JSON.
• This run block executes a terraform plan command. You specify what command you prefer to the take a look at to execute within the command = <command> argument. When you depart this out it should default to execute an apply command. Personally I believe it’s a good suggestion to be clear and at all times add the command = <command> argument to be specific in what the take a look at does.
• The run block can include zero or extra nested assert blocks. Every assert block has a situation = <expression> argument the place <expression> ought to consider to true or false to point if the take a look at passes (true) or fails (false). If <expression> evaluates to false then the expression in error_message = <expression> will probably be exhibited to the terminal (or in Terraform Cloud). On this case the error message is The file shouldn't be legitimate JSON.

Though this instance confirmed a single run block containing a single assert block, do not forget that you can embrace a number of run blocks, every containing a number of assert blocks.

Sample 2: Anticipating failures

The second sample issues exams the place we anticipate the take a look at to fail, and we would like the take a look at to report success if it does. It is a frequent testing technique. The next run block has a nested variable block and an expect_failures = [ ... ] argument:

run "bad_input_url_should_stop_deployment" {
  command = plan

  variables {
    config_url = "http://instance.com"
  }

  expect_failures = [
    var.config_url
  ]
}

There are some new issues to have a look at on this run block so let’s break it down:

• The variables block lets you present enter values for any variables that your module expects. On this case the variables block is offered as a nested block to the run block, however it is also offered as a standalone block exterior of any run blocks. In that case the values would apply to all run blocks in your entire file. When you embrace a standalone variables block you may nonetheless embrace nested variables block inside a run block to override the worldwide values.
• The expect_failures = [ ... ] specifies that we anticipate this take a look at to fail, and we record the explanations for failure within the array expression of the argument. On this explicit instance I say that I anticipate this take a look at to fail as a result of variable named config_url. This mainly imply that I validate the worth offered for the config_url variable in my Terraform module, and the worth offered on this take a look at (http://instance.com) ought to end in a failing validation. If the plan can proceed as regular with none failures, then this take a look at would fail.

It’s price spending a while discussing expect_failures. The values on this array have to be checkable objects with related customized circumstances. In my previous article I wrote rather a lot about customized circumstances. Objects that may embrace customized circumstances are variables, outputs, assets, information sources, and verify blocks.

An necessary level about these customized circumstances is that every one of them aside from the verify block will truly trigger Terraform to halt the execution of a plan or apply operation. What does this imply on your exams? It signifies that if you wish to mix expect_failures with assert blocks you must watch out in the way you assemble your module and your corresponding exams. When you embrace a variable within the expect_failures array of values and on the similar time have an assert block that expects a plan to complete, then the assert block would by no means even be evaluated as a result of the customized situation for the variable would halt the execution of the plan.

For that reason I counsel you retain your exams to both use a number of assert blocks, or use the expect_failures = [ ... ] argument, however not each until you actually know what you’re doing.

Word that the array worth to expect_failures may include a number of values. However you almost certainly wouldn’t wish to combine the kind of checkable objects you embrace on this array due to the rationale mentioned above.

Sample 3: Utilizing helper modules

Generally it’s essential to create supporting infrastructure earlier than you may take a look at your module. This could possibly be the case in case your module creates assets in Azure and it expects that you simply use an present useful resource group for all of the assets. So as to take a look at a module like that there have to be an present useful resource group you should utilize. A easy resolution to that is to create a useful resource group up entrance and simply let it sit there in your cloud surroundings for so long as required. A greater resolution is to create the useful resource group once you launch the terraform take a look at command.

For example what this seems like we now have the next listing construction:

$ tree .
.
├── foremost.tf
├── outputs.tf
├── testing
│   └── setup
│       └── foremost.tf
├── exams
│   └── foremost.tftest.hcl
└── variables.tf

4 directories, 5 information

I’ve created a testing listing that accommodates a setup listing with a foremost.tf file. The contents of this file is:

// testing/setup/foremost.tf
variable "resource_group_name" {
  sort = string
}

variable "location" {
  sort = string
}

useful resource "azurerm_resource_group" "rg" {
  title     = var.resource_group_name
  location = var.location
}

It’s a easy file that makes use of the azurerm supplier to create a useful resource group. The module underneath take a look at is:

// foremost.tf
terraform {
  required_providers {
    azurerm = {
      supply  = "hashicorp/azurerm"
      model = ">= 3.0.0"
    }
  }
}

locals {
  resource_name_suffix = "${var.name_suffix}-${var.location}"
}

useful resource "azurerm_service_plan" "plan" {
  title                = "plan-${native.resource_name_suffix}"
  resource_group_name = var.resource_group_name
  location            = var.location
  os_type             = "Linux"
  sku_name            = var.appservice_plan_sku
}

useful resource "azurerm_linux_web_app" "app" {
  title                = "app-${native.resource_name_suffix}"
  service_plan_id     = azurerm_service_plan.plan.id
  resource_group_name = var.resource_group_name
  location            = var.location

  https_only = true

  site_config {
    always_on           = accommodates(["Free", "F1", "D1"], var.appservice_plan_sku) ? false : true
    http2_enabled       = true
    minimum_tls_version = "1.2"
  }
}

This module creates two assets. An App Service plan and a Linux Net App. The variables.tf file has the next content material:

// variables.tf
variable "name_suffix" {
  sort = string
}

variable "resource_group_name" {
  sort = string
}

variable "location" {
  sort = string
}

variable "appservice_plan_sku" {
  sort = string

  validation {
    situation = accommodates([
      "B1", "B2", "B3", "D1", "F1", "S1", "S2", "S3"
    ], var.appservice_plan_sku)
    error_message = "Please present a legitimate App Service Plan SKU"
  }
}

How can we create the useful resource group module earlier than we run our exams? The take a look at file foremost.tftest.hcl seems like this:

// exams/foremost.tftest.hcl
supplier "azurerm" {
  options {}
}

variables {
  resource_group_name = "rg-app-service-test"
  location            = "swedencentral"
  appservice_plan_sku = "F1"
  name_suffix         = "apptest"
}

run "setup" {
  module {
    supply = "./testing/setup"
  }
}

run "http_should_not_be_allowed" {
  command = plan

  assert {
    situation = azurerm_linux_web_app.app.https_only == true
    error_message = "Net App accepts HTTP visitors"
  }
}

run "confirm_always_on" {
  command = plan

  variables {
    name_suffix = "testingalwayson"
    appservice_plan_sku = "S1"
  }

  assert {
    situation = azurerm_linux_web_app.app.site_config[0].always_on == true
    error_message = "At all times-On is off for S1 SKU"
  }
}

There are few new issues to have a look at on this take a look at file. Let’s break it down.

To start with we configure the azurerm supplier:

supplier "azurerm" {
  options {}
}

This lets you configure the supplier in any method that matches your exams. On this case I exploit default settings (an empty options block is required). Word that you can additionally configure the supplier to make use of a separate take a look at subscription as an alternative of every other default subscription you have got configured.

The subsequent piece within the take a look at file defines world variables:

variables {
  resource_group_name = "rg-app-service-test"
  location            = "swedencentral"
  appservice_plan_sku = "F1"
  name_suffix         = "apptest"
}

These variables will probably be used for the setup module and all the next exams, until the exams override these values.

Subsequent we now have the setup module:

run "setup" {
  module {
    supply = "./testing/setup"
  }
}

A setup (or helper) module is created in its personal run block. I set the label of this block to setup, however you may set it to no matter matches your context. The run block solely accommodates a nested module block that specifies the supply of the module to be my module positioned within the testing/setup listing. This run block is the primary run block within the take a look at file, so it will likely be run first (they’re run so as). If I place the setup run block elsewhere within the file then the exams outlined above the setup block would fail as a result of the useful resource group wouldn’t exist.

The remainder of the file accommodates two exams in two separate run blocks. The primary block is just like what we now have seen earlier than, however discover that we now have a nested variables block within the different run block:

run "confirm_always_on" {
  command = plan

  variables {
    name_suffix = "testingalwayson"
    appservice_plan_sku = "S1"
  }

  assert {
    situation = azurerm_linux_web_app.app.site_config[0].always_on == true
    error_message = "At all times-On is off for S1 SKU"
  }
}

Which means that for this take a look at we override the name_suffix and appservice_plan_sku variables.

I can run the exams with terraform take a look at:

$ terraform take a look at
exams/foremost.tftest.hcl... in progress
  run "setup"... go
  run "http_should_not_be_allowed"... go
  run "confirm_always_on"... go
exams/foremost.tftest.hcl... tearing down
exams/foremost.tftest.hcl... go

Success! 3 handed, 0 failed.

Discover that the output says 3 handed though we solely actually had two exams. How come? It is because the setup module runs inside a run block, so it’s thought of to be a take a look at from Terraform’s standpoint. I believe it is a bit unlucky, however for now we’ll should stay with it.

My exams on this case used command = plan, so they’re comparatively quick to run. While you use command = apply you must put together for a doubtlessly lengthy take a look at run, relying on what assets your module creates. Think about when you have a module that creates a number of Kubernetes clusters and installs varied elements in these clusters, then an apply may take a while. Particularly when you run a number of impartial exams.

Terraform take a look at state file and useful resource destruction

How does Terraform know what assets it ought to take away and in what order it ought to do it? In case you are conversant in Terraform you understand that there’s often a state file someplace. While you run exams Terraform retains the state information in reminiscence, so you will not see any state information showing in your module listing.

Terraform creates one state file for the primary configuration underneath take a look at, and one state file for every alternate module that you simply create by a run block. An instance of an alternate module is the setup module we noticed in an instance above.

The state information are created within the order of the exams, and they’re destroyed within the reverse order. An illustrative pattern of what state information are created, up to date, and destroyed and in what order, is proven beneath:

// first alternate module name
// creates state file for modules/setup-1.tf
run "setup-1" {
  module {
    supply = "modules/setup-1.tf"
  }
}

// second alternate module name
// creates state file for modules/setup-2.tf
run "setup-2" {
  module {
    supply = "modules/setup-2.tf"
  }
}

// take a look at the primary configuration
// creates the primary statefile for the module's foremost.tf
run "test-1" {
  assert {
    ...
  }
}

// third alternate module name, as soon as once more to the setup-2 module
// updates the state file for modules/setup-2.tf
run "setup-2-again" {
  module {
    supply = "modules/setup-2.tf"
  }
}

// second take a look at for the primary configuration
// updates the primary statefile for the module's foremost.tf
run "test-2" {
  assert {
    ...
  }
}

// After all the things is run clean-up begins:
// 1. The module's foremost.tf state file is destroyed
// 2. The alternate modules state information are destroyed in
//    reverse order from how they had been created
//      - first the modules/setup-2 state file is destroyed
//      - then the modules/setup-1 state file is destroyed

A query that got here to my thoughts once I first heard concerning the take a look at framework was what occurs if the take a look at fails the destruction of assets? Let’s examine what occurs!

I’ll run a take a look at the place an Azure App Service is created. I’ll use the setup module from earlier than the place I created a useful resource group. To make the take a look at fail I’ll difficulty the next Azure CLI command so as to lock the useful resource group in order that Terraform cannot destroy it:

$ az lock create 
  --name failure 
  --resource-group rg-app-service-test 
  --lock-type ReadOnly

The take a look at output is the next:

$ terraform take a look at
exams/foremost.tftest.hcl... in progress
  run "setup"... go
  .
  . (output truncated)
  .
Terraform left the next assets in state after executing
exams/foremost.tftest.hcl/http_should_not_be_allowed, and they should
be cleaned up manually:
  - azurerm_linux_web_app.app
  - azurerm_service_plan.plan

There we go!

We’re instructed that a variety of assets should be cleaned up manually. In Azure that is often comparatively easy when you have put all assets in the identical useful resource group. Nevertheless, in case you are working with AWS you could be in for a tedious cleanup course of in case your module created a number of assets!

I can see that this conduct could possibly be a difficulty throughout growth of your module and exams the place you aren’t certain if all the things works as supposed. You’ll most probably find yourself with just a few failing take a look at cleanups.

Exploring the take a look at command within the CLI

To cowl all the things we will concerning the take a look at framework let’s have a look at what else we will do with the terraform take a look at command:

$ terraform take a look at -h
Utilization: terraform [global options] take a look at [options]

[ background description truncated for brevity ... ]

Choices:

  -cloud-run=supply     If specified, Terraform will execute this take a look at run
                        remotely utilizing Terraform Cloud. You could specify the
                        supply of a module registered in a personal module
                        registry because the argument to this flag. This permits
                        Terraform to affiliate the cloud run with the right
                        Terraform Cloud module and group.

  -filter=testfile      If specified, Terraform will solely execute the take a look at information
                        specified by this flag. You need to use this feature a number of
                        instances to execute a couple of take a look at file.

  -json                 If specified, machine readable output will probably be printed in
                        JSON format

  -no-color             If specified, output will not include any colour.

  -test-directory=path  Set the Terraform take a look at listing, defaults to "exams".

  -var 'foo=bar'        Set a price for one of many enter variables within the root
                        module of the configuration. Use this feature greater than
                        as soon as to set a couple of variable.

  -var-file=filename    Load variable values from the given file, as well as
                        to the default information terraform.tfvars and *.auto.tfvars.
                        Use this feature greater than as soon as to incorporate a couple of
                        variables file.

  -verbose              Print the plan or state for every take a look at run block because it
                        executes.

There are just a few fascinating flags we will use. I wish to spotlight just a few:

• -cloud-run=supply is helpful when you have your module in a personal registry in Terraform Cloud, and also you wish to set off a take a look at run within the cloud. I’ll cowl testing in Terraform Cloud in a future submit.
• -filter is helpful when you have a number of take a look at information and you’d solely need one or just a few of the information to run. That is particularly helpful in case you are testing a big module the place your exams execute apply operations that take a very long time.
• -test-directory is helpful when you place your take a look at information elsewhere than within the exams listing. However as I discussed earlier on this article you must most likely follow utilizing the exams listing.

Abstract

On this submit we now have checked out most of what the brand new testing framework for Terraform 1.6 has to supply. That’s actually not true. There are extra we will say concerning the testing framework in terms of Terraform Cloud. In a future submit I’ll cowl how we run exams in Terraform Cloud and among the distinctive options which are accessible there.

The instance patterns on this submit have deliberately been left comparatively easy. In actuality creating good exams on your modules would require a number of work. My goal of this submit has been for example what we will do, what syntax is accessible, and some of the behaviors we will anticipate from this framework.

I anticipate that there will probably be extra options added as HashiCorp receives suggestions from customers of this framework. We stay in thrilling instances!

1. Aside from this single new characteristic there have been enhancements and bug fixes included as effectively. ↩

2. I name them patterns right here to make use of a well-recognized nomenclature. As with all patterns you’ll most probably not see them remoted in the actual world. All patterns I current are most probably blended and matched for actual Terraform modules. The concept with patterns right here is to introduce the testing framework piece by piece. ↩