Late Night Terraform: The Vault Door

🎙️ Opening Monologue

There’s a specific kind of panic that only happens at 2:00 AM — the moment you realize an API key might have slipped into a public repo.

The house is silent, but my heart isn’t. In a world where we’re taught to codify everything, some things should never be written down in plain sight. Late nights have a way of making that painfully clear.

Tonight is about learning restraint in a language designed to reveal intent. About deciding what must be visible, and what must remain unseen, even when automation pushes toward transparency.

Infrastructure can be reproducible, but secrets must remain sacred.

Let’s secure the vault.

🎯Episode Objective

This episode aligns with the Terraform Associate (004) exam objectives listed below.

Understand best practices for managing sensitive data, including secrets management with Vault

Managing secrets in Terraform is a balancing act. You need to provide sensitive data (like API keys, passwords, or certificates) to your infrastructure without accidentally committing those “crown jewels” to version control or exposing them in plaintext logs.

The Exposure Risk: Why Secrets Demand Special Handling

Before looking at tools, you must understand that Terraform state files (.tfstate) store everything in plaintext. Even if you use sensitive variables or encrypted backends, the resulting state file will contain your secrets in the clear.

Rule #1: Never commit .tfstate files to Git.
Rule #2: Always use a Remote Backend (like AWS S3, Azure Blob Storage, or HashiCorp Cloud) that supports encryption at rest.

The Layered Defense: Strategies for Managing Sensitive Values

1. Marking Variables as Sensitive

The sensitive argument is your primary defense against “accidental peering.” When you mark a variable or an output as sensitive, Terraform replaces the value with (sensitive value) in the console.

variable "admin_password" {
  type      = string
  sensitive = true
}

Protecting Outputs

If you need to export a value (for example, to another module), you must also mark the output as sensitive or Terraform will throw an error to prevent accidental leakage.

output "db_connect_string" {
  value     = "admin:${var.admin_password}@db.example.com"
  sensitive = true
}

2. Variable Definition Files (`.tfvars`)

You can use a secrets.tfvars file, but you must ensure your .gitignore is configured to ignore it. Note: This is generally discouraged for production because it’s too easy for someone to accidentally commit the file.

Environment Variables (The Simple Way)

Terraform automatically looks for environment variables prefixed with TF_VAR_. This keeps secrets out of your code entirely.

How it works: If you have a variable named db_password, set it in your terminal: export TF_VAR_db_password="supersecretpassword"
Best for: Local testing or simple CI/CD pipelines.

3. Protecting Source Control: Dynamic Fetching

The best way to keep secrets out of source control is to ensure they never enter your .tf files to begin with. Instead of hardcoding values, use Data Sources to fetch secrets at runtime.

By using a data source, the secret exists in memory during the Terraform run but is never written into your code repository.

data "azurerm_key_vault_secret" "db_password" {
  name         = "admin-password"
  key_vault_id = data.azurerm_key_vault.main.id
}

resource "azurerm_linux_virtual_machine" "example" {
  # ... other config ...
  admin_password = data.azurerm_key_vault_secret.db_password.value
}

The Root of Trust: Authenticating Providers Without Hardcoding Secrets

Terraform Providers (like the AWS, Azure, or Google Cloud providers) often have built-in intelligence to protect you automatically.

Many resource attributes are hardcoded as “sensitive” by the developers of the provider. This means even if you forget to use the sensitive = true flag in your own code, the provider acts as a safety net.

How Provider-Level Sensitivity Works

When a provider developer defines a resource — for example, a database password or a private key — they mark that specific attribute as sensitive in the resource schema.

During plan and apply:** Terraform reads the schema, sees the “sensitive” tag, and automatically masks the value in your terminal.
Nested Sensitivity: If a sensitive attribute is part of a complex object (like a list or a map), Terraform will often mask the entire block to be safe.

The Ghost in the Machine: How Sensitive Data Lives in State

While the sensitive flag merely hides data from your screen, ephemeral values ensure that sensitive data never touches your disk. They exist only in memory during a single Terraform operation (like a plan or apply) and are completely omitted from the .tfstate file and plan artifacts.

How It Works: The “Memory-Only” Lifecycle

Normally, every data source and resource attribute is saved to your state file so Terraform can track changes. Ephemeral values break this cycle by being transient by design.

Opening: Terraform fetches or generates the value (e.g., calling an API or generating a password) only when it is needed during the run.
Usage: The value is passed to a provider or a “write-only” argument.
Closing: Once the operation finishes, the value is flushed from memory. No trace is left in the local or remote .tfstate.

Ways to Define Ephemeral Values

You can inject ephemerality into your configuration at different entry points:

Ephemeral Variables: Mark a variable with ephemeral = true. This is perfect for short-lived session tokens or passwords passed into Terraform via the command line or environment variables.
Ephemeral Blocks: A new block type (like resource or data) that fetches or generates data on-the-fly. Providers must specifically support these (e.g., ephemeral "random_password").
Child Module Outputs: Mark a module’s output as ephemeral = true to pass temporary data to a parent module without that data becoming part of the parent’s persistent state.
Write-only Arguments: These are the “exit points.” A managed resource attribute (like password_wo) can accept an ephemeral value, use it to configure the cloud resource, and then immediately discard it.

Where Can You Reference Them?

Terraform enforces a strict sandbox for ephemeral values. If you try to use an ephemeral value in a standard resource attribute (like a regular tags map or a non-write-only password), Terraform will throw an error. This is to ensure an ephemeral value doesn’t “leak” into a persistent field and end up in the state file.

Allowed Contexts:

Locals: You can process ephemeral values (e.g., local.pw = "prefix-${ephemeral.random_password.pw.result}"). The resulting local value automatically inherits the “ephemeral” status.
Provider Blocks: Use ephemeral tokens to authenticate the provider itself (e.g., a temporary AWS session token).
Write-only Arguments: Pass secrets into specific resource arguments designed not to save to state.
Provisioner & Connection Blocks: Use them to provide temporary SSH keys or passwords for remote-exec or file provisioners.
Other Ephemeral Blocks: Pass one ephemeral result into the configuration of another ephemeral resource.

Write-Only Arguments (_wo)

Introduced in Terraform 1.11, these are the “terminal points” for ephemeral data. Standard resource arguments save data to state; Write-only arguments (which often end in the suffix _wo) accept the data, send it to the cloud provider’s API, and then immediately discard it.

# Fetch the secret from Key Vault ephemerally
ephemeral "azurerm_key_vault_secret" "fetched_pass" {
  name         = azurerm_key_vault_secret.sql_secret.name
  key_vault_id = azurerm_key_vault.main.id
}

resource "azurerm_mssql_server" "sql" {
  name                         = "my-secure-sql-server"
  resource_group_name          = azurerm_resource_group.main.name
  location                     = azurerm_resource_group.main.location
  version                      = "12.0"
  administrator_login          = "sqladmin"
  # Uses the WRITE-ONLY argument to prevent state-logging
  administrator_login_password_wo         = ephemeral.azurerm_key_vault_secret.fetched_pass.value
  administrator_login_password_wo_version = "1"
}

🌙 Late-Night Reflection

Complexity doesn’t disappear; it just changes shape. Moving toward automation at scale means you’re no longer just a builder, but a factory foreman. If you don’t build your components with the intention of them being reused, you’re just creating a bigger mess at a faster speed.

✅ Key Takeaways

Terraform state is always the risk surface Terraform stores resource attributes in plaintext inside the state file. Masking output does not equal securing data. Protecting state storage is non-negotiable.
sensitive = true controls visibility, not storage Sensitive variables and outputs hide values from CLI output and logs, but the values still exist in the state unless explicitly prevented.
Never commit .tfstate or secret .tfvars files Secrets should never live in version control. Use remote backends with encryption at rest and strict access controls.
Remote backends are mandatory for secure teams S3, Azure Blob, GCS, or HCP Terraform provide encryption, locking, and recovery. Local state is unsafe beyond experimentation.
Prefer dynamic secret retrieval over static secrets Fetch secrets at runtime using data sources (Vault, AWS Secrets Manager, Azure Key Vault) instead of embedding them in variables or files.
Provider schemas enforce hidden sensitivity automatically Many provider-defined attributes (passwords, private keys) are marked sensitive by default, masking them even if you forget to do so.
Ephemeral values prevent persistence, not misuse Ephemeral variables and outputs are designed to avoid being written to state or plan files — but operational hygiene (logging, CI artifacts) still matters.
Write-only arguments are the safest exit point for secrets Provider-supported write-only fields (*_wo) allow Terraform to send secrets to APIs without persisting them in state.
Sensitive data propagates downstream If a sensitive value is used to compute another value, Terraform automatically treats the derived value as sensitive as well.
Security is layered, not a single flag

📚 Further Reading

Manage sensitive data documentation

🎬 What’s Next

Everything is secure and smart — but it’s getting crowded. Repetition at scale is a recipe for chaos.

We’ll break infrastructure into reusable parts designed to scale cleanly.