Remote Backend – Sesame Disk Group

What is Terraform State?

Terraform tracks your real infrastructure using a state file (terraform.tfstate). This file is the source of truth for resource mappings, attributes, and dependencies. Every terraform plan or terraform apply operation consults the state to compute the delta between configuration and reality.

Resource metadata (IDs, attributes, outputs)
Dependency graph for correct update ordering
Provider configuration references
Sensitive data (sometimes in plain text, e.g., secrets or passwords)

Losing, corrupting, or mishandling state can lead to orphaned resources, duplicate creations, or even cloud resource deletion. In production, how you manage state is as important as how you write your Terraform code.

Why Local State is a Production Risk

By default, Terraform writes state to a local file, terraform.tfstate, in the working directory. This approach is acceptable only for disposable test projects or single-user experiments. As soon as you have a team or need auditability, local state is a liability:

No access control or audit trail: Anyone with the file can modify or leak infrastructure details.
No locking: Multiple people can run terraform apply simultaneously, corrupting the state.
No versioning or backups: A bad apply or accidental rm is irreversible.
Secrets exposure: State may contain credentials in plain text.

Here’s a visualization of how local state can break down in a team:


graph TD
    A[Developer A] -->|Reads local state| B[terraform.tfstate on A's machine]
    C[Developer B] -->|Reads local state| D[terraform.tfstate on B's machine]
    B -->|Diverges from| D
    B -->|Applies changes| E[Cloud Resources]
    D -->|Applies conflicting changes| E
    E -->|State mismatch| F[Corrupted Infrastructure]

For any real-world, team-based, or auditable infrastructure, move state off local disks and into a hardened, remote backend.

Remote Backends, State Locking, and Organization

A remote backend is a storage location (S3, GCS, Azure Blob Storage, HashiCorp Terraform Cloud, etc.) that holds your state file and (ideally) provides:

Centralized, shared state for team collaboration
State locking to prevent concurrent modification
Encryption at rest and in transit
Versioning and recovery

Here’s how a typical AWS S3 + DynamoDB backend with locking and encryption is defined:


# backend.tf
terraform {
  backend "s3" {
    bucket         = "my-company-terraform-state"
    key            = "projects/web-app/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-locks"
    encrypt        = true
  }
}

Before using this, you must bootstrap the backend resources securely:


# bootstrap/main.tf
resource "aws_s3_bucket" "terraform_state" {
  bucket = "my-company-terraform-state"
  lifecycle { prevent_destroy = true }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-state-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}

This setup ensures:

Only one terraform apply can proceed at a time (state locking via DynamoDB)
State is versioned (recovery from accidental deletion/corruption is possible)
Encryption is enforced at rest (KMS) and in transit (TLS by default)
Public access is blocked at the storage level

For GCP and Azure, similar patterns apply:


# GCP backend.tf
terraform {
  backend "gcs" {
    bucket = "my-company-terraform-state"
    prefix = "projects/web-app"
  }
}

# Azure backend.tf
terraform {
  backend "azurerm" {
    resource_group_name  = "terraform-state-rg"
    storage_account_name = "mycompanytfstate"
    container_name       = "tfstate"
    key                  = "projects/web-app/terraform.tfstate"
  }
}

Not all backends support full locking. Always check the backend documentation for support details.

Production-Grade Terraform State Backend Examples

Below are real configuration snippets from production deployments for major cloud providers. Each ensures versioning, encryption, and state isolation.

AWS S3 with DynamoDB Locking


terraform {
  backend "s3" {
    bucket         = "sesamedisk-tfstate"
    key            = "prod/network/terraform.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "tfstate-locks"
    encrypt        = true
  }
}

GCP Cloud Storage


terraform {
  backend "gcs" {
    bucket = "sesamedisk-tfstate"
    prefix = "prod/network"
  }
}

Azure Blob Storage


terraform {
  backend "azurerm" {
    resource_group_name  = "sesamedisk-terraform-rg"
    storage_account_name = "sesameterraform"
    container_name       = "tfstate"
    key                  = "prod/network/terraform.tfstate"
  }
}

Each example above assumes you have locked down access with IAM roles, enabled encryption, and set up versioning where the backend supports it. Never leave buckets, blobs, or containers public or unencrypted.

State File Segmentation and Team Collaboration

As your infrastructure grows, a single monolithic state file becomes a bottleneck and a risk. Best practice is to segment state both by environment (dev, staging, prod) and by component (network, database, app). This limits blast radius and improves performance and collaboration.

Example organization:


terraform-state-bucket/
  networking/
    dev/terraform.tfstate
    staging/terraform.tfstate
    prod/terraform.tfstate
  database/
    dev/terraform.tfstate
    staging/terraform.tfstate
    prod/terraform.tfstate
  application/
    dev/terraform.tfstate
    staging/terraform.tfstate
    prod/terraform.tfstate

This structure allows teams to work independently on different stacks. For example, networking changes won’t lock the application state and vice versa. For DR and compliance, each state file should be versioned and backed up independently.

Referencing Outputs Across State Files

You can safely reference outputs from another state file using the terraform_remote_state data source:


data "terraform_remote_state" "networking" {
  backend = "s3"
  config = {
    bucket = "sesamedisk-tfstate"
    key    = "networking/prod/terraform.tfstate"
    region = "eu-west-1"
  }
}
# Use an output from the networking state
resource "aws_instance" "web" {
  ami           = "ami-0abcdef1234567890"
  instance_type = "t3.medium"
  subnet_id     = data.terraform_remote_state.networking.outputs.private_subnet_id
}

Avoid chaining too many dependencies; keep references between state files explicit and minimal.

Manual State Operations and Drift Management

Sometimes you must operate directly on state—when refactoring, importing, or repairing after partial failures. Use these commands with caution, always after a state file backup:

terraform state list — List all tracked resources
terraform state show <resource> — Show details for a resource
terraform state mv <old> <new> — Move resource addresses (refactoring)
terraform state rm <resource> — Remove resource from state (without destroying it)
terraform import <resource> <id> — Import existing resources into state
terraform state pull > backup.json — Download remote state for inspection/backup
terraform force-unlock <LOCK_ID> — Release a stuck lock (only if you’re certain no other operation is running)

For drift management, regularly run terraform plan and use tools like AWS Config, GCP Asset Inventory, or Azure Resource Graph to detect real-world changes outside of Terraform.

Security Hardening for Terraform State

Never treat state files as harmless data. They often contain sensitive information (tokens, passwords, resource ARNs). Production hardening includes:

Enforce encryption at rest (KMS, GCP CMEK, Azure Key Vault-backed keys)
Restrict IAM permissions to least privilege (separate read/write roles; no anonymous access)
Enable versioning and configure lifecycle policies to prevent accidental destruction
Block public access at the backend level (S3 public access blocks, GCS uniform bucket-level access, Azure Blob public access disabled)
Never commit state files to Git or any VCS
Review sensitive flags on variables and outputs so that secrets don’t leak to logs or plans

Example of marking a variable and output as sensitive:


variable "database_password" {
  description = "Database password"
  type        = string
  sensitive   = true
}

output "db_connection_string" {
  description = "Database connection string"
  value       = "postgresql://admin:${var.database_password}@db.example.com/app"
  sensitive   = true
}

Backend Comparison Table

Backend	State Locking	Encryption	Versioning	Suitable for Production?	Notes
Local	No	No (OS-dependent)	No	No	For testing only; no collaboration
S3 + DynamoDB	Yes (via DynamoDB)	Yes (KMS, TLS)	Yes (S3 Versioning)	Yes	Industry standard; supports large teams
GCS	Yes (built-in)	Yes (CMEK, TLS)	Yes	Yes	Simple setup for GCP shops
Azure Blob	Yes (built-in)	Yes (Key Vault, TLS)	Yes	Yes	Use for Azure-native teams
Terraform Cloud/Enterprise	Yes (managed)	Yes	Yes	Yes	Managed SaaS; SOC2 compliant

Troubleshooting Common Terraform State Errors

Error: state lock already held
Cause: Another apply or plan is running, or a previous run crashed.
Fix: Wait for the process to complete. If the lock is stuck, use terraform force-unlock <LOCK_ID> (only if you’re sure no other process is running).
Error: Backend configuration changed
Cause: Backend config in terraform block was modified.
Fix: Run terraform init and follow prompts to migrate state. Always backup your state file first.
Error: Access denied when writing state
Cause: Insufficient IAM permissions or backend object lock.
Fix: Review IAM policies. Ensure the user/principal has read/write and lock permissions (for S3, s3:PutObject, s3:GetObject, dynamodb:PutItem, etc.).
State file corruption or loss
Fix: Recover using backend versioning (S3/GCS/Azure Blob), or, in worst cases, reconstruct from infrastructure using terraform import.
State drift
Cause: Out-of-band changes in the cloud provider.
Fix: Run terraform plan regularly and remediate drift; consider integrating drift detection into your CI/CD pipelines.

Key Takeaways

Key Takeaways:

Always use a remote backend with locking and encryption for production. Never use local state for real infrastructure.

Segment state files by environment and component to minimize blast radius and improve collaboration.

Enable backend versioning and regularly back up your state. Practice recovery out-of-band before disaster strikes.

Harden IAM permissions and never commit state files to version control.

Be cautious with manual state operations and always backup before direct edits.

Monitor for drift and keep your state the single source of truth for infrastructure.