After building and manually deploying infrastructure, I faced the real DevOps question:<\/p>\n\n\n\n
“How do I deploy infrastructure changes safely across multiple environments without manual terraform apply commands?”<\/p>\n\n\n\n
This is where Infrastructure as Code meets DevOps automated, repeatable, multi-environment deployments with AWS CodePipeline.<\/p>\n\n\n\n
A complete CI\/CD pipeline that automatically deploys Terraform infrastructure to Dev and Staging environments with:<\/p>\n\n\n\n
Developer \u2192 Git Push \u2192 GitHub\n \u2193\n AWS CodePipeline (triggered)\n \u2193\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502\n Dev Branch Staging Branch\n \u2502 \u2502\n \u2193 \u2193\n CodeBuild (Dev) CodeBuild (Staging)\n \u2502 \u2502\n buildspec-dev.yml buildspec-stag.yml\n \u2502 \u2502\n terraform init terraform init\n -backend-config=dev.conf -backend-config=stag.conf\n \u2502 \u2502\n terraform apply terraform apply\n -var-file=dev.tfvars -var-file=stag.tfvars\n \u2502 \u2502\n \u2193 \u2193\n S3 State (Dev) S3 State (Staging)\n DynamoDB Lock (Dev) DynamoDB Lock (Staging)\n \u2502 \u2502\n \u2193 \u2193\n AWS Infrastructure (Dev) AWS Infrastructure (Staging)\n devdemo5.rezaops.com stagedemo5.rezaops.com<\/code><\/pre>\n\n\nMulti-Environment Configuration Strategy<\/h3>\n\n\n\nEnvironment Separation with Backend Configs<\/h3>\n\n\n\n
This is the key pattern – separate backend configs per environment:<\/p>\n\n\n\n
dev.conf:<\/p>\n\n\n\n
bucket = \"terraform-on-aws-for-ec2-lowyi\"\nkey = \"iacdevops\/dev\/terraform.tfstate\"\nregion = \"us-east-1\" \ndynamodb_table = \"iacdevops-dev-tfstate\"<\/code><\/pre>\n\n\n\nstag.conf:<\/p>\n\n\n\n
bucket = \"terraform-on-aws-for-ec2-lowyi\"\nkey = \"iacdevops\/stag\/terraform.tfstate\"\nregion = \"us-east-1\" \ndynamodb_table = \"iacdevops-stag-tfstate\"<\/code><\/pre>\n\n\n\nWhy separate state files?<\/p>\n\n\n\n
Without separation:<\/p>\n\n\n\n
Single state file: terraform.tfstate\nDev changes \u2192 Applies to production \u2717\nStaging changes \u2192 Overwrites dev state \u2717\nNo rollback per environment \u2717<\/code><\/pre>\n\n\n\nWith separation:<\/p>\n\n\n\n
Dev state: iacdevops\/dev\/terraform.tfstate\nStaging state: iacdevops\/stag\/terraform.tfstate\nProduction state: iacdevops\/prod\/terraform.tfstate\n\nDev changes \u2192 Only affects dev \u2713\nStaging is independent \u2713\nCan rollback each environment separately \u2713<\/code><\/pre>\n\n\n\nDynamoDB for state locking:<\/p>\n\n\n\n
dynamodb_table = \"iacdevops-dev-tfstate\"<\/code><\/pre>\n\n\n\nWhat happens without locking:<\/p>\n\n\n\n
User A runs: terraform apply\nUser B runs: terraform apply (simultaneously)\n\nBoth read same state file\nBoth make conflicting changes\nState file corrupted \u2717<\/code><\/pre>\n\n\n\nWith DynamoDB locking:<\/p>\n\n\n\n
User A runs: terraform apply\n \u2192 Acquires lock in DynamoDB\n \nUser B runs: terraform apply\n \u2192 Waits for lock: \"State locked by user A\"\n \nUser A completes \u2192 Releases lock\nUser B acquires lock \u2192 Applies safely<\/code><\/pre>\n\n\n\nDynamoDB table structure:<\/p>\n\n\n\n
Table: iacdevops-dev-tfstate\nPrimary Key: LockID\n\nItem when locked:\n{\n \"LockID\": \"terraform-on-aws-for-ec2-lowyi\/iacdevops\/dev\/terraform.tfstate-md5\",\n \"Info\": \"user@hostname running terraform apply\",\n \"Digest\": \"abc123...\",\n \"Created\": \"2025-01-15T10:30:00Z\"\n}<\/code><\/pre>\n\n\nEnvironment-Specific Variables<\/h3>\n\n\n\n
dev.tfvars:<\/p>\n\n\n\n
environment = \"dev\"\n\nvpc_availability_zones = [\"us-east-1a\", \"us-east-1b\", \"us-east-1c\"]\nvpc_public_subnets = [\"10.0.101.0\/24\", \"10.0.102.0\/24\", \"10.0.103.0\/24\"]\n\ninstance_type = \"t2.micro\"\ndns_name = \"devdemo5.rezaops.com\"<\/code><\/pre>\n\n\n\nstag.tfvars:<\/p>\n\n\n\n
environment = \"stag\"\n\nvpc_availability_zones = [\"us-east-1a\", \"us-east-1b\", \"us-east-1c\"]\nvpc_public_subnets = [\"10.0.101.0\/24\", \"10.0.102.0\/24\", \"10.0.103.0\/24\"]\n\ninstance_type = \"t2.micro\" # Could be t3.small for staging\ndns_name = \"stagedemo5.rezaops.com\"<\/code><\/pre>\n\n\n\nHow environment variable affects everything:<\/p>\n\n\n\n
# Local values (c3-local-values.tf)\nlocals {\n environment = var.environment\n name = \"${var.business_divsion}-${var.environment}\"\n # name = \"hr-dev\" or \"hr-stag\"\n}<\/code><\/pre>\n\n\n\nThis prefix propagates everywhere:<\/p>\n\n\n\n
Resource names:<\/p>\n\n\n\n
# VPC\nname = \"${local.name}-${var.vpc_name}\"\n# Result: \"hr-dev-myvpc\" or \"hr-stag-myvpc\"\n\n# ALB\nname = \"${local.name}-alb\"\n# Result: \"hr-dev-alb\" or \"hr-stag-alb\"\n\n# Security Groups\nname = \"${local.name}-public-bastion-sg\"\n# Result: \"hr-dev-public-bastion-sg\"\n\n# Launch Template\nname = \"${local.name}\"\n# Result: \"hr-dev\" or \"hr-stag\"<\/code><\/pre>\n\n\n\nBenefits:<\/p>\n\n\n\n
Easy identification:<\/p>\n\n\n\n
aws ec2 describe-vpcs --filters \"Name=tag:environment,Values=dev\"\n# Shows only dev VPCs<\/code><\/pre>\n\n\n\nNo naming conflicts:<\/p>\n\n\n\n
Dev VPC: hr-dev-myvpc\nStaging VPC: hr-stag-myvpc\nBoth exist simultaneously \u2713<\/code><\/pre>\n\n\n\nResource tagging:<\/p>\n\n\n\n
common_tags = {\n owners = \"hr\"\n environment = \"dev\" # or \"stag\"\n}<\/code><\/pre>\n\n\n\nAWS CodeBuild Buildspec Files<\/h3>\n\n\n\nDev Environment Buildspec<\/h3>\n\n\n\n
buildspec-dev.yml:<\/p>\n\n\n\n
version: 0.2\n\nenv:\n variables:\n TERRAFORM_VERSION: \"1.9.0\"\n TF_COMMAND: \"apply\"\n #TF_COMMAND: \"destroy\"\n parameter-store:\n AWS_ACCESS_KEY_ID: \"\/CodeBuild\/MY_AWS_ACCESS_KEY_ID\"\n AWS_SECRET_ACCESS_KEY: \"\/CodeBuild\/MY_AWS_SECRET_ACCESS_KEY\"\n\nphases:\n install:\n runtime-versions:\n python: 3.11\n on-failure: ABORT\n commands:\n - tf_version=$TERRAFORM_VERSION\n - wget https:\/\/releases.hashicorp.com\/terraform\/\"$TERRAFORM_VERSION\"\/terraform_\"$TERRAFORM_VERSION\"_linux_amd64.zip\n - unzip terraform_\"$TERRAFORM_VERSION\"_linux_amd64.zip\n - mv terraform \/usr\/local\/bin\/\n \n pre_build:\n on-failure: ABORT\n commands:\n - echo terraform execution started on `date`\n \n build:\n on-failure: ABORT\n commands:\n - cd \"$CODEBUILD_SRC_DIR\/terraform-manifests\"\n - ls -lrt \"$CODEBUILD_SRC_DIR\/terraform-manifests\"\n - terraform --version\n - terraform init -input=false --backend-config=dev.conf\n - terraform validate\n - terraform plan -lock=false -input=false -var-file=dev.tfvars\n - terraform $TF_COMMAND -input=false -var-file=dev.tfvars -auto-approve\n \n post_build:\n on-failure: CONTINUE\n commands:\n - echo terraform execution completed on `date`<\/code><\/pre>\n\n\nBuildspec Phases Explained<\/h3>\n\n\n\n
Phase 1: Install<\/p>\n\n\n\n
install:\n runtime-versions:\n python: 3.11\n commands:\n - wget https:\/\/releases.hashicorp.com\/terraform\/1.9.0\/terraform_1.9.0_linux_amd64.zip\n - unzip terraform_1.9.0_linux_amd64.zip\n - mv terraform \/usr\/local\/bin\/<\/code><\/pre>\n\n\n\nWhat happens:<\/p>\n\n\n\n
\n- CodeBuild starts Ubuntu container<\/li>\n\n\n\n
- Downloads Terraform binary from HashiCorp<\/li>\n\n\n\n
- Extracts and moves to PATH<\/li>\n\n\n\n
- Makes Terraform available globally<\/li>\n<\/ol>\n\n\n\n
Why not use pre-installed Terraform?<\/p>\n\n\n\n
CodeBuild may have:<\/p>\n\n\n\n
\n- Outdated Terraform version<\/li>\n\n\n\n
- No Terraform at all<\/li>\n\n\n\n
- Wrong version for your code<\/li>\n<\/ul>\n\n\n\n
Downloading specific version:<\/p>\n\n\n\n
\n- Guarantees consistent version<\/li>\n\n\n\n
- Works with your Terraform code<\/li>\n\n\n\n
- Prevents version mismatch issues<\/li>\n<\/ul>\n\n\n\n
Phase 2: Pre-build<\/p>\n\n\n\n
pre_build:\n commands:\n - echo terraform execution started on `date`<\/code><\/pre>\n\n\n\nSimple logging for CloudWatch. Could add:<\/p>\n\n\n\n
- aws sts get-caller-identity # Verify credentials\n- terraform --version # Log Terraform version<\/code><\/pre>\n\n\n\nPhase 3: Build (Main Phase)<\/p>\n\n\n\n
build:\n commands:\n - cd \"$CODEBUILD_SRC_DIR\/terraform-manifests\"\n - terraform init -input=false --backend-config=dev.conf\n - terraform validate\n - terraform plan -lock=false -input=false -var-file=dev.tfvars\n - terraform $TF_COMMAND -input=false -var-file=dev.tfvars -auto-approve<\/code><\/pre>\n\n\n\n$CODEBUILD_SRC_DIR explained:<\/p>\n\n\n\n
CODEBUILD_SRC_DIR=\/codebuild\/output\/src123456789\/src<\/code><\/pre>\n\n\n\nCodeBuild clones your git repo here. We need to navigate to terraform-manifests folder.<\/p>\n\n\n\n
terraform init -input=false –backend-config=dev.conf:<\/p>\n\n\n\n
# In c1-versions.tf:\nbackend \"s3\" {} # Empty! No hardcoded values<\/code><\/pre>\n\n\n\nBackend config provided at runtime:<\/p>\n\n\n\n
terraform init --backend-config=dev.conf<\/code><\/pre>\n\n\n\nThis pattern allows:<\/p>\n\n\n\n
\n- Same Terraform code for all environments<\/li>\n\n\n\n
- Backend configuration determined by buildspec<\/li>\n\n\n\n
- No environment-specific code branches<\/li>\n<\/ul>\n\n\n\n
-lock=false in terraform plan:<\/p>\n\n\n\n
terraform plan -lock=false<\/code><\/pre>\n\n\n\nplan doesn’t modify state, so locking is optional. Skipping lock:<\/p>\n\n\n\n
\n- Allows parallel plan operations<\/li>\n\n\n\n
- Prevents unnecessary DynamoDB writes<\/li>\n\n\n\n
- Speeds up pipeline<\/li>\n<\/ul>\n\n\n\n
Important: apply ALWAYS uses locking (can’t disable).<\/p>\n\n\n\n
-auto-approve:<\/p>\n\n\n\n
terraform apply -auto-approve<\/code><\/pre>\n\n\n\nWithout -auto-approve:<\/p>\n\n\n\n
Do you want to perform these actions?\n Terraform will perform the actions described above.\n Only 'yes' will be accepted to approve.\n\n Enter a value: _<\/code><\/pre>\n\n\n\nCodeBuild can’t interactively respond! Build would hang forever.<\/p>\n\n\n\n
Phase 4: Post-build<\/p>\n\n\n\n
post_build:\n on-failure: CONTINUE # Don't fail build if this fails\n commands:\n - echo terraform execution completed on `date`<\/code><\/pre>\n\n\n\non-failure: CONTINUE means:<\/p>\n\n\n\n
\n- If post_build fails, mark build as succeeded anyway<\/li>\n\n\n\n
- Useful for non-critical cleanup or notifications<\/li>\n\n\n\n
- Logging should never fail the build<\/li>\n<\/ul>\n\n\n
Environment Variables and Secrets<\/h3>\n\n\n\n
Variables in buildspec:<\/p>\n\n\n\n
env:\n variables:\n TERRAFORM_VERSION: \"1.9.0\"\n TF_COMMAND: \"apply\"<\/code><\/pre>\n\n\n\nThese are plain text, visible in:<\/p>\n\n\n\n
\n- Buildspec file<\/li>\n\n\n\n
- CodeBuild logs<\/li>\n\n\n\n
- Build history<\/li>\n<\/ul>\n\n\n\n
Secrets from Parameter Store:<\/p>\n\n\n\n
env:\n parameter-store:\n AWS_ACCESS_KEY_ID: \"\/CodeBuild\/MY_AWS_ACCESS_KEY_ID\"\n AWS_SECRET_ACCESS_KEY: \"\/CodeBuild\/MY_AWS_SECRET_ACCESS_KEY\"<\/code><\/pre>\n\n\n\nHow Parameter Store secrets work:<\/p>\n\n\n\n
Store secrets:<\/p>\n\n\n\n
aws ssm put-parameter \\\n --name \"\/CodeBuild\/MY_AWS_ACCESS_KEY_ID\" \\\n --value \"AKIAIOSFODNN7EXAMPLE\" \\\n --type \"SecureString\"\n\naws ssm put-parameter \\\n --name \"\/CodeBuild\/MY_AWS_SECRET_ACCESS_KEY\" \\\n --value \"wJalrXUtnFEMI\/K7MDENG\/bPxRfiCYEXAMPLEKEY\" \\\n --type \"SecureString\"<\/code><\/pre>\n\n\n\nCodeBuild fetches at runtime:<\/p>\n\n\n\n
Build starts \u2192 CodeBuild IAM role \u2192 Parameter Store\n \u2192 Decrypt SecureString \u2192 Inject as environment variable<\/code><\/pre>\n\n\n\nAvailable in build:<\/p>\n\n\n\n
echo $AWS_ACCESS_KEY_ID\n# AKIAIOSFODNN7EXAMPLE\n\nterraform plan # Uses credentials automatically<\/code><\/pre>\n\n\n\nBenefits:<\/p>\n\n\n\n
\u2713 Credentials not in code \u2713 Not visible in logs (masked) \u2713 Rotatable without code changes \u2713 IAM-controlled access \u2713 Audit trail in CloudTrail<\/p>\n\n\n\n
Alternative: Secrets Manager<\/p>\n\n\n\n
env:\n secrets-manager:\n AWS_ACCESS_KEY_ID: \"my-secret:access_key_id\"\n AWS_SECRET_ACCESS_KEY: \"my-secret:secret_access_key\"<\/code><\/pre>\n\n\n\nParameter Store vs Secrets Manager:<\/p>\n\n\n\n
Parameter Store:\n- Free (up to 10,000 parameters)\n- Simple key-value\n- Good for config values\n\nSecrets Manager:\n- $0.40 per secret per month\n- Automatic rotation support\n- Better for database passwords, API keys<\/code><\/pre>\n\n\n\nTerraform Backend Configuration<\/h3>\n\n\n\nBackend Declaration<\/h3>\n\n\n\n
c1-versions.tf:<\/p>\n\n\n\n
terraform {\n required_providers {\n aws = {\n source = \"hashicorp\/aws\"\n version = \"5.70.0\"\n }\n }\n \n backend \"s3\" {} # Empty backend block!\n}<\/code><\/pre>\n\n\n\nWhy empty backend?<\/p>\n\n\n\n
Traditional approach (hardcoded):<\/p>\n\n\n\n
backend \"s3\" {\n bucket = \"my-terraform-state\"\n key = \"dev\/terraform.tfstate\"\n region = \"us-east-1\"\n dynamodb_table = \"terraform-state-lock\"\n}<\/code><\/pre>\n\n\n\nProblems:<\/p>\n\n\n\n
\n- Dev and staging need different configurations<\/li>\n\n\n\n
- Must maintain separate code branches<\/li>\n\n\n\n
- Can’t use same code for multiple environments<\/li>\n<\/ul>\n\n\n\n
Dynamic backend configuration:<\/p>\n\n\n\n
# Dev environment\nterraform init -backend-config=dev.conf\n\n# Staging environment\nterraform init -backend-config=stag.conf -reconfigure<\/code><\/pre>\n\n\n\n-reconfigure flag:<\/p>\n\n\n\n
terraform init -backend-config=stag.conf -reconfigure<\/code><\/pre>\n\n\n\nWhat -reconfigure does:<\/p>\n\n\n\n
Without -reconfigure:\n Backend already configured for dev\n Error: Backend configuration changed\n \nWith -reconfigure:\n Detected configuration change\n Migrating state from dev backend to staging backend\n Asking: Do you want to copy state? (must type 'yes')\n \nIn CodeBuild with -reconfigure:\n Skips migration questions\n Configures new backend\n No interactive prompts \u2713<\/code><\/pre>\n\n\n\nState File Organization in S3<\/h3>\n\n\n\ns3:\/\/terraform-on-aws-for-ec2-lowyi\/\n\u251c\u2500\u2500 iacdevops\/\n\u2502 \u251c\u2500\u2500 dev\/\n\u2502 \u2502 \u2514\u2500\u2500 terraform.tfstate # Dev environment state\n\u2502 \u2502 \u2514\u2500\u2500 terraform.tfstate.backup\n\u2502 \u251c\u2500\u2500 stag\/\n\u2502 \u2502 \u2514\u2500\u2500 terraform.tfstate # Staging environment state\n\u2502 \u2502 \u2514\u2500\u2500 terraform.tfstate.backup\n\u2502 \u2514\u2500\u2500 prod\/\n\u2502 \u2514\u2500\u2500 terraform.tfstate # Production environment state (future)\n\u2502 \u2514\u2500\u2500 terraform.tfstate.backup<\/code><\/pre>\n\n\n\nState file versions:<\/p>\n\n\n\n
S3 versioning MUST be enabled:<\/p>\n\n\n\n
aws s3api put-bucket-versioning \\\n --bucket terraform-on-aws-for-ec2-lowyi \\\n --versioning-configuration Status=Enabled<\/code><\/pre>\n\n\n\nWhy versioning matters:<\/p>\n\n\n\n
Scenario: Accidental terraform destroy<\/p>\n\n\n\n
Without versioning:<\/p>\n\n\n\n
terraform destroy -auto-approve\n State file deleted\n No record of infrastructure \u2717\n Must rebuild from scratch \u2717<\/code><\/pre>\n\n\n\nWith versioning:<\/p>\n\n\n\n
terraform destroy -auto-approve\n State file deleted \u2192 S3 creates delete marker\n Previous versions preserved\n Can restore previous state:\n aws s3api list-object-versions --bucket ... --prefix ...\n aws s3api get-object --version-id ... \n Infrastructure recoverable \u2713<\/code><\/pre>\n\n\nAWS CodePipeline Setup<\/h3>\n\n\n\nPipeline Stages<\/h3>\n\n\n\nStage 1: Source\n \u2193\n GitHub repository (on push)\n \u2193\nStage 2: Build (Dev)\n \u2193\n CodeBuild project (Dev)\n \u2193\n Buildspec: buildspec-dev.yml\n \u2193\nStage 3: Build (Staging)\n \u2193\n CodeBuild project (Staging)\n \u2193\n Buildspec: buildspec-stag.yml<\/code><\/pre>\n\n\n\nPipeline Triggers<\/h3>\n\n\n\n
Automatic trigger on git push:<\/p>\n\n\n\n
# Developer workflow\ngit add terraform-manifests\/c10-02-ALB-application-loadbalancer.tf\ngit commit -m \"Updated ALB listener rules\"\ngit push origin main\n\n# Pipeline triggered automatically:\n 1. Source stage: Downloads latest code\n 2. Dev build: Deploys to dev environment\n 3. Staging build: Deploys to staging environment<\/code><\/pre>\n\n\n\nManual approval (optional):<\/p>\n\n\n\n
For production:<\/p>\n\n\n\n
Stage 1: Source\n \u2193\nStage 2: Build (Dev) - Automatic\n \u2193\nStage 3: Manual Approval\n \u2193\nStage 4: Build (Staging) - After approval\n \u2193\nStage 5: Manual Approval\n \u2193\nStage 6: Build (Production) - After approval<\/code><\/pre>\n\n\n\nCodeBuild IAM Permissions<\/h3>\n\n\n\n
Required IAM policy for CodeBuild:<\/p>\n\n\n\n
{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ec2:*\",\n \"elasticloadbalancing:*\",\n \"autoscaling:*\",\n \"route53:*\",\n \"acm:*\",\n \"sns:*\",\n \"s3:*\"\n ],\n \"Resource\": \"*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"s3:GetObject\",\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::terraform-on-aws-for-ec2-lowyi\/*\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"dynamodb:GetItem\",\n \"dynamodb:PutItem\",\n \"dynamodb:DeleteItem\"\n ],\n \"Resource\": \"arn:aws:dynamodb:*:*:table\/iacdevops-*-tfstate\"\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ssm:GetParameter\",\n \"ssm:GetParameters\"\n ],\n \"Resource\": \"arn:aws:ssm:*:*:parameter\/CodeBuild\/*\"\n }\n ]\n}<\/code><\/pre>\n\n\n\nWhy these permissions?<\/p>\n\n\n\n
\n- ec2:* – Create VPCs, instances, security groups<\/li>\n\n\n\n
- elasticloadbalancing:* – Create ALBs, target groups<\/li>\n\n\n\n
- autoscaling:* – Create launch templates, ASGs<\/li>\n\n\n\n
- route53:* – Create DNS records<\/li>\n\n\n\n
- acm:* – Request and validate certificates<\/li>\n\n\n\n
- s3:* (state bucket) – Read\/write state files<\/li>\n\n\n\n
- dynamodb:* (lock table) – State locking<\/li>\n\n\n\n
- ssm:GetParameter – Fetch secrets<\/li>\n<\/ul>\n\n\n\n
Principle of Least Privilege:<\/p>\n\n\n\n
For production, restrict further:<\/p>\n\n\n\n
{\n \"Effect\": \"Allow\",\n \"Action\": \"ec2:*\",\n \"Resource\": \"*\",\n \"Condition\": {\n \"StringEquals\": {\n \"ec2:ResourceTag\/Environment\": \"dev\"\n }\n }\n}<\/code><\/pre>\n\n\n\nThis limits EC2 actions to resources tagged with Environment=dev.<\/p>\n\n\n\n
Deploying Infrastructure via Pipeline<\/h3>\n\n\n\nDeployment Workflow<\/h3>\n\n\n\n1. Developer makes change:\n \u251c\u2500 Modify: c10-02-ALB-application-loadbalancer.tf\n \u2514\u2500 Change: Add new listener rule\n\n2. Commit and push:\n git add .\n git commit -m \"Add \/api path routing\"\n git push origin main\n\n3. CodePipeline triggered automatically:\n \u251c\u2500 Source stage: Downloads code\n \u2514\u2500 Status: Source Complete\n\n4. Dev Build (CodeBuild):\n \u251c\u2500 Start time: 14:30:00\n \u251c\u2500 Install: Download Terraform \n \u251c\u2500 Pre-build: Log start time\n \u251c\u2500 Build:\n \u2502 \u251c\u2500 terraform init -backend-config=dev.conf\n \u2502 \u2502 Backend initialized: s3:\/\/...\/ iacdevops\/dev\/terraform.tfstate\n \u2502 \u251c\u2500 terraform validate\n \u2502 \u2502 Success! Configuration is valid.\n \u2502 \u251c\u2500 terraform plan -var-file=dev.tfvars\n \u2502 \u2502 Plan: 1 to add, 2 to change, 0 to destroy.\n \u2502 \u2514\u2500 terraform apply -var-file=dev.tfvars -auto-approve\n \u2502 Apply complete! Resources: 1 added, 2 changed, 0 destroyed.\n \u251c\u2500 Post-build: Log completion time\n \u251c\u2500 Duration: 3 minutes 45 seconds\n \u2514\u2500 Status: Build Complete\n\n5. Staging Build (CodeBuild):\n \u251c\u2500 Start time: 14:35:00\n \u251c\u2500 [Same steps as dev build]\n \u251c\u2500 backend-config: stag.conf\n \u251c\u2500 var-file: stag.tfvars\n \u251c\u2500 Duration: 3 minutes 50 seconds\n \u2514\u2500 Status: Build Complete\n\n6. Total Pipeline Duration: ~8 minutes\n\n7. Verification:\n Dev: https:\/\/devdemo5.rezaops.com\n Staging: https:\/\/stagedemo5.rezaops.com\n Both updated with new listener rule \u2713<\/code><\/pre>\n\n\n\nViewing Build Logs<\/h3>\n\n\n\n
CloudWatch Logs:<\/p>\n\n\n\n
Log Group: \/aws\/codebuild\/terraform-dev-build<\/p>\n\n\n\n
Log Stream: abc123-def456-789012<\/p>\n\n\n\n
[Container] 2025\/01\/15 14:30:15 Running command terraform init -backend-config=dev.conf\nInitializing the backend...\n\nSuccessfully configured the backend \"s3\"! Terraform will automatically\nuse this backend unless the backend configuration changes.\n\n[Container] 2025\/01\/15 14:30:45 Running command terraform validate\nSuccess! The configuration is valid.\n\n[Container] 2025\/01\/15 14:31:00 Running command terraform plan -var-file=dev.tfvars\n...\nPlan: 1 to add, 2 to change, 0 to destroy.\n...\n\n[Container] 2025\/01\/15 14:31:30 Running command terraform apply -var-file=dev.tfvars -auto-approve\nmodule.alb.aws_lb_listener_rule.myapp1-rule: Modifying...\nmodule.alb.aws_lb_listener_rule.myapp1-rule: Modifications complete\nApply complete! Resources: 1 added, 2 changed, 0 destroyed.<\/code><\/pre>\n\n\n\nDestroying Infrastructure via Pipeline<\/h3>\n\n\n\n
To destroy environment:<\/p>\n\n\n\n
Update buildspec:<\/p>\n\n\n\n
env:\n variables:\n TF_COMMAND: \"destroy\" # Changed from \"apply\"<\/code><\/pre>\n\n\n\nCommit and push:<\/p>\n\n\n\n
git add buildspec-dev.yml\ngit commit -m \"Destroy dev environment\"\ngit push origin main<\/code><\/pre>\n\n\n\nPipeline executes:<\/p>\n\n\n\n
terraform destroy -input=false -var-file=dev.tfvars -auto-approve<\/code><\/pre>\n\n\n\nResult:<\/p>\n\n\n\n
Destroy complete! Resources: 45 destroyed.<\/code><\/pre>\n\n\n\nWarning for production:<\/p>\n\n\n\n
For production, never use auto-approve for destroy!<\/p>\n\n\n\n
Better approach:<\/p>\n\n\n\n
# Add manual approval before destroy\n- TF_COMMAND: \"destroy\"\n- Require manual confirmation in pipeline\n- Send SNS notification before destroy<\/code><\/pre>\n\n\nReal-World Scenarios<\/h3>\n\n\n\nScenario 1: Update ALB Listener Configuration<\/h3>\n\n\n\n
Change required: Add new health check path<\/p>\n\n\n\n
File: c10-02-ALB-application-loadbalancer.tf<\/p>\n\n\n\n
# Before:\nhealth_check = {\n path = \"\/app1\/index.html\"\n interval = 30\n}\n\n# After:\nhealth_check = {\n path = \"\/health\" # New health check endpoint\n interval = 15 # Faster checks\n}<\/code><\/pre>\n\n\n\nDeployment:<\/p>\n\n\n\n
git add c10-02-ALB-application-loadbalancer.tf\ngit commit -m \"Update ALB health check path and interval\"\ngit push origin main<\/code><\/pre>\n\n\n\nPipeline execution:<\/p>\n\n\n\n
14:00:00 - Source stage triggered\n14:00:30 - Dev build started\n14:01:15 - terraform plan output:\n module.alb.aws_lb_target_group.mytg1: will be updated in-place\n \n14:02:00 - terraform apply:\n module.alb.aws_lb_target_group.mytg1: Modifying...\n module.alb.aws_lb_target_group.mytg1: Modifications complete\n \n14:02:45 - Dev build complete\n14:03:00 - Staging build started\n14:05:45 - Staging build complete\n\nTotal time: 5 minutes 45 seconds<\/code><\/pre>\n\n\n\nVerification:<\/p>\n\n\n\n
# Check dev environment\naws elbv2 describe-target-groups \\\n --names hr-dev-mytg1 \\\n --query 'TargetGroups[0].HealthCheckPath'\n# Output: \"\/health\"\n\n# Check staging environment\naws elbv2 describe-target-groups \\\n --names hr-stag-mytg1 \\\n --query 'TargetGroups[0].HealthCheckPath'\n# Output: \"\/health\"<\/code><\/pre>\n\n\n\nScenario 2: Scale Up Auto Scaling Group<\/h3>\n\n\n\n
Change required: Increase ASG capacity for load testing<\/p>\n\n\n\n
File: dev.tfvars<\/p>\n\n\n\n
# Before:\ndesired_capacity = 2\nmax_size = 5\n\n# After:\ndesired_capacity = 5 # Scale up for load test\nmax_size = 10<\/code><\/pre>\n\n\n\nDeployment:<\/p>\n\n\n\n
git add terraform-manifests\/dev.tfvars\ngit commit -m \"Scale up dev ASG for load testing\"\ngit push origin main<\/code><\/pre>\n\n\n\nWhat happens:<\/p>\n\n\n\n
terraform plan output:\n # aws_autoscaling_group.my_asg will be updated in-place\n ~ resource \"aws_autoscaling_group\" \"my_asg\" {\n ~ desired_capacity = 2 -> 5\n ~ max_size = 5 -> 10\n }\n\nterraform apply:\n aws_autoscaling_group.my_asg: Modifying...\n aws_autoscaling_group.my_asg: Still modifying... [10s elapsed]\n \nASG launches 3 new instances:\n Instance 1: i-0abc123 launching...\n Instance 2: i-0def456 launching...\n Instance 3: i-0ghi789 launching...\n \nHealth checks:\n All instances healthy after 2 minutes\n \nLoad balancer:\n Target group now has 5 healthy targets<\/code><\/pre>\n\n\n\nImpact: Only dev environment scaled, staging unchanged!<\/p>\n\n\n\n
Scenario 3: Pipeline Failure – Terraform Validation Error<\/h3>\n\n\n\n
Bad change committed:<\/p>\n\n\n\n
# c13-06-autoscaling-ttsp.tf\nresource \"aws_autoscaling_policy\" \"avg_cpu_policy\" {\n name = \"cpu-policy\"\n target_value = \"invalid\" # Should be number, not string!\n}<\/code><\/pre>\n\n\n\nPipeline execution:<\/p>\n\n\n\n
Source stage: \u2713 Complete\nDev build: Started\n terraform validate:\n Error: Invalid value for target_value\n Expected number, got string\n \n Build status: FAILED\n Duration: 45 seconds<\/code><\/pre>\n\n\n\nStaging build: Never runs! (Dev failed)<\/p>\n\n\n\n
Result:<\/p>\n\n\n\n
\n- Dev environment unchanged (apply never ran)<\/li>\n\n\n\n
- Staging environment unchanged<\/li>\n\n\n\n
- Infrastructure protected from bad configuration<\/li>\n<\/ul>\n\n\n\n
Fix:<\/p>\n\n\n\n
target_value = 50.0 # Correct: number \ngit add c13-06-autoscaling-ttsp.tf\ngit commit -m \"Fix: target_value should be number\"\ngit push origin main<\/code><\/pre>\n\n\n\nPipeline re-runs successfully!<\/p>\n\n\n
Lessons Learned<\/h3>\n\n\n\n1. State Locking Failed – DynamoDB Table Missing<\/h3>\n\n\n\n
Error:<\/p>\n\n\n\n
Error acquiring state lock\nError: Error locking state: ResourceNotFoundException<\/code><\/pre>\n\n\n\nCause: DynamoDB table doesn’t exist!<\/p>\n\n\n\n
Solution:<\/p>\n\n\n\n
aws dynamodb create-table \\\n --table-name iacdevops-dev-tfstate \\\n --attribute-definitions AttributeName=LockID,AttributeType=S \\\n --key-schema AttributeName=LockID,KeyType=HASH \\\n --billing-mode PAY_PER_REQUEST<\/code><\/pre>\n\n\n\n2. CodeBuild Can’t Access Parameter Store<\/h3>\n\n\n\n
Error:<\/p>\n\n\n\n
Error retrieving value from parameter store\nAccessDeniedException: User is not authorized<\/code><\/pre>\n\n\n\nCause: CodeBuild IAM role missing SSM permissions.<\/p>\n\n\n\n
Solution: Add to CodeBuild service role:<\/p>\n\n\n\n
{\n \"Effect\": \"Allow\",\n \"Action\": \"ssm:GetParameter\",\n \"Resource\": \"arn:aws:ssm:*:*:parameter\/CodeBuild\/*\"\n}<\/code><\/pre>\n\n\n\n3. Terraform Version Mismatch<\/h3>\n\n\n\n
Error:<\/p>\n\n\n\n
Error: state file format version 4 is too new\nThis version of Terraform requires state file version 3<\/code><\/pre>\n\n\n\nCause: State created with Terraform 1.0, buildspec uses 1.9.0<\/p>\n\n\n\n
Solution: Update buildspec:<\/p>\n\n\n\n
TERRAFORM_VERSION: \"1.0.0\" # Match state file version<\/code><\/pre>\n\n\n\n4. Backend Initialization in Wrong Directory<\/h3>\n\n\n\n
Error:<\/p>\n\n\n\n
terraform init\nError: No configuration files<\/code><\/pre>\n\n\n\nCause:<\/p>\n\n\n\n
# Missing this:\ncd \"$CODEBUILD_SRC_DIR\/terraform-manifests\"<\/code><\/pre>\n\n\n\nTerraform ran in root directory, no .tf files found!<\/p>\n\n\n\n
5. ACM Certificate Validation Timeout in Pipeline<\/h3>\n\n\n\n
Problem: Pipeline takes 20+ minutes, times out waiting for ACM validation.<\/p>\n\n\n\n
Cause: ACM DNS validation requires Route53 records, takes 5-10 minutes.<\/p>\n\n\n\n
Solution 1: Increase CodeBuild timeout:<\/p>\n\n\n\n
# In CodePipeline build stage:\ntimeout_in_minutes: 30<\/code><\/pre>\n\n\n\nSolution 2: Pre-create and reuse ACM certificate:<\/p>\n\n\n\n
# Instead of creating new cert:\ndata \"aws_acm_certificate\" \"existing\" {\n domain = \"*.rezaops.com\"\n}\n\n# Use existing cert:\ncertificate_arn = data.aws_acm_certificate.existing.arn<\/code><\/pre>\n\n\n\n6. Auto-Approve Destroyed Wrong Environment<\/h3>\n\n\n\n
Horror story:<\/p>\n\n\n\n
Updated wrong buildspec:<\/p>\n\n\n\n
# In buildspec-PROD.yml (!!!)\nTF_COMMAND: \"destroy\"<\/code><\/pre>\n\n\n\nPushed to main \u2192 Pipeline destroyed production!<\/p>\n\n\n\n
Prevention:<\/p>\n\n\n\n
Never auto-approve destroy in production<\/p>\n\n\n\n
Use branch protection:<\/p>\n\n\n\n
main branch:\n - Requires pull request\n - Requires approvals: 2\n - No direct push<\/code><\/pre>\n\n\n\nAdd confirmation comment:<\/p>\n\n\n\n
TF_COMMAND: \"apply\"\n# To destroy, MUST manually edit this to \"destroy\" AND commit message MUST contain \"DESTROY-CONFIRMED\"<\/code><\/pre>\n\n\n\nSeparate pipelines:<\/p>\n\n\n\n
dev-pipeline: Auto-deploys on push to dev branch\nstag-pipeline: Auto-deploys on push to staging branch \nprod-pipeline: Manual approval required, no auto-destroy<\/code><\/pre>\n\n\n\n7. Concurrent Builds Caused State Lock Conflict<\/h3>\n\n\n\n
Problem: Pushed 2 commits quickly:<\/p>\n\n\n\n
Commit 1: Update ALB \u2192 Pipeline starts\nCommit 2: Update ASG \u2192 Pipeline starts immediately<\/code><\/pre>\n\n\n\nBoth tried to acquire state lock:<\/p>\n\n\n\n
Build 1: Acquired lock at 14:00:00\nBuild 2: Waiting for lock... timeout after 5 minutes<\/code><\/pre>\n\n\n\nSolution:<\/p>\n\n\n\n
# In CodePipeline:\nconcurrent_build_limit: 1<\/code><\/pre>\n\n\n\nOr better – queue builds:<\/p>\n\n\n\n
Build 1 running \u2192 Build 2 queued\nBuild 1 completes \u2192 Build 2 starts<\/code><\/pre>\n\n\n\nWhat I Learned<\/h3>\n\n\n\n\n- Separate state files are essential – Never share state between environments<\/li>\n\n\n\n
- DynamoDB locking prevents disasters – Concurrent applies can corrupt state<\/li>\n\n\n\n
- Backend configs enable multi-env – Same code, different backends<\/li>\n\n\n\n
- Parameter Store for secrets – Never hardcode credentials<\/li>\n\n\n\n
- Buildspec phases matter – Install, pre-build, build, post-build structure<\/li>\n\n\n\n
- Auto-approve is dangerous – Especially for destroy commands<\/li>\n\n\n\n
- Version consistency critical – Terraform version must match state file<\/li>\n\n\n\n
- CodeBuild IAM needs careful setup – Must have all necessary permissions<\/li>\n\n\n\n
- Pipeline feedback is fast – See failures in minutes, not hours<\/li>\n\n\n\n
- Infrastructure as Code is powerful – Click a button, deploy entire environment<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"
After building and manually deploying infrastructure, I faced the real DevOps question: “How do I deploy infrastructure changes safely across multiple environments…<\/p>\n","protected":false},"author":2,"featured_media":154,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"6","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/infrastructure-code-devops-automated-deployment-aws-reza-chegini-og5pc\/","footnotes":""},"categories":[85,10],"tags":[60,30,95,8,55,87],"series":[86],"class_list":["post-152","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aws","category-devops","tag-aws","tag-ci-cd","tag-codepipeline","tag-devops","tag-iac","tag-terraform","series-aws-with-terraform"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/infrastructure-code-devops-codepipeline-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/152","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=152"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/152\/revisions"}],"predecessor-version":[{"id":153,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/152\/revisions\/153"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/154"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=152"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=152"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=152"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=152"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}