Turn Terraform drift into a GitHub issue

The infrastructure series began by removing permanent cloud keys from GitHub Actions for AWS and Google Cloud, then used Terraform and GitOps to create an EKS cluster. The next operational problem is inevitable: reality drifts away from code.

A console edit, an API call, or another automation can change a managed resource. Terraform cannot prevent every out-of-band mutation. It can make the difference visible before the next apply turns a surprise into an incident.

This workflow runs terraform plan -detailed-exitcode and opens a GitHub issue when a difference exists.

Start with short-lived AWS credentials #

The workflow assumes the OIDC trust described earlier in this series. GitHub presents its signed identity token and assumes a narrowly scoped AWS role; there is no access key in repository secrets.

Restrict the role’s sub claim by repository and, for production, branch:

locals {
  sub_conditions = length(var.allowed_branches) > 0 ? flatten([
    for repo in var.github_repositories : [
      for branch in var.allowed_branches : "repo:${repo}:ref:${branch}"
    ]
  ]) : [for repo in var.github_repositories : "repo:${repo}:*"]
}

Different environments can supply different repositories, policies, and branches through one env_config map. Production may accept only refs/heads/main; development can remain less restrictive.

Exit code 2 is information, not failure #

With -detailed-exitcode, Terraform returns:

GitHub Actions normally treats every non-zero code as failure. Capture the result before restoring strict shell behavior, and disable the setup action’s wrapper so Terraform’s real exit code survives:

- uses: hashicorp/setup-terraform@v3
  with:
    terraform_version: '1.14.4'
    terraform_wrapper: false
 
- name: Terraform plan
  id: plan
  working-directory: terraform/aws
  run: |
    set +e
    terraform plan -var env=${{ inputs.environment }} \
      -no-color -detailed-exitcode -out=tfplan.binary > plan.txt 2>&1
    exit_code=$?
    set -e
    cat plan.txt
    echo "exitcode=${exit_code}" >> "$GITHUB_OUTPUT"
    if [ "$exit_code" -eq 1 ]; then exit 1; fi

The issue step runs only when exitcode == 2. It uses terraform show tfplan.binary so the ticket contains the readable plan rather than sending an operator spelunking through workflow logs.

Make repeated checks idempotent #

The workflow reads the state serial and includes it in the title:

[production] Terraform Drift Detected (tf-state-serial@42)

Before creating a ticket, gh issue list searches open issues carrying the terraform-drift label for that title. Repeated scheduled runs therefore leave one issue for the same state. After an apply updates the state serial, a later difference receives a new issue.

This is a useful first stage, but it deliberately does not decide whether a plan is unintended. A pull request may fully explain the difference. A later stage could compare the plan with the commit or ask a model to summarize the mismatch, but a human should still own the decision to apply.

Run drift checks on a schedule or manually. Push-triggered checks can create noisy races while infrastructure changes are already in flight. Detection is simple; keeping alerts actionable is the engineering work.