Ansible Vault Changes

Problem

Ansible Vault files in secret_vars are encrypted, making merge conflicts difficult to resolve as standard Git tools cannot diff or merge them automatically. This leads to manual decryption, diffing, and merging, which is error-prone and time-consuming. In a GitOps workflow, especially during the transition to IaC in Dutch government organizations, frequent vault changes across feature branches can cause conflicts, delaying promotions and complicating hotfixes.

Context

Vault changes often involve sensitive updates like passwords or keys, which may need immediate deployment to production without full OTAP progression. In Ansible inventory projects using environment branches, merging encrypted changes across branches risks conflicts. Encrypted files also hinder searchability for variable names, affecting audits and troubleshooting. A structured procedure is essential to minimize conflicts, ensure audit trails, and support GitOps practices in regulated environments like government data centers.

Solution

Introduce a dedicated procedure for handling vault changes to prevent merge conflicts by isolating them in short-lived branches merged directly to main. This approach simplifies promotions, enables hotfixes, and integrates with GitOps workflows.

Follow these steps:

1. Create a Dedicated Vault Branch:

   • From your feature branch, switch to main and pull the latest:

     git checkout main && git pull
     

   • Create a new branch for the vault change (e.g., vault/PROJECT-123):

     git checkout -b vault/PROJECT-123
     

2. Apply the Vault Change:

   • Edit the vault file:

     ansible-vault edit path/to/secret_vars/file.yml --vault-password-file ~/.vault_pass
     

   • Commit and push:

     git add path/to/secret_vars/file.yml
     git commit -m "Vault: PROJECT-123 — short reason for the change"
     git push -u origin vault/PROJECT-123
     

3. Open and Merge a Merge Request (MR):

   • Create an MR targeting main.
   • Assign reviewers (e.g., ops/security), document the change reason, and run CI pipelines.
   • Merge after approvals and tests.

4. Incorporate into Feature Branches:

   • Switch back to your feature branch and merge main:

     git checkout feat/... && git merge main
     

For production hotfixes, follow the same steps, merging directly to main.

Handling Merge Conflicts: If a conflict occurs (e.g., due to concurrent changes):

• Decrypt both versions manually:

  ansible-vault decrypt --output file-local.yml path/to/secret_vars/file.yml --vault-password-file ~/.vault_pass
  ansible-vault decrypt --output file-remote.yml <remote-branch> -- path/to/secret_vars/file.yml --vault-password-file ~/.vault_pass
  

• Diff and merge using tools like diff or meld.

• Re-encrypt the resolved file and commit.

• Document the resolution in the MR for auditing.

5. Documentation:

   • Update CHANGELOG.md in the feature branch.
   • Add placeholders in group_vars (e.g., my_secret: "see vault").
   • Record approvals and rollback steps in the MR.

Benefits

• Conflict Reduction: Isolates changes to avoid encrypted merges.
• Efficiency: Enables quick hotfixes without cherry-picking.
• Auditability: MRs provide traceable documentation.
• Searchability: Placeholders improve variable discovery.

Alternatives

Without a procedure, teams face manual conflict resolution for every merge, requiring decryption, manual diffs, and re-encryption. This is challenging in Dutch government transitions to GitOps and IaC, increasing errors and delays. While tools like Git filters could decrypt on-the-fly, they risk exposing secrets and complicate setups; the branch-based procedure is simpler and more secure.

Examples and Implementation

In an inventory project, a vault change for a new API key:

• Create vault/PROJECT-123, edit the file, commit, and merge to main via MR.
• Merge main into feature branches to incorporate.
• For conflicts: Decrypt, resolve manually, re-encrypt, and push.

Example Git workflow with:

gitGraph
   commit id: "main commit"
   branch feature/999-new-feature
   checkout feature/999-new-feature
   commit id: "feature work"
   checkout main
   branch vault/PROJECT-123
   checkout vault/PROJECT-123
   commit id: "Vault change"
   checkout main
   merge vault/PROJECT-123 id: "Merge to main"
   checkout feature/999-new-feature
   merge main id: "Incorporate into feature"
   commit id: "Continue feature"

Additional Information

• Ansible Vault: Ansible Vault offers a secure method to encrypt sensitive data within Ansible files.
• Managing Secrets with Ansible Vault in AAP / AWX: Guidance on managing secrets using Ansible Vault in Ansible projects, with a focus on Ansible Automation Platform (AAP) and AWX.