Managing Secrets with Ansible Vault in AAP / AWX

Manage secrets using Ansible Vault in a custom folder secret_vars. And use a generic Ansible role to read secrets from this directory.

Problem

Ansible Automation Platform (AAP)/AWX does not have built-in support for Ansible Vault  , which poses challenges when integrating Vault-encrypted files in the group_vars directory. This limitation affects inventory projects that rely on Git as the foundation for AAP / AWX deployments, causing update failures due to the inability to configure an Ansible vault secret for such projects. It is crucial to address this issue and find a workaround.

Context

Managing secrets is a critical task in Ansible projects, and Ansible Vault offers a standard and straightforward solution. However, using Ansible Vault effectively within the context of AAP / AWX requires specific setup and considerations.

In Ansible and automation projects, particularly in environments like the Dutch government, a dedicated secrets management tool is not always immediately available. In such cases, Ansible Vault is a simple and practical choice. It is not a point of no return, as Ansible Vault is built into Ansible (with no additional costs or licenses) and is straightforward enough to replace if a more advanced secrets management solution becomes available later.

Solution

Using a Custom Secrets Directory

The core of the solution is to avoid storing secrets in the standard group_vars directory. This approach only works reliably with the Ansible CLI and not with AAP / AWX, because you cannot provide a Vault password to the inventory project in AAP / AWX—and this limitation is intentional. As a result, syncing an inventory project that includes a group_vars directory with Vault-encrypted files will fail due to the missing Vault password. It is not possible to configure a Vault password directly on an inventory project, which helps prevent accidental exposure of secrets.

Instead, create a separate folder to store these secrets. In the C2 Platform, we use a convention of naming this folder secret_vars. This name is intuitive and clear, making it easy for teams to understand its purpose. By moving Vault-encrypted files to secret_vars, you bypass the limitations of AAP / AWX inventory projects while still leveraging Ansible Vault for encryption. This folder can then be loaded dynamically using Ansible tasks, such as include_vars, in a custom role. This ensures compatibility across CLI and AAP / AWX environments without compromising security.

Ansible Secrets Role

Develop a generic and flexible Ansible “Secrets” role capable of utilizing the secret_vars folder. This role should be compatible with both AAP and the Ansible CLI. Create a custom folder named secret_vars to store secrets that can be included using include_vars. This folder will serve as an alternative for storing secrets to the default group_vars directory.

Start in Development

In addition to the above solution, it is recommended to also use Ansible Vault during development, as managing secrets is integral to inventory projects.

For example, in the C2 Platform approach, the primary development environment is open source, and passwords are kept simple for convenience—typically always secret, or Supersecret! if the tool requires stronger passwords. Since all passwords are open and do not need to be secure (as everything is open source and open data), a vault is still used during the development process. This is because development typically identifies which passwords are needed. By including them immediately in the vault, it becomes clear for other environments which passwords are required.

Even if the development environment uses easy-to-remember passwords, as is the practice within the C2 Platform’s development environment, Ansible Vault is still used.

Provide a Procedure for Updating the Vault

The key point here is to ensure that the whole team is aware of the fact that the vault is an encrypted file and that Git cannot merge encrypted files. So, without a procedure that the whole team is aware of, this will cause merge conflicts that will be very hard and time-consuming to resolve. Preventing merge conflicts is key, and the procedure below will help ensure that.

To prevent merge conflicts when changing secrets in the vault, introduce a recommended branching procedure for the project similar to:

1. Create a temporary branch named vault from the latest commit of the master branch. This ensures you’re working on the most up-to-date version without affecting ongoing work.
2. Add or update the secrets in the secret_vars folder on this vault branch. Use Ansible Vault to encrypt or re-encrypt as needed.
3. Immediately merge the vault branch back into master using a merge commit. This integrates the changes quickly and minimizes the window for conflicts.
4. If the secrets are intended for a specific environment (e.g., test), sync the updated master branch to the environment branch (e.g., test) via a merge or rebase, depending on your Git workflow.

This process isolates secret updates, reducing the risk of conflicts from parallel changes in other branches. Always ensure that only authorized users handle vault operations and that vault passwords are securely managed.

Resolving Merge Conflicts in the Vault

In cases where merge conflicts occur in the vault—often due to accidental changes made outside the recommended procedure (e.g., someone modifies the vault on a feature branch, violating the process)—resolving them requires a manual, local approach. This cannot be done via the GitLab web interface. Instead, follow this procedure to resolve the conflict locally:

1. Clone the branch with the conflict locally.
2. To resolve, decrypt and compare the vault contents manually:
   • On your branch, decrypt the vault file and save its contents to a temporary local file (e.g., temp_mybranch.yml).
   • Switch to the master branch, decrypt the vault file, and save its contents to another temporary file (e.g., temp_master.yml).
   • Use a diff tool to compare temp_mybranch.yml and temp_master.yml. Manually create a resolved version of the vault contents based on the diff.
3. Now fetch and pull the latest changes from master into your local branch. This will trigger the merge conflict locally.
4. Encrypt the resolved contents back into the vault file on your branch.
5. Commit the changes and push the branch.
6. Now, the merge request or pull to master should succeed without conflicts.

This method ensures conflicts are handled securely without exposing secrets. Emphasize to the team the importance of following the update procedure to avoid these situations.

Examples and implementation

Refer to the Ansible “Secrets” role c2platform.core.secrets  within the c2platform.core collection for an implementation example. This vault is included via include_vars in this role:

 roles/secrets/tasks/main.yml

20    - name: Include secrets
21      ansible.builtin.include_vars:
22        dir: "{{ secrets_dir_item['secrets_dir_item'] }}"
23      loop: >-
24        {{ secrets_dirs_stats['results']
25        | selectattr('stat.exists', 'equalto', True) }}        
26      loop_control:
27        label: "{{ secrets_dir_item['secrets_dir_item'] }}"
28        loop_var: secrets_dir_item
29      when: secrets_dir_item.stat.exists

The role utilizes the secrets_dirs list, which can be configured with multiple locations for the secret_vars folder. The following example works for both the Ansible CLI and AAP / AWX. When using AAP, AAP will place the vault in the specific location for example /runner/project/secret_vars/development. When testing Ansible playbooks on a Ansible development desktop, the vault will be found by the secrets role secret_vars/development.

 group_vars/all/secrets.yml

1---
2secrets_dirs:
3  - "{{ inventory_dir }}/secret_vars/{{ px_env }}"
4  - "/runner/project/secret_vars/{{ px_env }}"  #  awx / aap

Additional Information

• Ansible Vault: Securely manage secrets using Ansible Vault.
• Securely Accessing Ansible Vault in Development: Guideline for securely accessing Ansible Vault during development without storing passwords in plain files, using environment variables and scripts.
• Protecting sensitive data with Ansible vault — Ansible Community Documentation