Handling secrets in your Ansible playbooks

It’s finally happened. You’ve gone all-in with Ansible. You’ve read all the great articles, seen the use cases, and are excited to start building repeatable infrastructure and managing your configuration as code. There’s just one problem: You have a configuration file or a task that requires a password or other piece of mission-critical information. You know that you shouldn’t store the password in your plaintext files, so you’re not quite sure where it should go.

Fear not, this article guides you through the different options for handling sensitive information in your playbooks. Whether you’re looking for simple solutions, such as prompting an administrator to enter a password, or more complex options, such as integrating with an existing secrets management environment, Ansible has you covered.

[ You might also enjoy: Demystifying Ansible for Linux sysadmins ]

Prompts

If you’re just starting your Ansible journey and running all of your playbooks manually, then using an interactive prompt directly in your playbook is an easy solution. A prompt causes Ansible to ask the user for the desired variables and store them each time a playbook is run. Consider the following playbook, which ensures that an API key exists in a configuration file:

---

- hosts: all
  gather_facts: false
  vars_prompt:
    - name: api_key
      prompt: Enter the API key
  tasks:
    - name: Ensure API key is present in config file
      ansible.builtin.lineinfile:
        path: /etc/app/configuration.ini
        line: "API_KEY={{ api_key }}"

When I run this playbook, Ansible prompts me at the command line using the message in the prompt parameter:

# ansible-playbook -i inventory.ini main.yml
Enter the API key:

The input provided at the command line will be stored in the api_key variable, which can then be used in the play like any regular variable.

While variable prompts are easy to implement, you will outgrow them if you are invested in using Ansible for full configuration management. As your configuration management matures, you will begin running playbooks non-interactively, and there will be nobody in front of the terminal to enter passwords. That’s where Ansible Vault comes in.

Ansible Vault

One of my personal favorite Ansible capabilities is the Ansible Vault, which provides native content encryption capabilities. Ansible Vault can encrypt and decrypt arbitrary variables and files, which means you can use it to protect variable files that contain secrets or even encrypt entire sensitive configuration files. Ansible Vaults have many advanced features, but this article will focus on the basics.

Standard YAML files containing plaintext secrets can be easily encrypted with the ansible-vault encrypt command:

# Plaintext YAML file
$ cat secrets_file.enc
api_key: SuperSecretPassword

# Encrypt the file with ansible-vault
$ ansible-vault encrypt secrets_file.enc
New Vault password:
Confirm New Vault password:
Encryption successful

# Confirm that the file now contains encrypted content
$ cat secrets_file.enc
$ANSIBLE_VAULT;1.1;AES256
38396162626134393935663839666463306231653861336630613938303662633538633836656465
3637353766613339663032363538626430316135623665340a653961303730353962386134393162
62343936366265353935346336643865643833353737613962643539373230616239346133653464
6435353361373263640a376632613336366430663761363339333737386637383961363833303830
34336535623736313031313162353831666139343662653665366134633832646661

When I run my playbook, I can pass the encrypted variables file and tell Ansible to prompt me for the password. Ansible will decrypt the file and use the variables that I defined, just as if I had passed a regular variables file:

$ cat main.yml
---

- hosts: all
  gather_facts: false
  tasks:
    - name: Ensure API key is present in config file
      ansible.builtin.lineinfile:
        path: /etc/app/configuration.ini
        line: "API_KEY={{ api_key }}"


$ ansible-playbook -i inventory.ini -e @secrets_file.enc --ask-vault-pass main.yml
Vault password:

PLAY [all] ***********************************************************************************

TASK [Ensure API key is present in config file] **********************************************
changed: [localhost]

PLAY RECAP ***********************************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

$ cat /etc/app/configuration.ini
API_KEY=SuperSecretPassword

I previously described why using a vars_prompt isn’t ideal in an automated environment because it requires manual user intervention. So how is an Ansible Vault different? Ansible Vault allows you to specify a password file that contains the decryption password for the Vault:

$ cat password_file 
password

$ ansible-playbook -i inventory.ini -e @secrets_file.enc --vault-password-file password_file main.yml

PLAY [all] ***********************************************************************************

TASK [Ensure API key is present in config file] **********************************************
changed: [localhost]

PLAY RECAP ***********************************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0 

Be sure to set appropriate permissions on the decryption password file so that only the user running the playbook can access it. Alternatively, consider using a script to access the password at runtime from an external password storage system.

Now that my variable file is encrypted, I need a way to edit it. There are two ways to edit an encrypted Ansible Vault. You can either edit the file in-place, or you can fully decrypt it, edit it, and then re-encrypt it. Both methods are shown below.

# The edit command will launch a text editor, such as vim
$ ansible-vault edit secrets_file.enc 
Vault password: 

# The decrypt command will fully decrypt the file, allowing you to manipulate it how you see fit.
$ ansible-vault decrypt secrets_file.enc 
Vault password: 
Decryption successful

# Notice that the file has been decrypted
$ cat secrets_file.enc 
api_key: SuperSecretPassword

# Don't forget to re-encrypt the file when you're done!
$ ansible-vault encrypt secrets_file.enc 
New Vault password: 
Confirm New Vault password: 
Encryption successful
$ cat secrets_file.enc 
$ANSIBLE_VAULT;1.1;AES256
33373832393864613335393836616538373639353538306462366464303939303838316337336662
6235303936636465366363643761383462356335336239640a356161653166643134663762323136
34616431303434646338343265666135666263633162383662323164396266616638313936303863
3337626365313666630a326465663239653731613637303437666164346531636361653837326166
34396232623138616364393130303036653564643435636639316264636531336161

Using an Ansible Vault for secrets is one of my favorite methods of storing sensitive data. The benefit of this approach is that you can actually store your sensitive data in source control, side-by-side with your regular playbooks. Since these files are encrypted, there’s little risk in this approach as long as you pick a strong password. Like any shared secret, it’s a good idea to rotate the encryption password frequently. Ansible also offers several advanced features for Vaults, such as the ability to have different passwords for different Vaults. Be sure to review the documentation for great ways to secure your secrets using Ansible’s native capabilities.

Using an existing password manager

The previous two approaches are purely Ansible approaches to addressing secrets management. However, many organizations already have tools, such as HashiCorp Vault or Thycotic Secret Server. The Ansible community has written a number of custom modules for interacting with these types of systems.

The following playbook uses a lookup to obtain a secret from HashiCorp Vault and then use that secret in a task:

---

- hosts: all
  gather_facts: false
  tasks:
    - name: Ensure API key is present in config file
      ansible.builtin.lineinfile:
        path: /etc/app/configuration.ini
        line: "API_KEY={{ lookup('hashi_vault', 'secret=config-secrets/data/app/api-key:data token=s.FOmpGEHjzSdxGixLNi0AkdA7 url=http://localhost:8201')['key'] }}"

You can find a variety of plugins for different secret management tools on the Ansible Galaxy. As with any community-supported project, it’s a good idea to audit the code so that you understand how it is handling your data and secrets. You might even want to write your own.

The use of a lookup plugin or module is a good fit for organizations that already have a mature secrets management tool in place and simply want Ansible to consume secrets from this existing system. The obvious tradeoff is reduced simplicity: Playbook runs are now dependent on the availability of an external system, and relying on a community-supported module (or writing your own) can be time-consuming.

A note about logging

It’s important to remember that encrypting data at rest (e.g., in an Ansible Vault or external secrets system) doesn’t mean that data is protected from being accidentally output in an Ansible log file. If the module you call logs your secret during its normal operations or when an error occurs, that secret may be exposed in your log files. Whether you are storing these logs in a central system or simply using the default standard output view, it is important to protect your secrets from accidental exposure.

The output below is from the same Ansible playbook that I have been using for this tutorial. However, I’ve increased the Ansible debug level with -vvv. Notice that my secret (API_KEY=SuperSecretPassword) is directly exposed in the debug output. I’ve cleaned up this snippet a bit, so don’t worry if you try this, and your output looks slightly different.

TASK [Ensure API key is present in config file] ***********************************************************************************************************************************************
fatal: [localhost]: FAILED! => changed=false 
  ansible_facts:
    discovered_interpreter_python: /usr/bin/python
  invocation:
    module_args:
      attributes: null
      backrefs: false
      backup: false
      content: null
      create: false
      delimiter: null
      directory_mode: null
      firstmatch: false
      follow: false
      force: null
      group: null
      insertafter: null
      insertbefore: null
      line: API_KEY=SuperSecretPassword
      mode: null
      owner: null
      path: /etc/app/configuration.ini
      regexp: null
      remote_src: null
      selevel: null
      serole: null
      setype: null
      seuser: null
      src: null
      state: present
      unsafe_writes: null
      validate: null
  msg: Destination /etc/app/configuration.ini does not exist !
  rc: 257

This is definitely not ideal: My secret is right there, in plain view. Thankfully, Ansible provides a no_log parameter for tasks that protects sensitive data:

---

- hosts: all
  gather_facts: false
  tasks:
    - name: Ensure API key is present in config file
      ansible.builtin.lineinfile:
        path: /etc/app/configuration.ini
        line: "API_KEY={{ api_key }}"
      no_log: True

By adding this parameter to the task that interacts with sensitive data, the output of the failed task is suppressed, and the confidentiality of my secret is preserved:

TASK [Ensure API key is present in config file] ***********************************************************************************************************************************************
fatal: [localhost]: FAILED! => changed=false 
  censored: 'the output has been hidden due to the fact that ''no_log: true'' was specified for this result'

It’s a good idea to use no_log on any task that interacts with sensitive data. You should also be aware of its limitations: It will not prevent logging if Ansible debugging is turned on.

[ Looking for more on system automation? Get started with The Automated Enterprise, a free book from Red Hat. ] 

Final thoughts

Properly managing secrets is a common early challenge that many systems administrators face when working on deploying automation. In this article, I described and demonstrated three different methods that you can use to protect sensitive data when using Ansible in your environment. This article only scratched the surface of possibilities, so be sure to review the documentation that I linked throughout this discussion. Security is everyone’s job. You can do your part as a systems administrator by ensuring that you're treating private data with the sensitivity that it deserves in your automation pipelines.