How to create dynamic inventory files in Ansible

If you use Ansible, you know the inventory is one of its fundamental pieces. The inventory is just a list of machines and possible variables where you can run your Ansible playbooks.

[ Download now: A system administrator's guide to IT automation. ]

An inventory file can be written in YAML, JSON, or Windows INI format and can describe groups of machines. For example:

---
all:
  children:
    servers:
      hosts:
        macmini2:
        raspberrypi:
      vars:
        description: Linux servers for the Nunez family
    desktops:
      hosts:
        dmaf5:
        mac-pro-1-1:
      vars:
        description: Desktops for the Nunez family

You can confirm that the file has the proper structure, too. For example, you might filter only the machines that belong to the desktops pattern:

[josevnz@dmaf5 ExtendingAnsibleWithPython]$ ansible-inventory --yaml --inventory /home/josevnz/EnableSysadmin/BashHere/hosts.yaml --graph desktops
@desktops:
  |--dmaf5
  |--mac-pro-1-1

Having a static YAML inventory may not be entirely practical for the following reasons:

1. Your host inventory is extensive. Admit it; you have better things to do than edit YAML files, right?
2. Your inventory is in a format that is not compatible with Ansible YAML. It may be in a database or plain-text file.
3. The servers that are part of your inventory are really dynamic. You create machines on your private cloud as you need them, and their IP addresses change frequently. Or perhaps you have a home network with many roaming devices (tablets and phones). Do you want to maintain that by hand?

Ways to manage inventories in Ansible

There are many ways to manage your inventories in Ansible. Here are some possibilities:

• Convert inventories from legacy formats into Ansible.
• Use dynamic inventories with plugins, specifically Nmap.
• Write your own inventory script to generate inventories dynamically.
• Write an Ansible inventory plugin.

This article goes into the first two items on that list, and follow-up articles will explain the latter two. Remember that whatever method you use, you must accomplish this while following good practices of packaging the tools, using virtual environments, and unit testing the code.

[ Learn more: Ansible vs. Terraform, clarified ]

Don't repeat yourself: Check first if someone wrote it for you

Chances are they did. You can quickly see if someone wrote a plugin that can generate an inventory from a different source like this:

$ ansible-doc -t inventory -l

For example:

$ ansible-doc -t inventory -l
advanced_host_list  Parses a 'host list' with ranges                                                                                                                                                                     
auto                Loads and executes an inventory plugin specified in a YAML config                                                                                                                                    
aws_ec2             EC2 inventory source                                                                                                                                                                                 
aws_rds             rds instance source                                                                                                                                                                                  
azure_rm            Azure Resource Manager inventory plugin                                                                                                                                                              
cloudscale          cloudscale.ch inventory source                                                                                                                                                                       
constructed         Uses Jinja2 to construct vars and groups based on existing inventory                                                                                                                                 
docker_machine      Docker Machine inventory source                                                                                                                                                                      
docker_swarm        Ansible dynamic inventory plugin for Docker swarm nodes                                                                                                                                              
foreman             foreman inventory source                                                                                                                                                                             
gcp_compute         Google Cloud Compute Engine inventory source                                                                                                                                                         
generator           Uses Jinja2 to construct hosts and groups from patterns                                                                                                                                              
gitlab_runners      Ansible dynamic inventory plugin for GitLab runners                                                                                                                                                  
hcloud              Ansible dynamic inventory plugin for the Hetzner Cloud                                                                                                                                               
host_list           Parses a 'host list' string                                                                                                                                                                          
ini                 Uses an Ansible INI file as inventory source                                                                                                                                                         
k8s                 Kubernetes (K8s) inventory source                                                                                                                                                                    
kubevirt            KubeVirt inventory source                                                                                                                                                                            
linode              Ansible dynamic inventory plugin for Linode                                                                                                                                                          
netbox              NetBox inventory source                                                                                                                                                                              
nmap                Uses nmap to find hosts to target                                                                                                                                                                    
online              Online inventory source                                                                                                                                                                              
openshift           OpenShift inventory source                                                                                                                                                                           
openstack           OpenStack inventory source                                                                                                                                                                           
scaleway            Scaleway inventory source                                                                                                                                                                            
script              Executes an inventory script that returns JSON                                                                                                                                                       
toml                Uses a specific TOML file as an inventory source                                                                                                                                                     
tower               Ansible dynamic inventory plugin for Ansible Tower                                                                                                                                                   
virtualbox          virtualbox inventory source                                                                                                                                                                          
vmware_vm_inventory VMware Guest inventory source                                                                                                                                                                        
vultr               Vultr inventory source                                                                                                                                                                               
yaml                Uses a specific YAML file as an inventory source  

Use the host_list plugin

This is the simplest plugin. You pass a list of machines or IP addresses, and you're good to go.

[ Need more on Ansible? Take a free technical overview course from Red Hat. Ansible Essentials: Simplicity in Automation Technical Overview. ]

Here's an example using the ping module and the remote user josevnz:

$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.17 mac-pro-1-1
192.168.1.16 macmini2
192.168.1.11 raspberrypi

$ cat /etc/hosts| /bin/cut -f1 -d' '|/bin/grep -P '^[a-z1]'
127.0.0.1
192.168.1.17
192.168.1.16
192.168.1.11

$ ansible -u josevnz -i $(/bin/cat /etc/hosts| /bin/cut -f1 -d' '|/bin/grep -P '^[a-z1]'|/bin/xargs|/bin/sed 's# #,#g') -m ping all
127.0.0.1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
192.168.1.11 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
192.168.1.16 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
192.168.1.17 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

No surprises here. But as you can see, this is not very convenient, as I had to do a little bit of Bash scripting to generate the host list. Also, the inventory is static (in the sense that it comes from the /etc/host file).

Here is a more interesting plugin that uses Nmap.

Use the Nmap plugin

The Nmap plugin allows you to use the well-known network scanner to build your inventory list. First, I'll explain how this works when you're running Nmap by hand.

[ Download a Bash shell scripting cheat sheet. ]

Crash course on Nmap

You can use Nmap on the command line to get a very good idea of what machines and services are on your network:

$ sudo nmap -v -n -p- -sT -sV -O --osscan-limit --max-os-tries 1 -oX $HOME/home_scan.xml 192.168.1.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2022-03-05 10:29 EST
NSE: Loaded 45 scripts for scanning.
Initiating ARP Ping Scan at 10:29
Scanning 254 hosts [1 port/host]
Completed ARP Ping Scan at 10:29, 5.10s elapsed (254 total hosts)
Nmap scan report for 192.168.1.0 [host down]
Nmap scan report for 192.168.1.2 [host down]
Initiating Connect Scan at 10:29
Scanning 4 hosts [65535 ports/host]
Discovered open port 443/tcp on 192.168.1.1
Discovered open port 8080/tcp on 192.168.1.1
Discovered open port 445/tcp on 192.168.1.1
Discovered open port 139/tcp on 192.168.1.1
Discovered open port 80/tcp on 192.168.1.1
Discovered open port 80/tcp on 192.168.1.4
Discovered open port 35387/tcp on 192.168.1.4

Keep in mind that this scan is a time-consuming operation. Nmap checks every port and host on your network, so this may take minutes or even hours if you don't tune up your query.

With that in mind, keep these useful links around. You will use them to tune your Nmap arguments:

• Nmap port-scanning techniques
• Nmap timing and performance
• Nmap timing templates (-T)

For this inventory, you care about machines where Ansible can use Secure Shell (SSH) and perform operations. Limiting the port number to TCP 22 will speed up the scanning considerably:

# '-n': 'Never do DNS resolution',
# '-p-': 'All ports. Use -p22 to limit scan to port 22',
# '-sV': 'Probe open ports to determine service/version info',
# '-T4': 'Aggressive timing template',
# '-PE': 'Enable this echo request behavior. Good for internal networks',
# '--version-intensity 1': 'Set version scan intensity. Default is 7',
# '--disable-arp-ping': 'No ARP or ND Ping',
# '--max-hostgroup 100': 'Hostgroup (batch of hosts scanned concurrently) size',
# '--min-parallelism 20': 'Number of probes that may be outstanding for a host group',
# '--osscan-limit': 'Limit OS detection to promising targets',
# '--max-os-tries 1': 'Maximum number of OS detection tries against a target',
# '-oX -': 'Send XML output to STDOUT, avoid creating a temp file'

$ nmap -v -n -p22 -sT -sV  --osscan-limit --max-os-tries 1 -oX $HOME/home_scan.xml 192.168.1.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2022-03-05 10:51 EST
NSE: Loaded 45 scripts for scanning.
Initiating Ping Scan at 10:51
Scanning 256 hosts [2 ports/host]
Completed Ping Scan at 10:51, 2.31s elapsed (256 total hosts)
Nmap scan report for 192.168.1.0 [host down]
Nmap scan report for 192.168.1.2 [host down]
Nmap scan report for 192.168.1.5 [host down]
Nmap scan report for 192.168.1.7 [host down]
...
Completed NSE at 10:51, 0.00s elapsed
Nmap scan report for 192.168.1.1
Host is up (0.0024s latency).

PORT   STATE  SERVICE VERSION
22/tcp closed ssh

Nmap scan report for 192.168.1.3
Host is up (0.070s latency).
...
Nmap scan report for 192.168.1.11
Host is up (0.00036s latency).
PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 8.2p1 Ubuntu 4ubuntu0.4 (Ubuntu Linux; protocol 2.0)
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel
...
Read data files from: /usr/bin/../share/nmap
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 256 IP addresses (8 hosts up) scanned in 2.71 seconds

But if you don't care about port scanning, then replace -p22 with the -sn (ping scan) flag:

$ time nmap -v -n -sn --osscan-limit --max-os-tries 1 -oX $HOME/home_scan.xml 192.168.1.0/24
Read data files from: /usr/bin/../share/nmap
Nmap done: 256 IP addresses (8 hosts up) scanned in 2.52 seconds

Skip to the bottom of list

Image

Download now

This small difference may be a factor for bigger networks, but this example keeps the port scanning on port 22.

One more thing: Going forward, I will not use the -n switch. If it's passed, DNS name resolution is disabled and you won't know the names of the machines you just scanned.

Is it possible to extend Nmap? Yes. Ansible has a nice open source plugin that wraps the Nmap command-line program and parses the results. To illustrate how that works, I want to share a Python script called nmap_scan_rpt.py that can correlate services found by Nmap with security advisories (that's one of the ways you can extend the original tool):

$ git clone git@github.com:josevnz/home_nmap.git $HOME/home_nmap.git
pushd home_nmap/
python3 -m venv $HOME/virtualenv/home_nmap/
. ~/virtualenv/home_nmap/bin/activate
nmap_scan_rpt.py $HOME/home_scan.xml

Image

Feel free to play with the code. You can also run Nmap as a web service, but now I will move back to Ansible with Nmap.

Ansible's Nmap plugin

Now you are ready to explore the Ansible Nmap plugin:

# We do not want to do a port scan, only get the list of hosts dynamically
---
plugin: nmap
address: 192.168.1.0/24
strict: False
ipv4: yes
ports: no
groups:
  appliance: "'Amazon' in hostname"
  regular: "'host' in hostname"

Then test it:

$ ansible-inventory -i ExtendingAnsibleWithPython/Inventories/home_nmap_inventory.yaml --lis

That produces a nice JSON that Ansible consumes:

{
    "_meta": {
      "hostvars": {
        "android-1c5660ab7065af69.home": {
          "ip": "192.168.1.4",
          "ports": []
        },
        "dmaf5.home": {
          "ip": "192.168.1.26",
          "ports": []
        }
      },
      "all": {
        "children": [
          "ungrouped"
        ]
      },
      "ungrouped": {
        "hosts": [
          "android-1c5660ab7065af69.home",
          "dmaf5.home",
          "macmini2",
          "new-host-2.home",
          "new-host-6.home",
          "raspberrypi"
        ]
      }
    }
}

NOTE: I could not get the jinja2 'groups' feature to work. Its purpose is to put hosts into dynamic groups based on their hostname.

Dynamic inventories in Ansible

I covered a lot of material, so here is a quick summary of what you have done:

• Introduced the host_list plugin and saw some of its obvious limitations, specifically when using a large number of hosts.
• Did a quick lesson on Nmap and how to use it to scan all network hosts. You created an inventory using this scan.
• Configured and ran the community Ansible Nmap tool and compared its functionality with a manual Nmap scan.

In the next article in this series, I'll show you how to write your own dynamic inventory script and why this may be a better option than using ready-to-use plugins. And in the final article in the series, I'll explain why it might be better to write an Ansible plugin instead of using an inventory script.

Remember, you can download the code and experiment. The best way to learn is by doing and making mistakes.