Speeding up ansible playbooks or running many ansible playbooks in parallel

Home > CentOS > CentOS 6.x > System administration tools > ansible > Ansible tips and tricks > Speeding up ansible playbooks or running many ansible playbooks in parallel

To speed up ansible playbook module named firewall was introduced a while back. Information about fireball is available at http://jpmens.net/2012/10/01/dramatically-speeding-up-ansible-runs/ However as per https://github.com/ansible/ansible/issues/10599 it was recommended to use ansible acceleration feature instead of using fireball from ansible 1.3 onwards. Further as per http://docs.ansible.com/ansible/playbooks_acceleration.html ansible 1.5 onwards there is no need to use acceleration feature. Ansible 2.0 has introduced strategy defined at http://docs.ansible.com/ansible/playbooks_strategies.html but ansible 2.0 is not available as a package in latest updated CentOS 6.x at time of this writing. Thus, other ways of speeding up ansible playbooks are discussed here.

Example sequential site

Various methods of speed-up will be discussed based on this example sequential site:

hosts

[test1]
192.168.122.102

[test2]
192.168.122.103

main.yaml

- include: test1.yaml
- include: test2.yaml

test1.yaml

- name: Configure test1 machine
  hosts: test1
  remote_user: root

  vars:
  - test1_color : "Yellow"

  tasks:
  - name: Sleep for 30 seconds
    shell: sleep 30
    notify:
    - print1
    - print2

  - name: Echo test1_color
    debug: msg="color is {{test1_color}}"

  handlers:
  - name: print1
    shell: echo "print1 handler executed"

  - name: print2
    debug: msg="print2 handler executed"

test2.yaml

- name: Configure test1 machine
  hosts: test2
  remote_user: root

  vars:
  - test2_color : "Green"

  tasks:
  - name: Echo test2_color
    debug: msg="color is {{test2_color}}"

Here main.yaml includes the other two host configuration playbooks which execute sequentially. Thus setup of machine test2 does not starts till 30 second sleep times out on test1

Shell script based speed-up

A shell-script based speed up is possible by using same hosts, test1.yaml and test2.yaml files. But instead of using main.yaml use:

main.sh

#!/bin/bash

ansible-playbook -i hosts test1.yaml > test1.log &
ansible-playbook -i hosts test2.yaml > test2.log &

In this case test2.log would have test2.yaml output well before test1.log contains output related to test1.yaml. This is same as using multiple shells and executing test1.yaml and test2.yaml in different shells parallely.

ansible main.yaml based parallelization

The running of two shell-scripts in background (as done in previous 'Shell script based speed up' section) can also be done using modified main.yaml as follows:

main.yaml

- name: Run test1 and test2 in parallel
  hosts: ansible
  remote_user: root
  
  vars:
  - working_dir: "/root/ansible/03-ansible"


  tasks:
  - name: Run test1.yaml file
    shell: ansible-playbook -i hosts test1.yaml > test1.log &
    args:
      chdir: "{{working_dir}}"
    register: test1_output   

  - name: Run test2.yaml file
    shell: ansible-playbook -i hosts test2.yaml > test2.log &
    args:
      chdir: "/root/ansible/03-ansible"
    register: test2_output

  - name: Print test1 and test2 error output, if any
    debug: msg="Test1 basic status is {{test1_output}} and Test2 basic status is {{test2_output}}.  For normal output refer test1.log and test2.log (Not working)"

This example was created hoping that output of both ansible-playbook commands would get captured in respective log files. Unfortunately that is not happening. The example needs more work for obtaining ansible output for troubleshooting or status check purposes. However, both scripts do execute in parallel so this achieves speed up same as before. This method also has overhead of connecting to localhost (ansible-server) over SSH.

Thus best method is to use shell script as described before

fireball based speed-up (Obsolete)

Fireball has a few dependencies that can be installed using:

one_time_setup.yaml

- name: One time setup of test1 and test2 for fireball
  hosts: ansible:test1:test2
  remote_user: root
  gather_facts: false
  connection: ssh
  tasks:
  - yum: name={{item}}
    with_items:
      - python-pip
      - gcc
      - python-devel
      - gcc-c++
      - python-zmq
  - pip: name=pyzmq state=present
  - pip: name=pyasn1
  - pip: name=PyCrypto 
  - pip: name=python-keyczar

After this a modified main.yaml file, such as:

- name: Start firewall on test1 and test2
  hosts: ansible:test1:test2
  remote_user: root
  
  tasks:
  - name: Start fireball
    action: fireball

  
- include: test1.yaml
- include: test2.yaml 

should start fireball on all three nodes before starting execution of test1.yaml and test2.yaml. These two .yaml files must have modification such as:

- name: Configure test1 machine
  hosts: test1
  connection: fireball
  remote_user: root

to indicate use of fireball connection.

This example was found to be Not working. Suggestions on how to make it work are welcome. Refer http://docs.ansible.com/ansible/fireball_module.html

Also refer http://stackoverflow.com/questions/21158689/ansible-deploy-on-multiple-hosts-in-the-same-time for good answer on same topic.

Home > CentOS > CentOS 6.x > System administration tools > ansible > Ansible tips and tricks > Speeding up ansible playbooks or running many ansible playbooks in parallel