Ansible JSON - Parse JSON using Ansible json_query

Updated on:

Ansible json_query is an on-demand feature that every ansible user wants to explore. In this post we are going to how Ansible JSON processing works.

During the infrastructure automation, we might end up in a situation where we need to process Huge datasets, especially in JSON format.    We might also want to extract Single data or property from the huge dataset.

Typically, we would be looking for JSON parsers to do the job for us or we would create some in-house ones.

But here comes ansible's json_query filter to save the day

 

What is  json_query and how does it work?

As the name suggests, the Ansible json_query filter is helping you to query the JSON document and get the elements in the JSON tree structure.

json_query is using the jmespath  Query language. It is a powerful query language to parse JSON content. It helps you to parse JSON content and filter the elements you want. You can pretty much do everything with JSON

Look at the following screenshot, A Home Page of JMESPATH

It has a nice toolset as well, where you could write and test your queries before you try it in realtime.

It is more like regex101 we use for Regular expression testing.

 

 

Some Sample JSON for testing - Ansible JSON

This is the sample data we are going to take for our testing. This is taken from the JSON Placeholder users endpoint URL.

JSON Placeholder has a set of JSON endpoints/URLs you can use for testing. It is more like a lorem ipsum for JSON. You can visit the site and explore.

[
  {
    "id": 1,
    "name": "Leanne Graham",
    "username": "Bret",
    "email": "[email protected]",
    "address": {
      "street": "Kulas Light",
      "suite": "Apt. 556",
      "city": "Gwenborough",
      "zipcode": "92998-3874",
      "geo": {
        "lat": "-37.3159",
        "lng": "81.1496"
      }
    },
    "phone": "1-770-736-8031 x56442",
    "website": "hildegard.org",
    "company": {
      "name": "Romaguera-Crona",
      "catchPhrase": "Multi-layered client-server neural-net",
      "bs": "harness real-time e-markets"
    }
  },
  {
    "id": 2,
    "name": "Ervin Howell",
    "username": "Antonette",
    "email": "[email protected]",
    "address": {
      "street": "Victor Plains",
      "suite": "Suite 879",
      "city": "Wisokyburgh",
      "zipcode": "90566-7771",
      "geo": {
        "lat": "-43.9509",
        "lng": "-34.4618"
      }
    },
    "phone": "010-692-6593 x09125",
    "website": "anastasia.net",
    "company": {
      "name": "Deckow-Crist",
      "catchPhrase": "Proactive didactic contingency",
      "bs": "synergize scalable supply-chains"
    }
  }]

 

In this article, we are going to parse JSON response given by the URL directly in case if you would like to read JSON file and parse them with ansible. It is possible too.

I covered how to Parse JSON file using ansible here

Parse JSON data from URL response - using Ansible json_query

Let us consider the JSON content given above and try to filter only the following fields for each user using ansible json_query and create a Business card type of data for each user.

So now we are going to create a playbook, which is going to do the following operation

  1. Download the preceding JSON users data from the JSON placeholder URL
  2. Using the ansible json_query filter. Print the following fields for each user ( with custom key names)
    1. Name
    2. Email
    3. Phone
    4. CompanyName
    5. WebSite
    6. City

Now we need to come up with the JMESpath  query that we are going to use in ansible json_query filter to get only these elements from on otherwise huge dataset.

I have used the JMESPath examples and tester to come up with the query and this is what the query I came up with

We are using the MultiSelect syntax of jmespath for this.

[*].{Name: name, Email: email, Phone: phone, CompanyName: company.name, WebSite: website, City: address.city}

Here is the playbook.

---
 - name: JsonQuery Playbook
   hosts: localhost
   tasks:
     - name: Download JSON content play
       uri:
         url: https://jsonplaceholder.typicode.com/users
         return_content: yes
       register: jsoncontent


     - name: Business Card
       debug: msg="{{ jsoncontent.json | json_query(jmesquery) }}"
       vars:
         jmesquery: "[*].{Name: name, Email: email, Phone: phone, CompanyName: company.name, WebSite: website, City: address.city}"

If you go through the preceding playbook you would understand that there is two tasks or Plays. the former one is to download the JSON content from the remote JSON PlaceHolder URL and the latter one is to parse the JSON content we got in the previous task and prepare a Business Card by choosing a set of selective fields.

I am assigning our query to the variable named jmesquery and using the variable as a parameter inside the ansible json_query.

When I run the playbook, My output looked something like this ( I have truncated some of the users to keep it simple) Ideally it would give me 10 business cards as the endpoint has 10 set of users.

➜  AnsibleWorkSpace git:(master) ✗ ansible-playbook ansible_json.yml

PLAY [JsonQuery Playbook] ********************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************
ok: [localhost]

TASK [Download JSON content play] ************************************************************************************************************************************************
ok: [localhost]

TASK [Business Card] *************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        {
            "City": "Gwenborough",
            "CompanyName": "Romaguera-Crona",
            "Email": "[email protected]",
            "Name": "Leanne Graham",
            "Phone": "1-770-736-8031 x56442",
            "WebSite": "hildegard.org"
        },
        {
            "City": "Wisokyburgh",
            "CompanyName": "Deckow-Crist",
            "Email": "[email protected]",
            "Name": "Ervin Howell",
            "Phone": "010-692-6593 x09125",
            "WebSite": "anastasia.net"
        },
............
............
PLAY RECAP ***********************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

From the result shown. You could understand that the result has been Sorted by Ansible, despite our query is designed to give the City as the last value.

 

 

Example2: List only the Names

Here is one more example to list only the names from the JSON Dataset

---
 - name: JsonQuery Playbook
   hosts: localhost
   tasks:
     - name: Download JSON content play
       uri:
         url: https://jsonplaceholder.typicode.com/users
         return_content: yes
       register: jsoncontent


     - name: Just the Names
       debug: msg="{{ jsoncontent.json | json_query(jmesquery)}}"
       vars:
         jmesquery: "[*].name"

here is the result of the playbook

➜  AnsibleWorkSpace git:(master) ✗ ansible-playbook ansible_json.yml

PLAY [JsonQuery Playbook] **********************************************************************************

TASK [Gathering Facts] *************************************************************************************
ok: [localhost]

TASK [Download JSON content play] **************************************************************************
ok: [localhost]

TASK [Just the Names] ***************************************************************************************
ok: [localhost] => {
    "msg": [
        "Leanne Graham",
        "Ervin Howell",
        "Clementine Bauch",
        "Patricia Lebsack",
        "Chelsey Dietrich",
        "Mrs. Dennis Schulist",
        "Kurtis Weissnat",
        "Nicholas Runolfsdottir V",
        "Glenna Reichert",
        "Clementina DuBuque"
    ]
}

PLAY RECAP *************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

 

 

Example3: Filter only specific user or users

In the first example, we have seen how to filter out the necessary fields to create business card type data set. But there we got all 10 users data. What if we want to limit it to a specific user

Here is the query we are going to use to limit the user by name

 In the following query the name is surrounded by a backquote/backtick ` not a Single quote.

[? name==`Leanne Graham`].{Name: name, Email: email, Phone: phone, CompanyName: company.name, WebSite: website, City: address.city}

The preceding JMESPath Query finds the user with a name Leanne Graham and getting his business card.

to Run this in a playbook, You just have to modify the jmesquery variable

---
 - name: JsonQuery Playbook
   hosts: localhost
   tasks:
     - name: Download JSON content play
       uri:
         url: https://jsonplaceholder.typicode.com/users
         return_content: yes
       register: jsoncontent


     - name: Business Card
       debug: msg="{{ jsoncontent.json | json_query(jmesquery)}}"
       vars:
         jmesquery: "[? name==`Leanne Graham`].{Name: name, Email: email, Phone: phone, CompanyName: company.name, WebSite: website, City: address.city}"

 

The output of this playbook is given below.

➜  AnsibleWorkSpace git:(master) ✗ ansible-playbook ansible_json.yml

PLAY [JsonQuery Playbook] **********************************************************************************

TASK [Gathering Facts] *************************************************************************************
ok: [localhost]

TASK [Download JSON content play] **************************************************************************
ok: [localhost]

TASK [Business Card] ***************************************************************************************
ok: [localhost] => {
    "msg": [
        {
            "City": "Gwenborough",
            "CompanyName": "Romaguera-Crona",
            "Email": "[email protected]",
            "Name": "Leanne Graham",
            "Phone": "1-770-736-8031 x56442",
            "WebSite": "hildegard.org"
        }
    ]
}

PLAY RECAP *************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

 

If you want to use multiple users in your search. You can do it just like you would do any other programming language by using or ||

Here is the jmespath query you can use in your ansible json_query

[? name==`Leanne Graham` || name==`Ervin Howell`].{Name: name, Email: email, Phone: phone, CompayName: company.name, WebSite: website, City: address.city}

You can write many jmes queries using the JMESPath testing tool and use it in your playbook.

 

 

 

Example4: Parse json_query result in a loop and access it as variable

So far we have seen jmespath queries and how to use them in ansible json_query module. Now we are going to see how to parse the json_query result in an ansible loop and create some dynamic variables (or) use them in the task.

Now we are going to create a Phonebook with the dataset we have.

Here we are NOT going to use major jmes query. Rather, we are going to use Ansible's inbuilt features like combine to create the phonebook.

The Expected result is a set of

PersonName: PhoneNumber

Here is the playbook we use to get this done for us

---
 - name: JsonQuery Playbook
   hosts: localhost
   tasks:
     - name: Download JSON content play
       uri:
         url: https://jsonplaceholder.typicode.com/users
         return_content: yes
       register: jsoncontent

     - name: Collecting UserName and Mobile Number info
       no_log: True
       set_fact:
         phonebook: "{{phonebook|default({}) | combine ( {item.name : item.phone}) }}"
       with_items: "{{ jsoncontent.json | json_query('[*]')}}"

     - name: The Phonebook
       debug: var=phonebook

you can see that the json_query we have used is pretty simple, All the logic part is being done by Ansible.

A quick explanation of the logic

- phonebook|default({})  creates a new dictionary named phonebook

- combine ({item.name : item.phone}) used to add an element in the dictionary.

There is a new keyword used in this playbook which deserves an explanation

- no_log this prevents the content being displayed as ansible loop iterate the json which is otherwise displayed. You can try to remove this and execute the playbook to see what does it result.

 

Here is the execution output of this playbook

➜  AnsibleWorkSpace git:(master) ✗ ansible-playbook ansible_json.yml

PLAY [JsonQuery Playbook] **********************************************************************************

TASK [Gathering Facts] *************************************************************************************
ok: [localhost]

TASK [Download JSON content play] **************************************************************************
ok: [localhost]

TASK [Collecting UserName and Mobile Number info] **********************************************************
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost]

TASK [The Phonebook] ***************************************************************************************
ok: [localhost] => {
    "phonebook": {
        "Chelsey Dietrich": "(254)954-1289",
        "Clementina DuBuque": "024-648-3804",
        "Clementine Bauch": "1-463-123-4447",
        "Ervin Howell": "010-692-6593 x09125",
        "Glenna Reichert": "(775)976-6794 x41206",
        "Kurtis Weissnat": "210.067.6132",
        "Leanne Graham": "1-770-736-8031 x56442",
        "Mrs. Dennis Schulist": "1-477-935-8478 x6430",
        "Nicholas Runolfsdottir V": "586.493.6943 x140",
        "Patricia Lebsack": "493-170-9623 x156"
    }
}

PLAY RECAP *************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

We have seen various examples of ansible json_query and how ansible works with json. Hope this helps.

If you have any questions. Feel free to comment

 

Cheers
Sarav AK

Follow me on Linkedin My Profile
Follow DevopsJunction onFacebook orTwitter
For more practical videos and tutorials. Subscribe to our channel

Buy Me a Coffee at ko-fi.com

Signup for Exclusive "Subscriber-only" Content

More from Middleware Inventory

  • How to Run Ansible Playbook Locally | Devops Junction

    In this post, we are going to see how to run the playbook locally on our Localhost on the control machine. How to run Ansible Playbook locally is one of the questions I used to have before I learnt it. Let me tell you how to Run Ansible Playbook Locally…

  • Ansible Playbook Examples
    Ansible Playbook Examples - Sample Ansible Playbooks | Devops Junction

    In this post, we are going to see examples of Ansible playbook and various different modules and playbook examples with various modules and multiple hosts. We will start with a basic Ansible playbook and learn what is task and play and what is playbook etc. What is Ansible Playbook It…

  • Ansible Dry run
    Ansible Dry Run - How to Run Playbook in Ansible Check mode

    Ansible Dry Run or Ansible Check mode feature is to validate your playbook before execution With the Ansible Dry Run feature, you can execute the playbook without having to actually make changes on the server.  this enables us to see what changes are going to be made when the playbook…

  • Ansible Sudo
    Ansible sudo - ansible become example

    Ansible Sudo or Ansible become Introduction Ansible Sudo or become is a method to run a particular task in a playbook with Special Privileges like root user or some other user. In the earlier versions of ansible there is an option named as sudo which is deprecated now, Since ansible…

  • ansible lineinfile multiple lines
    Ansible lineinfile multiple lines - Replace multiple Lines | Devops Junction

    In this post, we are going to see how to use Ansible lineinfile module to replace multiple Lines at the same time. How to use Multiple Regular Expressions or Regex at the same time. How to Match Multiple lines. For this example, we are going to take apache httpd.conf file…