Ansible get_url Examples - How to download file from URL

In this post, we are going to see how to download a file from URL using ansible get_url module in other words ansible curl. As we know it already, Ansible has a lot of built-in modules to accomplish all the tasks we might need for configuration management and automation.

Ansible get_url module is one amongst them which helps us to access the remote URL (HTTP/HTTPS/FTP) and download a file/package from there and save it locally.

Let us consider a situation where we need to download a package or a product online as we are provisioning an infrastructure. Let us say we build a machine with Apache or NGINX or Tomcat.

Based on the flavour of OS you choose you always have an option to download these products using their repository and installation management tool like yum (or) apt-get etc. But that does not suit all our needs. let us say I want to download a package or a file directly from the URL. How can I do that?

In a Simple approach, we can use the CURL or WGET command and get this job done. But why do you have to reinvent something which is already present and provides more features. ( You can always upgrade it )

So let's call this ansible get_url as Ansible CURL.

In this post, we are going to see how to use the ansible curl or ansible get_url module with examples.

If you want to simply access the web page or web site or work with SOAP or RESTFUL web services and crawl through the content of the body. you must consider using the ansible uri module, not get_url module

Table of Contents

The ansible get_url module

Some Key points about get_url module to get to know it better.

ansible get_url module is being used to download files from HTTPS, HTTP and FTP servers (websites/URLs)
By Default use the default proxy configuration of the node, You can use custom proxy and you can change the proxy address/url by setting environment variables such as http_proxy or https_proxy or by using the ansible built-in use_proxy option
you can use get_url to crawl through the page and check if the web page or server is available using --check option
Supports HTTPS and Secure Site accessing by letting you define the client certificates
Timeout configuration is available ( something important I believe)
Can pass through or handle the Basic Authentication. In other words, you can access the web sites/pages which are secure with Basic Authentication.
Supports checksum validation to make sure the downloaded software/package is uncorrupted and legit.

The ansible get_url Examples

How to download a file from URL [ Download Tomcat version 8.40 from HTTPS URL with checksum validation.(SHA512) ]
- Validate Checksum with SHA256/MD5 keys
Download a file ( tomcat) with checksum validation and take checksum value from remote URL
Download a file Secured with Basic Authentication
Timeout a Request when the URL is not responding (or) taking longer than usual
Downloading a FILE from FTP server using ansible get_url

Example #1 - Ansible - How to download a file from URL

Here is a quick playbook that downloads Tomcat 8.40 version from tomcat.apache.org

Before I write this playbook. I manually have to be aware of the downloadable link and the SHA512 checksum to validate. I have collected them from here

Here appserver is the host group defined in the ansible_hosts file which has one remote node or server underneath. Named mwivmapp01

The Playbook

The Playbook consists of two tasks and a couple of subtasks

Create a Directory named /opt/tomcat8
1. Make Sure the Directory has 0755 permission set
2. Set the Owner and the Group of the Directory as tomcat
Download the Tomcat8 tar.gz file into the /opt/tomcat8 directory we created in Step1
1. Access the Download link of the tomcat8 using url parameter of get_url module
2. Set the Download location as /opt/tomcat8
3. Make Sure the downloaded file at 0755 permission
4. Make Sure the owner and group of the downloaded tar.gz file are tomcat
5. Validate the sha512 checksum of the downloaded file, to make sure the file is valid.

Here the checksum value was pre declared (or) written in side the playbook. you can optionally specify a URL in which the checksum value of the desired file.

---
- name: Download Tomcat8 from tomcat.apache.org
  hosts: appserver
  tasks:
   - name: Create a Directory /opt/tomcat8
     become: yes
     file:
       path: /opt/tomcat8
       state: directory
       mode: 0755
       owner: tomcat
       group: tomcat

   - name: Download Tomcat using get_url
     become: yes
     get_url:
       url: https://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.40/bin/apache-tomcat-8.5.40.tar.gz
       dest: /opt/tomcat8
       mode: 0755
       checksum: sha512:5bdea5414713c9ba39e226f062701fa14998b1a798c9750f956a0f59b5edabb8d83af9ec9f81cf9f47fa92c21b560c9b2be1b543d0bd8f1b49579b69101d3a8f
       group: tomcat
       owner: tomcat

The Execution Output ( When the CheckSum Match Perfectly)

$ ansible-playbook download-tomcat.yml -i ansible_hosts

PLAY [Download Tomcat8 from tomcat.apache.org] *****************************************

TASK [Gathering Facts] *****************************************************************
ok: [mwivmapp01]

TASK [Create a Directory /opt/tomcat8] *************************************************
changed: [mwivmapp01]

TASK [Download Tomcat using get_url] ***************************************************
changed: [mwivmapp01]

PLAY RECAP *****************************************************************************
mwivmapp01                 : ok=3    changed=2    unreachable=0    failed=0

The Execution Output ( When the Checksum Do not match)

Since we have given checksum validation as part of the get_url module. When the checksum is not matching, The Task would fail and deployment would not be completed

$ ansible-playbook download-tomcat.yml -i ansible_hosts

PLAY [Download Tomcat8 from tomcat.apache.org] *****************************************

TASK [Gathering Facts] *****************************************************************
ok: [mwivmapp01]

TASK [Create a Directory /opt/tomcat8] *************************************************
ok: [mwivmapp01]

TASK [Download Tomcat using get_url] ***************************************************
fatal: [mwivmapp01]: FAILED! => {"changed": false, "msg": "The checksum for /opt/tomcat8/apache-tomcat-8.5.40.tar.gz did not match 5bdea5414713c9ba39e226f062701fa14998b1a798c9750f956a0f59b5edabb8d83af9ec9f81cf9f47fa92c21b560c9b2be1b543d0bd8f1b49579b69101d3; it was 5bdea5414713c9ba39e226f062701fa14998b1a798c9750f956a0f59b5edabb8d83af9ec9f81cf9f47fa92c21b560c9b2be1b543d0bd8f1b49579b69101d3a8f."}
	to retry, use: – limit @/apps/vagrant/webinfra/download-tomcat.retry

PLAY RECAP *****************************************************************************
mwivmapp01                 : ok=2    changed=0    unreachable=0    failed=1

Quick Validation - Ad hoc command to check if the file is present

ansible appserver -a "ls -lrt /opt/tomcat8" -i ansible_hosts

How to use SHA256 and MD5 Checksums while downloading the file

How to use SHA256 and MD5 Checksums while downloading the file.

To use SHA256 of MD5 checksums. All you have to do is update the checksum parameter of the get_url module

If you want to use SHA256 checksum

checksum: sha256:[sha256 checksum value]

If you want to use MD5 checksum

checksum: MD5:[MD5 checksum value]

You just have to define what is the Key algorithm before the value. That's all. It applies to all SHA256,MD5 or SHA256

Example #2 - Ansible - Download a file and get checksum from the URL

In the last example, we have seen how to download a file and do the checksum validation.

If you have noticed, the checksum was hardcoded or written inside the playbook. as we mentioned earlier you can also tell the ansible get_url module to fetch the checksum value from another URL

There is no major difference between example1 and example2 except the value of the checksum option.

The Playbook

---
- name: Download Tomcat8 from tomcat.apache.org
  hosts: appserver
  tasks:
   - name: Create a Directory /opt/tomcat8
     become: yes
     file:
       path: /opt/tomcat8
       state: directory
       mode: 0755
       owner: tomcat
       group: tomcat

   - name: Download Tomcat using get_url
     become: yes
     get_url:
       url: https://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.40/bin/apache-tomcat-8.5.40.tar.gz
       dest: /opt/tomcat8
       mode: 0755
       checksum: sha512:https://www.apache.org/dist/tomcat/tomcat-8/v8.5.40/bin/apache-tomcat-8.5.40.tar.gz.sha512
       group: tomcat
       owner: tomcat

Example #3 - Ansible - Download a Protected file with Basic Authentication

Sometimes, we might need to cross the security layer like Basic Authentication Prompt before downloading the file. In those kinds of scenarios, it is important that we pass the credentials and get authenticated before downloading the file.

Ansible get_url provides a way to pass username and password to Basic Authentication prompts. So that you can cross the security layer and reach and download the file

When you are not passing any credentials to the secured web page/ web site. You will get 401 Unauthorized HTTP Response

A Sample Basic Authentication Enabled File/URL

For testing, we have tweaked a simple Apache WebSite configuration to enable Basic Authentication Security for a specific file named "secure.txt" under the /var/www/html/admin document root

These are the apache configuration directives we had to add in httpd.conf file to achieve the Basic Authentication to a specific directory ( /var/www/html/admin )

Alias /admin /var/www/html/admin

<Directory /var/www/html/admin>
  <Files secure.txt>
    AuthType basic
    AuthName "Secured Files area"
    AuthUserFile /etc/httpd/conf/.htpasswd
    Require user sarav
  </Files>
  order allow,deny
  deny from all
  satisfy any
</Directory>

The Playbook

Here you can see we have three major tasks

Accessing the URL to make sure it is alive and Secured by validating it returns 401 status_code
Access the same URL with username and password this time to download the file
Display the file contents downloaded using a Basic Shell command

---
- name: Get-URL Download the Protected File
  hosts: appserver
  tasks:
    - name: Access the WebSite and make sure the URL is live
      uri:
        url: http://mwiweb02/admin/secure.txt
        status_code: 401
      register: validateurl

    - name: "INFO: HTTP Response for the URL"
      debug: var=validateurl.msg

    - name: Access the same URL with Basic Authentication and Download the file
      get_url:
        url: http://mwiweb02/admin/secure.txt
        url_password: Test@123 
        url_username: sarav
        dest: /tmp/secure.txt

    - name: Display the File contents
      shell: >
         cat /tmp/secure.txt
      register: shellout

    - name: "INFO: Display the File Contents"
      debug: var=shellout.stdout_lines

The Execution Result

$ ansible-playbook secure_download-basic.yml -i ansible_hosts

PLAY [Get-URL Download the Protected File] ***************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************
ok: [mwivmapp01]

TASK [Access the WebSite and make sure the URL is live] **************************************************************************************************************************
ok: [mwivmapp01]

TASK [INFO: HTTP Response for the URL] *******************************************************************************************************************************************
ok: [mwivmapp01] => {
    "validateurl.msg": "HTTP Error 401: Unauthorized"
}

TASK [Access the same URL with Basic Authentication and Download the file] *******************************************************************************************************
changed: [mwivmapp01]

TASK [Display the File contents] *************************************************************************************************************************************************
changed: [mwivmapp01]

TASK [INFO: Display the File Contents] *******************************************************************************************************************************************
ok: [mwivmapp01] => {
    "shellout.stdout_lines": [
        "this is a secret file with a secret password to open a Swizz bank locker",
        ":) :) "
    ]
}

PLAY RECAP ***********************************************************************************************************************************************************************
mwivmapp01                 : ok=6    changed=2    unreachable=0    failed=0

Example#4: Timeout When the URL is not responding ( Default is 10 Seconds)

We will take the same playbook we have used in Example1, and quickly update the timeout option for the get_url module. By default, the ansible get_url module would timeout in 10 seconds if gets no reply from the URL

Here is the modified playbook with the timeout option

---
- name: Download Tomcat8 from tomcat.apache.org
  hosts: appserver
  tasks:
   - name: Create a Directory /opt/tomcat8
     become: yes
     file:
       path: /opt/tomcat8
       state: directory
       mode: 0755
       owner: tomcat
       group: tomcat

   - name: Download Tomcat using get_url
     become: yes
     get_url:
       url: https://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.40/bin/apache-tomcat-8.5.40.tar.gz
       dest: /opt/tomcat8
       mode: 0755
       group: tomcat
       owner: tomcat
       # the request will timeout after 20 seconds
       timeout: 20

Example#5: Download a file from FTP server using ansible get_url

I have my FTP server running at ftp://mwiweb02/ and I need to access a Secure file like we shown in the previous example which is available at the URL ftp://mwiweb02/admin/secure.txt

Just like the other examples, this is also very simple and all we have to do is replace the HTTP URL with an FTP URL. That's all.

---
- name: Get-URL Download the Protected File
  hosts: appserver
  tasks:

    - name: Access the FTP URL and download the file
      get_url:
        url: ftp://mwiweb02/admin/secure.txt
        dest: /tmp/secure.txt

    - name: Display the File contents
      shell: >
         cat /tmp/secure.txt
      register: shellout

    - name: "INFO: Display the File Contents"
      debug: var=shellout.stdout_lines

The Execution Output

$ ansible-playbook download_ftp_url.yml -i ansible_hosts

PLAY [Get-URL Download the Protected File] *********************************************

TASK [Gathering Facts] *****************************************************************
ok: [mwivmapp01]

TASK [Access the FTP URL and download the file] ****************************************
changed: [mwivmapp01]

TASK [Display the File contents] *******************************************************
changed: [mwivmapp01]

TASK [INFO: Display the File Contents] *************************************************
ok: [mwivmapp01] => {
    "shellout.stdout_lines": [
        "this is a secret file with a secret password to open a Swizz bank locker",
        ":) :) "
    ]
}

PLAY RECAP *****************************************************************************
mwivmapp01                 : ok=4    changed=2    unreachable=0    failed=0

Hope this post helps. Read more Ansible articles here

Any help or feedback. Please let us know in comments.

Cheers,
Sarav AK

Follow me on Linkedin My Profile
Follow DevopsJunction onFacebook orTwitter
For more practical videos and tutorials. Subscribe to our channel

Signup for Exclusive "Subscriber-only" Content

The ansible get_url module

The ansible get_url Examples

Example #1 - Ansible - How to download a file from URL

How to use SHA256 and MD5 Checksums while downloading the file.

Example #2 - Ansible - Download a file and get checksum from the URL

Example #3 - Ansible - Download a Protected file with Basic Authentication

Example#4: Timeout When the URL is not responding ( Default is 10 Seconds)

Example#5: Download a file from FTP server using ansible get_url

More from Middleware Inventory