How to Copy files between remote hosts in ansible

How to copy files between remote servers in Ansible ?. is the purpose behind this article.  Every article I went and read was giving me suggestions but nothing practical. I simply wanted to do make ansible rsync. I wanted to execute rsync between remote hosts and copy files between remote hosts.

So here is the example oriented article.  In this post, we are going to discuss the two major options available to copy the files between remote servers or remote to remote.

When we want to copy files from one host to another host, in general, the first thing comes in our mind is SCP and ansible have a dedicated module for the same-named COPY

If you want to copy a file from an Ansible Control Master to remote hosts, the COPY (scp) module would be just fine. But we need something more when we want to copy the files between the remote hosts or to copy files between remote to remote.

We are going to use some of the ansible modules like copy, fetch, synchronize to accomplish the requirement of copying the files between hosts in ansible.

While we are already aware of what is Ansible Copy module is used for.

Here is the Simple introduction to Ansible Fetch and Ansible Synchronize.

Ansible Fetch is to pull the files from the remote server to the control machine.

Ansible Synchronize is used to copy the files between remote servers (or) target hosts.  This is more like performing RSYNC with help of Ansible.

 

The Index

1. The Plan / The Requirement
2. Method1: Copy files between remote hosts using fetch module
3. Method2: Copy file between remote hosts using synchronize module
   1. Synchronize Push
   2. Synchronize Pull
4. The Execution Output
5. References and Other Related Articles.

 

The Plan / The Requirement

To get started, let us define some requirement ourselves so that, we could relate and be on the same page.

We have two servers named as follows

1. mwiappi01
2. mwiapp02

The requirement is to copy a file from mwiapp01 to mwiapp02. Before we proceed, This is the infrastructure diagram of our setup.

 Do not get confused by the name Master.  It is just a control machine. Remember  Ansible is agent less. so there is no master and slave.

 

 

Method1: Copy from app01 to app02 using fetch module

  NO SSH Authentication required between remote nodes for this method

This is the best and easiest option to transfer files between the remote nodes When you do not have SSH Key based Authentication enabled between the remote nodes. Though, it is not the shortest path (or) beeline. It works well where our primary focus is to get the job done whatsoever the approach may be.

it is a two-step process when it comes to this method.

Step1:  Fetch the file from the remote server (source) to the ansible master

Step2:  Push (Copy) the file from the ansible master to the remote server ( destination)

Here the master acts like a buffer where the files are temporarily stored and then transferred.

Consider the following playbook which performs both these above-mentioned tasks.

---
 - hosts: app
   tasks:
     - name: Fetch the file from the mwiapp01 to master
       run_once: yes
       fetch: src=/tmp/app01-to-app02.jar dest=buffer/ flat=yes
       when: "{{ inventory_hostname == 'mwiapp01' }}"

     - name: Copy the file from master to mwiapp02
       copy: src=buffer/app01-to-app02.jar dest=/tmp/
       when: "{{ inventory_hostname == 'mwiapp02' }}"

here we use "when"  to perform the conditional based execution and  inventory_hostname is the ansible built-in variable which will point the current execution remote hostname.

 Sometimes you might need to use ansible_hostname in place of inventory_hostname

Read this article to know the difference  between ansible_hostname and inventory_hostname

We are basically instructing the Ansible to execute the  Fetch task only on the mwiapp01 server and copy task only when the mwiapp02 is the current remote execution host.

 

Method 2: Copy from app01 to app02 using synchronize module

SSH Key based authentication must be enabled between remote hosts

For Synchronize module to work hassle-free,SSH Key-based authentication must be enabled between remote nodes. otherwise, the synchronize task will get stuck and so does your Ansible play.

In other words, the remote nodes should be able to login to each other without having to enter the password manually. We achieve this mostly with the help of SSH key.

As said earlier. Ansible Synchronize is more like Ansible RSYNC. the typical rsync feature of Linux.

To know how to enable Key-based Authentication just in seconds, Refer to the following article

Enable SSH Key based authentication using Ansible

Once we are ready with the SSH Key-based authentication. We are ready to play the play. ( playbook).

There are two methods to copy files using the Synchronize module

1. Synchronize Pull
2. Synchronize Push

Consider the following playbooks which copies the file from mwiapp01 to mwiapp02 using both Synchronize Pull and Synchronize Push method

 

 

Synchronize Pull

Targeted to:  Source Server (mwiapp01)

Delegated to and Executed on: Destination Server (mwiapp02)

Explanation:  As it is being executed in the destination server where the file has to arrive.  The task is to pull the file from the Origin (or) source server.

 

Here is the playbook which gets executed on the destination server mwiapp02  with help of delegation and pulls the file from mwiapp01 to mwiapp02

---
- name: Sync Pull task - Executed on  the Destination host "{{groups['app'][1]}}"
  hosts: "{{groups['app'][0]}}"
  user: wlsusr
  tasks:   
    - name: Copy the file from mwiapp01 to mwiapp02 using Method Pull
      tags: sync-pull
      synchronize:
        src: "{{ item }}"
        dest: "{{ item }}"
        mode: pull
      delegate_to: "{{groups['app'][1]}}"
      register: syncfile
      run_once: true
      with_items:
       - "/tmp/app01-to-app02.jar"

 

You might think the playbook is set to execute on the first server as the hosts directive is targeting the first server. hosts: "{{group[app][0]}}" mwiapp01

But you have to notice that we are actually executing this task on the second server mwiapp02  with the help of delegate_to

Look at the following Screen record to understand this better.

 

 

Synchronize Push

Targeted to: Destination Server (mwiapp02)

Delegated to and Executed on: Source Server (mwiapp01)

Explanation:  As it is being executed in the source server where the file is already present.  The task is to push the file to the destination server

Here is the playbook which gets executed on the source server mwiapp01 and pushes the file from 01 to 02

- name: Sync Push task - Executed on source host "{{groups['app'][0]}}"
  hosts: "{{groups['app'][1]}}" 
  user: wlsusr
  tasks:
    - name: Copy the file from mwiapp01 to mwiapp02 using Method Push
      tags: sync-push
      synchronize:
        src: "{{ item }}"
        dest: "{{ item }}"
        mode: push
      delegate_to: "{{groups['app'][0]}}"
      register: syncfile
      with_items:
       - "/tmp/app01-to-app02.jar"

 

You might think the playbook is set to execute on the second server as the hosts directive is targeting the first server. hosts: "{{group[app][1]}}" mwiapp02

But you have to notice that we are actually executing this task on the first server mwiapp01 with the help of delegate_to

The Playbook is made up of two host-based tasks list.  The First one runs using Synchronize Pull will run on mwiapp02and the second one with Synchronize Push will run on mwiapp01

Though the playbook is intelligible ( easy to understand).  I feel there are some peculiar lines which need more explanation, like "{{groups['app'][0]}}" 

It represents the first element  (or) server of the host group named "app" just like an array

Hope the following image would make it clear.

Other two important lines in the file are as follows

• mode  It represents the model of the Synchronize. Either Synchronize pull or Synchronize push
• delegate_to It tells ansible, where the task (or) command should actually be executed.

Hope you might have got it by now.  delegate_to and the hosts  along with mode in ansible-playbook,  work together to make the remote execution (or) copy between remote nodes possible.

 

Execution Output

In the preceding screenshot, we have executed 3 commands. First one and the last one is an Ansible ad-hoc command to check the availability of the file under the /tmpdirectory in mwiapp02server

we use --limit to execute the command in only one server, though we use the hostgroup which contains two servers.

we have successfully copied a file from mwiapp01 to mwiapp02  using both Fetch and Synchronize modules.

 

Conclusion

In this post, we have Successfully learnt and executed how to copy files between remote hosts using ansible copy and fetch modules. We have learnt two methods to copy the files between hosts using ansible.

Hope you find this article helpful.

 

References and other related articles

1.  Ansible Copy module reference
2. Ansible Fetch module reference
3. Ansible Synchronize module reference
4. How to Copy or Exchange SSH Key between multiple hosts - Ansible
5. How to exchange the IP and update /etc/hosts file in multiple hosts - Ansible

 

Hope this article helps. If you like this article, please share it

 

Thanks,
Sarav AK

---