BASH Script: Copy Files From Source To Destination Within Given Date Range By RSYNC

 After the Previous Post, let's discuss another important task. Copy Files / Folders from one location to another but efficiently. There are many ways to do this task in LINUX. Also one can find multiple commands as well. We will discuss one specific scenario as follows.

Let's assume that you want to copy Folders / Files from source to destination but only if source location has the specified folders. Also this operation will be spanning for a range of dates or within start date and end date. Also one additional thing to remember is the directory structure. How you want to preserve your directory structure in destination location that also matters a lot. In our case, let's say that we have directories with dates (directory name is in the format of date. e.g. 20160101) in YYYYMMDD format. Do remember though that one can have any format for the directory as per their own requirements but I am going to discuss for above mentioned scenario only. If you want then you can change the code keeping in mind that your changes reflect the task you want to perform.

Summary of scenario,

1. Copy only if Source location has the specified files. Here we will use "rsync" command to make it efficient and fast.

2. Directory structure in the Destination location must match to Source location.

3. The Date range should be in YYYYMMDD Format.

4. Script should accept Date range as command line arguments.

Following is the sample code of the implementation. You can also get it from my GitHub repository.

#!/bin/bash
# This code takes Date as input in YYYYMMDD Format.
# Copies the Files / FOlders to Destination if Source contains the files.
# It uses rsync command to copy files.

StartDate=`date +"%Y%m%d" -d $1`  #"20160601"`
EndDate=`date +"%Y%m%d" -d $2`    #"20160605"`
src=/home/yogesh/
dest=/home/cstechera/
if [ $# -ne 2 ]
then
    echo "Usage:`basename $0` Start_Date End_Date"
    echo "bash `basename $0` 20160601 20160605"
    exit $E_BADARGS
fi

if [[ ! -d $src || ! -d $dest  ]]
then
 echo "Given Source or Destination Path does not Exists. Please check the Path."
 exit $E_NOFILE
fi

echo "StartDate: "$StartDate
echo "EndDate: "$EndDate
echo "Source Path: "$src
echo "Destination Path: "$dest

while [ "$StartDate" -le "$EndDate" ] ; 
do 
 if [ -d  $src$StartDate ]     
   echo $src$StartDate
   
   rsync $src$StartDate/*.* $dest$StartDate/
 fi    
   # Increment Date by one day
 StartDate=`date +"%Y%m%d" -d "$StartDate + 1 day"`; 
done
echo "All Done"

A sample command to run the above program would be like as follows.

$ bash CopyFilesDateAsInput.bash 20160101 20161231

Let's discuss about the above mentioned code. The BASH Script takes two command line arguments. Start Date as First & End Date as Second. It checks that valid arguments are provided or not at the time of execution of script otherwise it exits without processing further with a "Usage" message. It also checks the existence of Source as well as Destination paths.

If both the IF conditions are satisfied then; It copies all the files / folders in destination path only if source path contains the folder related to that date. Also, please note here that in this post  we have not  used "cp" command which overwrites all the contents in destination path if any files are already present into destination path.  As it reduces efficiency and speed of the operation without checking existence of files. For some applications this type of overwriting files may not be a good code design. So, instead we use rsync command to avoid the overwriting of the files which are present already. It helps a lot when one need to synchronize the locations with appropriate data. Also, imagine what will happen if your files which are being copied are very huge in size.

Following is the sample output of the script for quick reference.

$bash CopyFilesDateAsInput.bash 20160101 20160103
StartDate: 20160101
EndDate: 20160103
Source Path: /home/yogesh/
Destination Path: /home/cstechera/
/home/yogesh/20150601
/home/yogesh/20150602
/home/yogesh/20150603
All Done

I hope you understood the discussion so far and liked the post.

I would like to Thank You for visiting the Website & going through the post. Stay tuned for more interesting stuff.

==>Posted By Yogesh B. Desai


Next Post: BASH Home Page

Previous Post: BASH Script: Copy Files From Source To Destination Within Given Date Range