I needed to be able to automate the download of specifically named files from an FTP server and then delete them upon successful download. The need arose when the old script I was using was downloading some zero byte files (likely due to poor network conditions, workstation issues or remote server issues) then deleting the source file! D’oh!!

This script checks that the file is not a zero byte file before deleting from the remote server. Not great error/consistency checking, but better than I had before. It also copies the successfully downloaded files to an IMPORT directory and then archives the originally downloaded files to an ARCHIVE directory.

Hopefully you find it useful.

Note: I updated the post to include logging.

#!/bin/bash
################################################
# Set some of the variables that will be needed
# during the execution of this script
################################################
# set current date
now=`date +%Y-%m-%d-%H%M-%S`
#
# set the root dir
rootdir="/ftp_files"
#
# set the download directory
dldir="$rootdir/DOWNLOADS"
#
# set session download dir
sessdldir="$dldir/$now"
#
# set the import dir
importdir="$rootdir/IMPORT"
#
# set the archive dir
archivedir="$rootdir/ARCHIVE"
#
# set up logging
LOGFILE="$rootdir/LOGS/ftplog.txt"
#
################################################
# Lets get to work
#################################################
# create a directory based on todays date
mkdir $sessdldir
#
# change to the new directory prior to FTP connection
cd $sessdldir
#
if [ -f $LOGFILE ]
then
echo "****** FTP PROCESS STARTED AT: $now ******" >> $LOGFILE
else
echo "****** NEW LOG FILE CREATED AT: $now ******" > $LOGFILE
echo "****** FTP PROCESS STARTING AT: $now ******" >> $LOGFILE
fi

## FTP Connection parameters
ftpserver=server.com
username=youruser
password=yourpass
remotedir=/directoryname

## Connect to FTP server and download all TXT files
ftp -inv $ftpserver >> $LOGFILE <<MYEND
user $username $password
cd $remotedir
bin
mget *.TXT
bye
MYEND

## Test for existence of files in the session download dir
## (we cd'd here earlier)
if [ `ls | wc -l` -gt 0 ]
then
echo "" >> $LOGFILE
echo ">> Files have been downloaded" >> $LOGFILE
dlfiles=`ls`
echo ">>>> Downloaded files:" >> $LOGFILE
echo "$dlfiles" >> $LOGFILE
echo "" >> $LOGFILE

## Create the FTP delete script in order to remove the downloaded
## files from the FTP server. We will be sure to check for empty
## files below. The FTP delete script will be created in the session
## download directory and named ftp.$PID
echo "open $ftpserver
user $username $password
binary
cd $remotedir" > $sessdldir/ftp.$$
echo ">> Creating delete script to delete non-zero files from remote server" >> $LOGFILE

## Iterate through the downloaded files one by one
for i in $( ls *.TXT ); do
## If the file exists and is larger than 0 bytes, THEN add
## a delete command to the FTP delete script. This will allow
## us to delete all non-zero byte files from the FTP server.
## Non-zero byte files could be a sign that the file didn't
## download properly due to an FTP timeout or other network
## issues. By not deleting the non-zero byte file we can attempt
## to download it the next time the script is run.
if [ -s $i ]
then
echo "del $i" >> $sessdldir/ftp.$$
echo ">>>> $i will be added to delete script" >> $LOGFILE
fi
done

## Finish off the FTP delete script
echo "quit" >> $sessdldir/ftp.$$

echo "" >> $LOGFILE
echo ">> Here is the delete script">> $LOGFILE
echo "" >> $LOGFILE
cat $sessdldir/ftp.$$ >> $LOGFILE
echo "" >> $LOGFILE
echo ">> Running the delete script" >> $LOGFILE
echo "" >> $LOGFILE

## Run the FTP script
ftp -ivn < $sessdldir/ftp.$$ >> $LOGFILE

## Copy all the non-zero byte TXT files to the import folder
echo "" >> $LOGFILE
echo ">> Copying non-zero size files to IMPORT" >> $LOGFILE

for i in $( ls *.TXT ); do
if [ -s $i ]
then
cp $i $importdir
echo ">>>> Copying $i to IMPORT directory" >> $LOGFILE
fi
done
fi

## Copy session specific download dir to the ARCHIVE folder
echo "" >> $LOGFILE
echo ">> Copying entire $sessdldir to $archivedir" >> $LOGFILE

cp -R $sessdldir $archivedir/

## Delete the session specific download dir
rm -rf $sessdldir
echo "" >> $LOGFILE
echo ">> Deleting $sessdldir" >> $LOGFILE
echo "" >> $LOGFILE

## ENDING LOG FILE FOR SESSION
echo "****** FTP PROCESS ENDED AT: $now ******" >> $LOGFILE
echo "******************************************************" >> $LOGFILE
echo "" >> $LOGFILE

Please consider subscribing to GeoffManning.com to receive new posts in your RSS Reader or by Email.