Riding Linux

Below is the bash script I use to mirror Centos repository into our local server. Due to storage limitation, we have to be selective and only copies the packages/branches that we need. Currently we only mirror Centos 5 for “os and updates” base packages limited to i386 and x86_64 architecture only.

During the initial intensive downloading when the server is replicating for the first time, I run the script below in cron for every 30 minutes. The script will check whether the previous instance is still running or dead (sometimes we got disconnected). If dead then it will restart and continue the rsync process. Once everything is in place and all needed packages have been downloaded, I modify the cron job to do the job once in every 6 hours to make sure my server is up to date.

The script below is inspired by the script posted here by jlar310

#!/bin/sh
# created by Jalte @ http://www.ridinglinux.org

DATE=`/bin/date +%Y-%m-%d`
OUTFILE=/var/www/html/mirror/centos-mirror.log
RSYNC="/bin/nice /usr/bin/rsync --verbose --progress --delete-excluded --stats --archive --partial --timeout=600"
MIRROR=mirrors.kernel.org::centos
VER=5
ARCHLIST="i386 x86_64"
BASELIST="os updates"
LOCAL=/var/www/html/mirror/centos

date >> /var/log/rsynccentos.log

if [ -f "/var/run/rsynccentos.pid" ]; then
   RUNPID=`cat /var/run/rsynccentos.pid`
   if ps -p $RUNPID; then
      echo "Mirror is already running..."
      echo "Mirror is already running..." >> /var/log/rsynccentos.log
      exit 1
   else
      echo "Mirror pid found but process dead, cleaning up"
      rm -f /var/run/rsynccentos.pid
      echo "Mirror pid found but process dead, cleaning up" >> /var/log/rsynccentos.log

   fi
else
   echo "No process Detected"
   echo "No process Detected" >> /var/log/rsynccentos.log

fi

echo $$ > /var/run/rsynccentos.pid

echo -n "Rsync Started at "
echo "Rsync Started " >> /var/log/rsynccentos.log
date

for ARCH in $ARCHLIST
do
  for BASE in $BASELIST
    do
        DIR=$LOCAL/$VER/$BASE/$ARCH/
        if [ -d $DIR ]
        then
         echo "Directory exists."
        else
         echo "Directory does not exist, and will be created."
         mkdir -p $DIR
        fi
        REMOTE=$MIRROR/$VER/$BASE/$ARCH/
        $RSYNC $REMOTE $DIR > $OUTFILE 2>&1
    done
done

echo "`date`" > /var/www/html/mirror/centos-last-updated.log
chown -R apache:apache $LOCAL

Any comments are greatly appreciated

Popularity: 33% [?]

Stumble it!

This entry was posted on Wednesday, July 4th, 2007 at 6:35 pm and is filed under Server Administration. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.