Backup Software

[classweb]

Weet iemand nog een goeie backup software dat de bestanden uiteindelijk als .tar(gz) opslaat?

[animana]

Wat je nodig hebt hangt af van waar je precies een backup van wilt hebben. Simpelste is gewoon een scriptje maken/zoeken dat met het tar commando alles tarred en zippen tot .tar.gz. In dit scriptje kun je dan precies aangeven wat wel en niet en kun je het in een cron-job plaatsen.
Deze file dan ftp-en of mailen en het gaat helemaal automatisch.

[Qweb]

Zowel mail als ftp zijn, voor zover wij het ervaren, onbetrouwbare protocollen voor backup doeleinden. Wij willen er 100% zeker van zijn dat de backup gewoon aankomt op de remote server, en wij hebben goede ervaringen met scp (nog geen foute overdracht meegemaakt). Ook met nfs hebben we met wat grotere bestanden problemen gehad. Kleine bestanden gaan met alle protocollen wel goed, al post je hem met http

[classweb]

met cron-job vind ik het niet echt betrouwbaar.. ik heb wel een script gevonden maar dat was jammer genoeg allen voor Red Hat bestemd. Hij werkte perfect tot het backup , maar de bestanden konden niet gekopieerd worden

[animana]

Ik heb geen problemen met cron-jobs. Wij maken prima backups met rsync over ssh welke gestart wordt als cron-job.

Hoe wil je het dan wel "betrouwbaar" en automatisch laten starten? Enige ander optie die ik zo snel kan bedenken is dat de backup software draait als daemon of je moet het elke dag met de hand starten.

[Digitux]

Origineel geplaatst door classweb
Weet iemand nog een goeie backup software dat de bestanden uiteindelijk als .tar(gz) opslaat?

Amanda is een mooi pakket waarmee je kan backuppen met tar en bv naar tape.

[classweb]

met cron-job de backup laten starten is wat anders dan het laten doen

[animana]

Sorry, maar ik mis 'm even ...
Wat voor problemen kom jij dan met cron-jobs tegen?

[peterw]

Ik heb ook zelf een script draaien die met cron dagalijks een backup maakt en gzipt (ook van de DB).
Bovendien krijg ik wekelijks in m'n mail (wel Rijdael-256 encrypted trouwens) een kopie van de database als extra backup

gelukkig is de DB nog niet zo groot :P

[classweb]

Peterw,

Wil jij die dan effe mailen? dan kan ik het effe bekijken..

Alvast bedankt..

[Qweb]

Als je op de cron niet kunt vertrouwen klopt er iets niet.

Cron roept gewoon een script aan op een bepaald tijdstip, wat er daarna met die script gebeurt zal crond een worst wezen.

[classweb]

Qweb precies dat bedoelde ik,

maar er zijn mensen bij die het via cron doen, ze geven alle commands op in cron en proberen het van daar op te lossen.. Dus er wordt helemaal geen script gebruikt voor het backup. Only Cron

[LeaseWeb]

Ik vond laatst in deze thread een aantal erg goede ideen.

Een van de m.i. beste, was onderstaande.
Er is nog wel wat tweaking nodig om het goed te laten werken op je eigen systeem.

Zelf heb ik en aantal simpelere liggen als je interesse heb laat het maar even horen.

mvg,


#! /bin/bash
# /usr/local/bin/backup
#######################
# Backup Script
#######################
# Jason Calabrese <mail <at> jasoncalabrese.com>
# Modified by Peter Ruskin <Peter.Ruskin <at> dsl.pipex.com> 25-Sep-03

# This script uses tar to back up your whole system (with an EXCLUDE
# variable to handle what you want to skip) to a dedicated backup disk.
# I have three 200GB disks, one of which has a single hdg1 partition for
# backups. Using tar by itself is *much* quicker than using bz2
# compression.

# Most of the script is taken up with reporting. I found the best way
# to get capacity is to cat /proc/partitions for the backup partition to
# get the TOTALBLOCKS and divide that by 1024 twice. That reports 189GB
# here.
# cat /proc/partitions doesn't work for 2.6 kernels.

# I pipe some of the calculations to bc to prevent divide-by-zero errors
# when fractions are involved.

# CHANGES
# 04-Dec-03 Peter Ruskin <Peter.Ruskin <at> dsl.pipex.com>: if clause added
# to TOTALBLOCKS to allow for 2.6 kernels
# 06-Dec-03 Peter Ruskin <Peter.Ruskin <at> dsl.pipex.com>: separate backups
# for linux and Windows
# 18-Jan-04 Peter Ruskin <Peter.Ruskin <at> dsl.pipex.com>: removed USEDSPACE
# and RSYNCSPACE variables; they use `du`
# which takes up too much time (6-8 minutes/calculation).

#######################

# Initial backup variables
BK_DISK=hdg
BK_PART=hdg1
BACKUP_TO=/mnt/backup
HOST=$( hostname|cut -f1 -d. )

# Make sure there is nothing on the unmounted mount point
echo "Ensuring that an unmouted ${BACKUP_TO} is empty..."
echo
if [[ "$(mount | grep ${BACKUP_TO})" == "" ]]; then
echo " ${BACKUP_TO} is not mounted"
echo
if [[ "$(ls ${BACKUP_TO})" != "" ]]; then
rm ${BACKUP_TO}/* 2> /dev/null
fi
fi

echo " Backup of ${HOST} started at `date +'%H:%M'`"
echo
echo " Mounting boot partition..."
mount /boot
echo
echo " Mounting backup disk..."
echo " ...waiting 10 seconds for disk to spin up..."
mount ${BACKUP_TO}
sleep 10
if [[ "$(mount | grep ${BACKUP_TO})" == "" ]]; then
echo " ${BACKUP_TO} is not mounted"
exit 1
fi
echo

echo " ...calculating..."
echo

# Variables and calculations before backup
FILE_LINUX=${BACKUP_TO}/$HOST-linux-$( date +"%F-%H%M" ).tar
FILE_WIN=${BACKUP_TO}/$HOST-win-$( date +"%F-%H%M" ).tar
FILE_LIST_LINUX=${BACKUP_TO}/$HOST-linux-$( date +"%F-%H%M" ).list
FILE_LIST_WIN=${BACKUP_TO}/$HOST-win-$( date +"%F-%H%M" ).list
EXCLUDE_LINUX='--exclude /mdk --exclude /mnt/downloads/Mandrake \
--exclude /mnt/downloads/Pan --exclude /mnt/backup \
--exclude /mnt/cdrom --exclude /mnt/floppy --exclude /mnt/KROH \
--exclude /mnt/KROH-LIESE --exclude /mnt/KROH-PETER \
--exclude /mnt/PENGUIN --exclude /mnt/win --exclude /proc \
--exclude /sys --exclude /usr/portage/distfiles --exclude /tmp \
--exclude /root/.ccache --exclude /home/peter/.ccache'
EXCLUDE_WIN='--exclude /mnt/win/d/Pan --exclude /mnt/win/j \
--exclude /mnt/win/l --exclude /mnt/win/n --exclude /mnt/win/o'
#USEDSPACE=$( du -sh ${BACKUP_TO} | grep G | awk '{print $1}' | \
# cut -f1 -dG )

# test for first run
unset FIRSTTIME
if ! [[ $(ls ${BACKUP_TO}/${HOST}-*.tar 2> /dev/null) ]]; then
# To pevent divide by zero errors.
# TARSPACE is the space occupied by tarballs. You may have other stuff
# on this disk.
# I do a daily rsync backup to this partition for instant rescue.
TARSPACE=1
TARSPACE_LINUX=1
TARSPACE_WIN=1
NUMBAKUPS=1
NUMBAKUPS_LINUX=1
NUMBAKUPS_WIN=1
FIRSTTIME=1
else
TARSPACE=$( du -sch ${BACKUP_TO}/${HOST}*.tar | grep total | \
awk '{print $1}' | cut -f1 -dG )
TARSPACE_LINUX=$( du -sch ${BACKUP_TO}/${HOST}-linux*.tar | \
grep total | awk '{print $1}' | cut -f1 -dG )
TARSPACE_WIN=$( du -sch ${BACKUP_TO}/${HOST}-win*.tar | \
grep total | awk '{print $1}' | cut -f1 -dG )
NUMBAKUPS=$( ls ${BACKUP_TO}/${HOST}-*.tar | wc -w | awk '{print $1}' )
NUMBAKUPS_LINUX=$( ls ${BACKUP_TO}/${HOST}-linux*.tar | wc -w | \
awk '{print $1}' )
NUMBAKUPS_WIN=$( ls ${BACKUP_TO}/${HOST}-win*.tar | wc -w | \
awk '{print $1}' )
fi

if [[ $(uname -r | cut -d. -f2) -eq 6 ]]; then
# This is disk manufacturers' gigabytes (1000000000 bytes)
# CAPACITY=$( echo "$(cat /sys/block/${BK_DISK}/size)*512/1000000000" \
#| bc )
CAPACITY=$( echo "$(cat /sys/block/${BK_DISK}/size) \
*512/1024/1024/1024" \ | bc )
else
TOTALBLOCKS=$(cat /proc/partitions|grep \
ide/host2/bus1/target0/lun0/part1|awk '{print $3}')
CAPACITY=$(( ${TOTALBLOCKS}/1024/1024 ))
fi
#RSYNCSPACE=$( du -sch ${BACKUP_TO}/rsync* | grep total | \
#awk '{print $1}' | cut -f1 -dG)
#REALCAPACITY=$( echo "${CAPACITY} - ${RSYNCSPACE}" | bc )
#SPACELEFT=$( echo "${CAPACITY} - ${USEDSPACE}" | bc )
SPACELEFT=$(df -h ${BACKUP_TO} | grep ${BACKUP_TO} | awk '{print $4}' \
| cut -dG -f1)
AVSIZE_LINUX=$( echo "${TARSPACE_LINUX}/${NUMBAKUPS_LINUX}" | bc )
AVSIZE_WIN=$( echo "${TARSPACE_WIN}/${NUMBAKUPS_WIN}" | bc )
AVSIZE=$( echo "${AVSIZE_LINUX}+${AVSIZE_WIN}" | bc )
#ESTBKUPS=$( echo "2*${REALCAPACITY}/${AVSIZE}" | bc )
ESTBKUPS=$( echo "${NUMBAKUPS} + (2 * ${SPACELEFT}/${AVSIZE}) - 2" \
| bc )

# Variables and calculations before backup
echo "STATUS BEFORE BACKUP..."
echo " Capacity of backup partition: ${CAPACITY}GB"
#echo " Space occupied by backup partition: ${USEDSPACE}GB"
#echo " Space occupied by rsync backups: ${RSYNCSPACE}GB"
echo " Space occupied by tar backups: ${TARSPACE}GB"
echo " Space left on backup partition: ${SPACELEFT}GB"
if [ $FIRSTTIME ] ; then
echo " Number of backups: This is the first backup."
else
echo " Number of backups: ${NUMBAKUPS}"
fi
echo " Average size per backup run: ${AVSIZE}GB"
echo " Average size of linux backups: ${AVSIZE_LINUX}GB"
echo " Average size of Windows backups: ${AVSIZE_WIN}GB"
echo " Estimated total number of backups: ${ESTBKUPS}"
echo

# For when there's not enough space...
if [ "$( echo "${SPACELEFT} <= ${AVSIZE}" | bc )" == 1 ] ; then
ESTBKUPS=$(( ${ESTBKUPS} - 2 ))
# so remove earliest
echo " Removing earliest archive - not enough space for backup..."
ls ${BACKUP_TO}/$HOST-linux*.tar | sort | head -n1 | xargs rm
ls ${BACKUP_TO}/$HOST-linux*.list | sort | head -n1 | xargs rm
ls ${BACKUP_TO}/$HOST-win*.tar | sort | head -n1 | xargs rm
ls ${BACKUP_TO}/$HOST-win*.list | sort | head -n1 | xargs rm
NUMBAKUPS=$( ls ${BACKUP_TO}/${HOST}-*.tar | wc -w | awk '{print $1}' )
NUMBAKUPS_LINUX=$( ls ${BACKUP_TO}/${HOST}-linux*.tar | wc -w | \
awk '{print $1}' )
NUMBAKUPS_WIN=$( ls ${BACKUP_TO}/${HOST}-win*.tar | wc -w | \
awk '{print $1}' )
echo " Number of backups: ${NUMBAKUPS}"
echo " Estimated total number of backups: ${ESTBKUPS}"
fi

#######################
# Now here's the actual backup: the easiest bit.
echo
echo " Performing backup at `date +'%H:%M'`..."
echo
#/bin/tar -cpvvf ${FILE} / ${EXCLUDE} > ${FILE_LIST} > /dev/null 2>&1
/bin/tar -cpvvf ${FILE_LINUX} / ${EXCLUDE_LINUX} > ${FILE_LIST_LINUX} \
> /dev/null 2>&1
echo " Linux backup finished at `date +'%H:%M'`"
echo " Archive is ${FILE_LINUX}"
echo
echo " Now backing up Windows..."
echo
/bin/tar -cpvvf ${FILE_WIN} /mnt/win/ ${EXCLUDE_WIN} >${FILE_LIST_WIN} \
> /dev/null 2>&1

echo " Windows backup finished at `date +'%H:%M'`"
echo " Archive is ${FILE_WIN}"
#######################

# Variables and calculations after backup
echo
echo " ...calculating..."
echo
#USEDSPACE=$( du -sh ${BACKUP_TO} | grep G | awk '{print $1}' | \
#cut -f1 -dG )
#RSYNCSPACE=$( du -sch $BACKUP_TO/rsync* | grep total | \
#awk '{print $1}' | cut -f1 -dG)
#REALCAPACITY=$( echo "${CAPACITY} - ${RSYNCSPACE}" | bc )
TARSPACE=$( du -sch ${BACKUP_TO}/${HOST}*.tar | grep total | \
awk '{print $1}' | cut -f1 -dG )
TARSPACE_LINUX=$( du -sch ${BACKUP_TO}/${HOST}-linux*.tar | \
grep total | awk '{print $1}' | cut -f1 -dG )
TARSPACE_WIN=$( du -sch ${BACKUP_TO}/${HOST}-win*.tar | grep total | \
awk '{print $1}' | cut -f1 -dG )
NUMBAKUPS=$( ls ${BACKUP_TO}/${HOST}-*.tar | wc -w | awk '{print $1}' )
NUMBAKUPS_LINUX=$( ls ${BACKUP_TO}/${HOST}-linux*.tar | wc -w | \
awk '{print $1}' )
NUMBAKUPS_WIN=$( ls ${BACKUP_TO}/${HOST}-win*.tar | wc -w | \
awk '{print $1}' )
#SPACELEFT=$( echo "${CAPACITY} - ${USEDSPACE}" | bc )
SPACELEFT=$(df -h ${BACKUP_TO} | grep ${BACKUP_TO} | awk '{print $4}' \
| cut -dG -f1)
AVSIZE_LINUX=$( echo "${TARSPACE_LINUX}/${NUMBAKUPS_LINUX}" | bc )
AVSIZE_WIN=$( echo "${TARSPACE_WIN}/${NUMBAKUPS_WIN}" | bc )
AVSIZE=$( echo "${AVSIZE_LINUX}+${AVSIZE_WIN}" | bc )
#ESTBKUPS=$( echo "2*${REALCAPACITY}/${AVSIZE}" | bc )
ESTBKUPS=$( echo "${NUMBAKUPS} + (2 * ${SPACELEFT}/${AVSIZE}) - 2" \
| bc )

if [ "$( echo "${SPACELEFT} <= ${AVSIZE}" | bc )" == 1 ]; then
ESTBKUPS=$(( ${ESTBKUPS} - 2 ))
# so remove earliest
echo " Removing earliest archive - not enough space for tomorrow..."
ls ${BACKUP_TO}/$HOST-linux*.tar | sort | head -n1 | xargs rm
ls ${BACKUP_TO}/$HOST-linux*.list | sort | head -n1 | xargs rm
ls ${BACKUP_TO}/$HOST-win*.tar | sort | head -n1 | xargs rm
ls ${BACKUP_TO}/$HOST-win*.list | sort | head -n1 | xargs rm
echo
# USEDSPACE=$( du -sh ${BACKUP_TO} | grep G | awk '{print $1}' | \
# cut -f1 -dG )
# RSYNCSPACE=$( du -sch $BACKUP_TO/rsync* | grep total | \
# awk '{print $1}' | cut -f1 -dG)
# REALCAPACITY=$( echo "${CAPACITY} - ${RSYNCSPACE}" | bc )
TARSPACE=$( du -sch ${BACKUP_TO}/${HOST}*.tar | grep total \
| awk '{print $1}' | cut -f1 -dG )
TARSPACE_LINUX=$( du -sch ${BACKUP_TO}/${HOST}-linux*.tar | \
grep total | awk '{print $1}' | cut -f1 -dG )
TARSPACE_WIN=$( du -sch ${BACKUP_TO}/${HOST}-win*.tar | \
grep total | awk '{print $1}' | cut -f1 -dG )
NUMBAKUPS=$( ls ${BACKUP_TO}/${HOST}-*.tar | wc -w | awk '{print $1}' )
NUMBAKUPS_LINUX=$( ls ${BACKUP_TO}/${HOST}-linux*.tar | wc -w | \
awk '{print $1}' )
NUMBAKUPS_WIN=$( ls ${BACKUP_TO}/${HOST}-win*.tar | wc -w | \
awk '{print $1}' )
# SPACELEFT=$( echo "${CAPACITY} - ${USEDSPACE}" | bc )
SPACELEFT=$(df -h ${BACKUP_TO} | grep ${BACKUP_TO} | \
awk '{print $4}' | cut -dG -f1)
AVSIZE_LINUX=$( echo "${TARSPACE_LINUX}/${NUMBAKUPS_LINUX}" | bc )
AVSIZE_WIN=$( echo "${TARSPACE_WIN}/${NUMBAKUPS_WIN}" | bc )
AVSIZE=$( echo "${AVSIZE_LINUX}+${AVSIZE_WIN}" | bc )
# ESTBKUPS=$( echo "2*${REALCAPACITY}/${AVSIZE}" | bc )
ESTBKUPS=$( echo "${NUMBAKUPS} + (2 * ${SPACELEFT}/${AVSIZE}) - 2" \
| bc )
fi

echo "STATUS AFTER BACKUP..."
echo " Capacity of backup partition: ${CAPACITY}GB"
#echo " Space occupied by backup partition: ${USEDSPACE}GB"
#echo " Space occupied by rsync backups: ${RSYNCSPACE}GB"
echo " Space occupied by tar backups: ${TARSPACE}GB"
echo " Space left on backup partition: ${SPACELEFT}GB"
echo " Number of backups: $NUMBAKUPS"
echo " Average size per backup run: ${AVSIZE}GB"
echo " Average size of linux backups: ${AVSIZE_LINUX}GB"
echo " Average size of Windows backups: ${AVSIZE_WIN}GB"
echo " Estimated total number of backups: $ESTBKUPS"
echo

echo " Backup of ${HOST} finished at `date +'%H:%M'`"
echo
echo " Unmounting boot and backup partitions..."

/bin/umount -l /boot
/bin/umount -l ${BACKUP_TO}

echo " Spinning down backup disk..."
/sbin/hdparm -y /dev/${BK_DISK}

echo
echo " To restore a file, search for it in"
echo " ${FILE_LIST_LINUX}, e.g. etc/fstab"
echo " or ${FILE_LIST_WIN}, e.g."
echo " mnt/win/c/scsi/drvlight.hlp, then:"
echo " # cd /"
echo " # tar -xpvf <full path to tar file> etc/fstab"
echo
#================================================= =====================