Hi all,
just wanted to mention that the backup process described below seems to
work. The 100 files gap is still about the same and I further
investigated the cause. It is related to the meta information like
indices and caches that are present in some but not all folders.
Counting only files that contain the sequence ,S= and even summing all
file sizes led to the same number and the exactly same size of raw mail
data.
I also didn't receive any notification about really failed backups,
therefore I believe that the backup works correctly.
Regards
Christian
On 09.01.2022 21:57, Christian wrote:
Hi all,
first: I'm using version 2.3.4.1
I manage some rather large imap mailboxes which I want to backup on a
regular basis. Some of them have relatively heavy traffic and one of
them is greater than 30GB in size.
I studied the docs for doveadm backup
(https://wiki2.dovecot.org/Tools/Doveadm/Sync) and even did some code
research to better understand the process.
The docs state that using stateful synchronization is the most
efficient way to synchronize mailboxes, therefore I chose this approach.
Highlevel overview:
- store a copy of the whole maildir in a separate directory
(/var/vmail/backup)
- backup to this directory once a minute (trying to make most use of
transaction logs) using the last state stored within a file
- create a backup once a day using tar (full, differential and
incremental ones) blocking the backup process of the before mentioned
step
I quite often receive notifications that doveadm backup returned an
exit code of 2, which should be quite normal. These notifications look
like that:
dsync(another_address@my.domain): Warning: Failed to do incremental
sync for mailbox INBOX, retry with a full sync (Modseq 171631 no
longer in transaction log (highest=177818, last_common_uid=177308,
nextuid=177309))
dsync(another_address@my.domain): Warning: Mailbox changes caused a
desync. You may want to run dsync again: Remote lost mailbox GUID
e9149d0ae4e02d532505000026ca4352 (maybe it was just deleted?)
Synced another_address@my.domain successfully but missing some
changes. Took 3 seconds. Starting retry 1...
The first message seems to point out that the transaction log got
rolled and no more contains the messages from the backup dir, right? I
thought about setting mail_index_log_rotate_min_age to 1hour to
prevent rolling transaction logs too often, but abandoned this thought
and increased the backup interval to once a minute. The warnings still
appear so maybe my thoughts about transactions logs are wrong. The
second message seems less alarming to me.
How does doeveadm backup behave in such situations? Does it directly
fall back to a less efficient way of syncing mails? Does the state
store the information "retry with a full sync" and the next run uses
this mode? To investigate on this I simply measured runtimes an saw
that the second/retry run takes a bit longer (up to about 15 seconds)
to sync the dir.
I'm afraid of losing messages using my approach. Is it safe to always
use doveadm backup -s $state? Simply counting one maildirs files
within the live directory and the backup copy shows a 100 fewer files
within the backup dir although the script runs only since a few days.
For reference, see my backup script below.
Regards
Christian
#!/bin/bash
# * * * * * /root/bin/backup.sh --sync-only
# 12 2 1-7 * * test $(date +\%u) -eq 6 && /root/bin/backup.sh --full
# 12 2 8-31 * * test $(date +\%u) -eq 6 && /root/bin/backup.sh
--differential
# 12 2 * * * test $(date +\%u) -ne 6 && /root/bin/backup.sh
synconly=0
differential=0
fullbackup=0
if [ $# -gt 0 ] ; then
if [ "$1" == "--sync-only" ] ; then
synconly=1
elif [ "$1" == "--differential" ] ; then
differential=1
elif [ "$1" == "--full" ] ; then
fullbackup=1
fi
fi
basedir="/var/vmail/backup"
targetdir="/var/vmail/backup/done"
mailaddresses="one_address@my.domain another_address@my.domain
yet_another@my.domain"
if [ ! -d "$basedir" ] ; then
mkdir -p "$basedir"
chown vmail:vmail "$basedir"
fi
if [ ! -d "$targetdir" ] ; then
mkdir -p "$targetdir"
chown vmail:vmail "$targetdir"
fi
for mailaddr in ${mailaddresses} ; do
#echo "Creating backup for $mailaddr."
domainpart=${mailaddr#*@}
localpart=${mailaddr%%@*}
lockfile="$basedir/$mailaddr.lock"
statefile="$basedir/$mailaddr.state"
backupdir="$domainpart/$localpart/Maildir"
snapshotfile_full="$basedir/$mailaddr.full.snar"
snapshotfile="$basedir/$mailaddr.snar"
backup_basename="$basedir/${mailaddr}_$(date '+%Y%m%d_%H%M%S')"
(
if [ $synconly -eq 1 ] ; then
flock -xn 200
if [ $? -eq 1 ] ; then
# failed to acquire lock. Skip mailbox silently.
exit
fi
fi
# try to acquire exclusive lock for one minute
flock -xw 60 200
if [ $? -eq 1 ] ; then
echo "Failed to acquire write lock within 60 seconds. Skipping
$mailaddr."
exit
fi
retries=0
retval=1
until [ $retval -eq 0 ] || [ $retries -ge 3 ] ; do
let 'retries++'
if [ -f "$statefile" ] ; then
oldstate=$(head -1 "$statefile")
else
oldstate=""
fi
start_time=$(date +%s)
ERROR=$((doveadm backup -u "$mailaddr" -s "$oldstate"
"maildir:$basedir/$backupdir") 2>&1 > "$statefile")
retval=$?
end_time=$(date +%s)
let 'duration=end_time-start_time'
if [ $retval -eq 2 ] ; then
#if [ $retries -gt 1 ] ; then
echo "$ERROR"
echo "Synced $mailaddr successfully but missing some
changes. Took $duration seconds. Starting retry $retries..."
#fi
elif [ $retval -ne 0 ] ; then
echo "$ERROR"
echo "Syncing $mailaddr failed. Return code $retval. Took
$duration seconds. Removing backup directory and starting retry
$retries..."
rm -rf "$basedir/$backupdir"
rm -f "$statefile" "$snapshotfile"
elif [ $retries -gt 1 ] ; then
echo "Successful sync took $duration seconds."
fi
done
# downgrade lock to shared lock
flock -sn 200
[ $synconly -eq 1 ] && exit
if [ $retval -ne 0 ] ; then
echo "Too many retries. Aborting backup of $mailaddr."
exit
fi
cd "$basedir"
if [ $fullbackup -eq 1 ] || [ ! -f "$snapshotfile_full" ] ; then
tar -cpzf "${backup_basename}_full.tar.gz" --level=0 -g
"$snapshotfile_full" "$backupdir"
cp -f "$snapshotfile_full" "$snapshotfile"
else
suffix=""
if [ $differential -eq 1 ] ; then
cp -f "$snapshotfile_full" "$snapshotfile"
suffix="_diff"
fi
tar -cpzf "${backup_basename}${suffix}.tar.gz" -g "$snapshotfile"
"$backupdir"
fi
cd - > /dev/null
mv "${basedir}/"*.tar.gz "$targetdir"
) 200>"$lockfile"
[ $synconly -eq 1 ] && continue
# housekeeping
newest_full=$(ls -1 "${targetdir}/${mailaddr}_"*_full.tar.gz
2>/dev/null | sort | tail -1)
if [ -n "$newest_full" ] ; then
#echo "Cleaning up files older than $newest_full..."
find "$targetdir" -depth -maxdepth 1 -name "${mailaddr}_*" !
-newer "$newest_full" ! -samefile "$newest_full" -printf 'Deleting
%p...\n' -delete
fi
done
