Репозитории ALT
Группа :: Архивирование/Резервное копирование
Пакет: bontmia
навигация по пакетам
Главная Изменения Спек Патчи Исходники Загрузить Gear Bugs and FR Repocop
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html lang="en">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<META NAME="description" CONTENT="Secure efficient network backup solution.">
<META NAME="keywords" CONTENT="backup, security, changes, incremental,
delta, bontmia, linux, network, bandwidth, tape-rotation, tape, disk,
hard-link, gnu, gpl">
<title>Bontmia - (Backup Over Network To Multiple Incremental Archives)</title>
<link rel="stylesheet"
type="text/css"
title="Styles"
href="../styles/stil.css">
</head>
<body>
<h2>Bontmia (Backup Over Network To Multiple Incremental Archives)</h2>
<p>
Bontmia was written by John Enok Vollestad in april 2003 to merge the
functionality of glastree and rsync in one application with a more
flexible selection of long term storage. It has later gone through
some changes to enhance usability
</p>
<img src="backup.gif" alt="">
<h3>Why tapes is not a good idea anymore</h3>
<p>
The only reason for using tapes instead of backup to disks over
network is if no network or extra machine is available.
</p>
<dl compact>
<dt>Cost
<dd>Tapes cost more than disks and require a tape-drive and
possibly also a tape robot.
<dt>Reliability
<dd>Disks have a higher MTBF than tapes and they require a tape
drive of the same type. Disks can increase
MTBF by being connected together to form RAID. Tapes can
not. Incremental backup on a disk or a RAID volume does not
change the MTBF while incremental backup on tape drastically
lower the MTBF. Sometimes tapes written in one drive
is difficult to read in another drive. This is a
<i>silent</i> error if you do not test the tapes no and then.
No such thing on file systems like <a
href="http://www.namesys.com/">ReiserFS</a>.
<dt>Location
<dd>Tapes is removable media and thereby avoiding the risk of
localized disasters. A network connection does the same.
<dt>Time
<dd>Copying just the changes gives a fast and efficient backup.
Tape drives are very slow. Often slower than the network.
Remote backup will therefore usually go faster to disk.
Faster backup and efficient storage means one can do backup
more often. Once every hour is usually not a problem. This
also means rapid restore which is even more important.
<dt>Space
<dd>Tapes split the data needlessly since they can not be used
with RAID to form bigger volumes as disks can. Using
hard-links on a file system gives very good storage
efficiency and availability as ordinary file systems with
random access.
</dl>
<p>
There is a lot of remote backup software out there. I wrote Bontmia
since I could not find one that did all that I wanted.
</p>
<h3>Bontmia Functionality</h3>
<ul>
<li> Each backup is a complete "snapshot" of the backed up
directories.
<li> Files that is not changed since the last backup does not take up
extra space and is not transferred over the network.
<li> Preserves a configurable selection of backups in a similar manner
as tape rotation schemes.
<li> Remote backup is done encrypted over ssh for security reasons.
<li> Each backup is stored with absolute path.
<li> It is possible to limit the bandwidth to avoid affecting
production systems.
<li> Support copying files, directories, device files, named pipes,
symlinks and hard-links.
<li> Preserves user id, group id and permissions for all types of
files.
</ul>
<p>
<b>NB</b> File in the backup directores should <b>NOT</b> be changed.
It is outside the scope of Bontmia to protect against this.
You can mount the backup device read-only to handle this.
</p>
<p>
<b>NB</b> Please note that it does <b>not</b>, just as any other
program I know, in a sane manner support copying files that
is currently changing. So for databases one should
<b>not</b> copy the databasefiles but rather use the builtin
functions to extract a backup and then backup those files
with Bontmia.
</p>
<p>
<b>NB</b> When copying files from a remote host, please note that user
id and group id is not necessary the same user so keep the
backup directory unaccessable from ordinary users or
maintain synchronization or user ids and passwords between
the hosts. For the latter. PAM and LDAP is an exellent
choice for small to mid-range sites.
</p>
<p>
<b>NB</b> Bontmia uses rsync and rsync in its current form uses quite
a lot of RAM when used on large directory structures. It
might speed things up considerably to divide the backup into
several smaller ones.
</p>
<h3>Archive structure</h3>
<p>
The archives is placed in a directory structure like this:
</p>
<pre>
2003/05/06/04:00/
2003/05/07/04:00/
2003/05/08/04:00/
2003/05/08/05:00/
2003/05/08/06:44/
</pre>
<p>
which is YYYY/MM/DD/HH:MM/ when the backup was issued. Since the
granularity of the backup archives is one minute there is not possible
to run two different backups within the same minute.
</p>
<p>
Under these directories the archived files and directories is stored
within a directory named the hostname backed up and absolute
path. Like this:
</p>
<pre>
2003/05/06/04:00/foo:/home/jev
/bar:/home/jev
/baz:/home/jev
2003/05/07/04:00/foo:/home/jev
/bar:/home/jev
/baz:/home/jev
</pre>
<h3>Operation</h3>
<p>
The following shows an example of how to do backup once every day and
keep the last 7 days, 4 weeks, 12 monthly and 2 years. The hostname
is changed.
</p>
<pre>
$ bontmia --dest ./backup --rotation \
0minutes0hours7days4weeks12month2years \
foo.bar.com:/home/jev \
foo.bar.com:/etc \
foo.bar.com:/usr/local \
foo.bar.com:/var
</pre>
When this is run it outputs the following on my system (the hostname
is changed). Since the computer have not been on all the time not all
the backups have been run but the last x backups is saved for each
filter. Which filter that is active for each backup is shown. The
one removed is not longer filtered to be saved.
<pre>
Making a hard-link replication of the last backup
(/backup/2003/09/20/00:00)
Backing up by modifying the replication
foo.bar.com:/home/jev
foo.bar.com:/etc
foo.bar.com:/usr/local
foo.bar.com:/var
Deletes files that should not be in the latest snapshot
Moving the complete backup into the backup archive
(/backup/unfinished_backup -> /backup/
Calculates which backups to save
Saving /backup/2003/06/29/19:07 by filters: month
Saving /backup/2003/07/20/10:00 by filters: month
Saving /backup/2003/08/26/22:30 by filters: weeks month
Saving /backup/2003/09/07/00:00 by filters: weeks
Removing /backup/2003/09/13/00:00
Saving /backup/2003/09/14/00:00 by filters: days weeks
Saving /backup/2003/09/15/00:00 by filters: days
Saving /backup/2003/09/16/00:00 by filters: days
Saving /backup/2003/09/17/00:00 by filters: days
Saving /backup/2003/09/18/00:00 by filters: days
Saving /backup/2003/09/19/00:00 by filters: days
Saving /backup/2003/09/20/00:00 by filters: days weeks month years
</pre>
<p>
As one can see the last 7 days is saved, the last 4 weeks is saved and
the last 3 month is saved. The backup have only run for the last 3
month and therefore there is no more month backups. Similar for the
year backup.
</p>
<p>
<b>NB!</b> If you want to make copies of several directories then do
<b>not</b> run Bontmia multiple times against the same '--dest' but
instead list each source directory as arguments on one command or the
incremental storage becomes impossible to maintain and one end up
copying all the data all the time.
</p>
<!--
<p>
You might also want to look at the short <a
href="documentation.shtml">documentation</a>.
</p>
-->
<h3>How the rotation work</h3>
A filter will save one backup per unit for the last x units. A unite
can be minute, hour, day, week, month or year. Where x is specified
by the user. It is the last backup within the unit that is saved.
<br><br><i>Example 1</i><br>
In the following example the filter A have a unite size indicated by
the separation between the |'s. The filter A is saving the last 4
backups.
<pre>
backups b b b b b b b b b b b
filter A | | s |s | s| | s|
</pre>
<i>Example 2</i><br>
This example shows how different filters work together. Filter A
saves the last 4 backups, filter B saves the last 3 and filter C saves
the last 2.
<pre>
backups b b b b b b b b b b b b b b b b
filter A | | | | | | | | | | | | | | | | | | | | | | | | | |s|s|s| |s|
filter B | | |s | s| s| s|
filter C | s | s|
------------------------------------------------------------------------
To save s s s s s s s
</pre>
<h3>Short FAQ</h3>
<dl compact class=faq>
<dt>Why is this written as a shell script and not in language X?
<dd>This program relies heavily on rsync. It would be meaningless
to reimplement rsync. It works ok.
<dt>How stable is this?
<dd>I have been using it at a couple of sites since late April 2003
with minor changes and it have worked for me but I do not take
any responsibility for loss of data.
<dt>When I try to copy the destination of a symbolic link with '/.'
I get a long list of error messages from rsync.
<dd>Rsync does not support such notation and thereby neither do
bontmia.
<dt>How do I avoid typing in the password each time the backup is
run?
<dd>This is documented on the ssh client and servers manual pages.
On the localhost, which is the one you store the bakcup on, you
have to generate a private/public keypair and append the public
to the authorized hosts list on the remote computer. For
OpenSSH this is done by append the public key file
~/.ssh/id_dsa.pub from the local computer onto the
~/.ssh/authorized_keys file on the remote computer where local
computer means the computer one stores the backup on and remote
computer means the computer one backs up.
</dl>
<h3>Implementation</h3>
<p>
It is a single small shell script using <i>cp -lR</i> and <i>rsync</i>
to maintain incremental backups and some additional shell-code to
store old backups for a selected time in the same manner as ordinary
tape rotation schemes. Files that does not change since the last
backup become a hard-link to the version in the last file to save disk
space. Making hard-links also increase the speed of the backup.
Making a hard-link of a 2GB file is done equally fast as for a file of
1KB and thus avoiding transfer of files accross the network saves a
lot of time when changes is rare.
</p>
<h3>Changes</h3>
<p>
<dl compact class=changes>
<dt> 0.14 (Latest release)
<dd> <em>Added feature</em>
<ul>
<li>Added support for specifying the directory for temporary files.
</ul>
<dt> 0.13
<dd> <em>Bug fix</em>
<ul>
<li> Jared Rhine (jared AT wordzoo.com) commented on difficulty doing
local backups. Examination showed a embarasing bug. This has been
fixed. He also suggested using the PAGER environment variable when
calling a pager. This has been done. Some tests for GNU compliancy
were needed has been added by suggestion from the same.
<li> Lantiainen Pasi (baze AT cs.tut.fi) commented that the
"--rotation" option were mentioned as "--rotate" in the examples.
This has been fixed.
</ul>
<dt> 0.12
<dd> <em>Some changes in parameters and code rewrite</em>
<ul>
<li> Alex Landa (brooknet AT imap.cc) come up with a patch for
better supported options to sort. ('-k' instead of '--key=').
<li> Thomas Sigdestad (tsi AT enonic.com) wanted to change the parameters to
the way they are now.
</ul>
<dt> 0.11
<dd> <em>This is a clean-up of how bontmia does things.</em>
<ul>
<li> A snapshot is now created outside the archieve directory. This
were done since aborted or incomplete backups made the use of
hard-links more difficult and incomplete snapshots should not
be present in the achieve directory. Such incomplete snapshots
is removed when starting the next backup.
<li> The hostname is now also added on local backups and remote
hostnames is resolved to the long version. This is done to
make it more conform.<br>
<strong>NB</strong> this means the local backup changes name
and all the local data will be copied again completely. One
can avoid this by changing the last backup a little. Create a
directory with the name of the localhost, with a ':' at the end
of the name, in the backup directory where the root of the
previous backup is placed. You can run the command hostname to
get this name. Then move the local backup into this directory.
<li> Some changes in the textual output.
<li> Started the change log.
</ul>
</dl>
<h3>Get Bontmia</h3>
<p>
The program itself is a single script and is available for <a
href="bontmia-0.14.tar.gz">download</a>. To run it you have to have a
Posix shell like Bash (sh on Solaris does not work, I will make
Bontmia work with it when time), GNU date, GNU cp and ssh for remote
access.
</p>
<h3>Contact</h3>
<p>
If you have any comment or suggestion you might send them to <em>john.enok@vollestad.no</em> or visit my <a
href="http://folk.uio.no/johnen/">webpage</a>.
</p>
<p>
You might also rate this program by visiting the corresponding
<a href="http://freshmeat.net/projects/bontmia">FreshMeat</a> page.
</p>
<hr>
<a href="http://validator.w3.org/check/referer"><img
align="right" border="0"
src="http://www.w3.org/Icons/valid-html40"
alt="Valid HTML 4.0!" height="31" width="88"></a>
Last updated 2004-07-26.
</body>
</html>