automated backups of my computer

Replies:

Parents:

  • None.
The other day I set up automated backups of the most important
areas of my hard drive. This had been on my todo list for years,
so I'm pretty happy it's done now. (I already used my backup to
restore a trivial file I accidentally nuked yesterday.)

I set up daily, weekly, and monthly backups of my home directory
from one disk to another, and a weekly off-site backup of my home
directory and CVS repository (containing my web site) to my server
at MIT, in case my computer is stolen or lost in a fire or something.

Until now, I had been making backups by making huge tarballs of
various dirs whenever I happened to think about it, but that
recently stopped working for my home directory because the tar
ball exceeded linux's default 2 gig limit on the size of a file.
I'm sure I could get around that limit somehow, but I decided it
would be better to start using rsync anyway.

So I wrote a shell script [1] that copies /home/{gerald,cvsroot}
to un.impressive.net once a week (run from cron on un, using ssh
with .shosts for auth, which I already had set up for my cvs
mirroring stuff), then a few more cron jobs on devo for the local
disk backups:

0   7  1 * * nice rsync -a -q --stats /home/gerald /backup/devo/monthly/home >> misc/mm/devo-monthly-backup-log
30  6  * * 0 nice rsync -a -q --stats /home/gerald /backup/devo/weekly/home >> misc/mm/devo-weekly-backup-log
0   6  * * * nice rsync -a -q --stats /home/gerald /backup/devo/daily/home >> misc/mm/devo-daily-backup-log

(the daily one takes about 10 mins to complete.)

rsync would be much more efficient if I wasn't using mboxes [2]
for my mail; maybe I'll switch to maildirs [3] instead sometime.
(but I don't really care much about efficiency since this stuff
only runs when I'm sleeping.)

[1] http://impressive.net/people/gerald/2000/12/30/backup-devo
[2] http://www.qmail.org/qmail-manual-html/man5/mbox.html
[3] http://www.qmail.org/qmail-manual-html/man5/maildir.html

--
Gerald Oskoboiny <[email protected]>
http://impressive.net/people/gerald/

Re: automated backups of my computer

Replies:

  • None.

Parents:

On Fri, Jan 05, 2001, Gerald Oskoboiny wrote:
> The other day I set up automated backups of the most important
> areas of my hard drive. This had been on my todo list for years,
> so I'm pretty happy it's done now. (I already used my backup to
> restore a trivial file I accidentally nuked yesterday.)

Cool!

[..]
> So I wrote a shell script [1] that copies /home/{gerald,cvsroot}
> to un.impressive.net once a week (run from cron on un, using ssh
> with .shosts for auth, which I already had set up for my cvs
> mirroring stuff), then a few more cron jobs on devo for the local
> disk backups:
[..]

Why didn't you use some backup software such as Amanda[4]?

> rsync would be much more efficient if I wasn't using mboxes [2]
> for my mail; maybe I'll switch to maildirs [3] instead sometime.

I didn't know about the maildir format, and it looks pretty cool. I
think that I am going to try and use that. Mutt understand it. I would
have to do some performance tests though.

Did you investigate about getting Procmail to store mail in such a
format?

What about disk space by the way? One file for each email could result
in quite an increase of disk space used.

> (but I don't really care much about efficiency since this stuff
> only runs when I'm sleeping.)

Using fixed times? :-)

> [1] http://impressive.net/people/gerald/2000/12/30/backup-devo
> [2] http://www.qmail.org/qmail-manual-html/man5/mbox.html
> [3] http://www.qmail.org/qmail-manual-html/man5/maildir.html
 4. http://www.cs.umd.edu/projects/amanda/

--
Hugo Haas <[email protected]> - http://larve.net/people/hugo/
La vraie paresse, c'est de se lever � 6 heures du matin pour avoir plus
longtemps � ne rien faire. -- Tristan Bernard

Re: automated backups of my computer

Replies:

  • None.

Parents:


[]
>rsync would be much more efficient if I wasn't using mboxes [2]

rsync version 2.4.6 is a _significant_ improvement over earlier
versions, Red Hat and Debian packages from base install are the older,
less efficient (even buggy on large number of files).  I haven't
looked for packages of this version, but the build is simple enough
and you would want it on all machines involved.  You (Gerald only, not
necessarily the plural you) can get the binary off of tux.  Source can
be found at [4].

I took a quick look at your script [1] and suggest taking off the z
compress flag as unneeded since you're going through a compressed
tunnel, ssh and are just adding overhead (relaying advise given to me
a while back by Daniel Veillard former coworker and someone who
contributed to latest rsync).

Moving to maildir folders (was Re: automated backups of my computer)

Replies:

Parents:

On Fri, Jan 05, 2001, Gerald Oskoboiny wrote:
> rsync would be much more efficient if I wasn't using mboxes [2]
> for my mail; maybe I'll switch to maildirs [3] instead sometime.
> (but I don't really care much about efficiency since this stuff
> only runs when I'm sleeping.)

After reading all the advantages of the maildir format[3] and the
efficiency compared to the mbox format[2] (when a message is deleted,
you don't need to rewrite the whole folder, no escaping of "^From ..."
lines is necessary, etc), I have been convinced that I should use
maildir folder and I decided to have a look at the maildir support
around and see if I could switch

0. Can my MTA write into a maildir folder?

 Although maildir is a format used by qmail[0], it is implemented in
 other tools because of its advantages. Procmail[1] knows about
 maildir folders, so virtually every MTA can write mail into maildir
 format.

1. How to convert an mbox folder into a maildir folder?

 This is really easy by using mbox2maildir[4].

2. Will MUAs recognize the maildir format?

 Mutt[5] 1.2 can access maildir folders. Just for fun, I had a look
 at Pine[6]'s support. Apparently[7], patches have only been around
 for two years so they haven't been integrated in the source tree
 yet. :-)

 Basically, I don't think that a lot of clients support it. But Mutt
 is enough for me.

3. Can I access my folders via IMAP?

 I am not an IMAP expert, but I guess that the most common server is
 the one from University of Washington[8]. Well, that's the same
 source as Pine, so it doesn't support it.

 Courier-IMAP[9], on the other hand, has been developed for this
 task. It looks pretty cool, except that I couldn't get it to
 understand my maildir folder... I will have to work on this again.

 Anyway, Courier-IMAP doesn't understand mbox folders. The UW one can
 be patched to support both.

The conclusions I reached are:
- maildir folders are far better mbox ones.
- switching from an mbox solution to a maildir one is pretty easy.
- the only drawback is currently with IMAP.

> [2] http://www.qmail.org/qmail-manual-html/man5/mbox.html
> [3] http://www.qmail.org/qmail-manual-html/man5/maildir.html
 0. http://www.qmail.org/
 1. http://www.procmail.org/
 4. http://em.ca/~bruceg/mbox2maildir
 5. http://www.mutt.org/
 6. http://www.washington.edu/pine/
 7. http://x66.deja.com/getdoc.xp?AN=711642053&CONTEXT=981329394.465568006&hitnum=2
 8. http://www.washington.edu/imap/
 9. http://www.inter7.com/courierimap/

--
Hugo Haas <[email protected]> - http://larve.net/people/hugo/
Asleep at the switch? I wasn't asleep, I was drunk! -- Homer J. Simpson

Re: Moving to maildir folders

Replies:

Parents:

Hi,

after reading hugo's mail[1], I decided to switch to maildirs (for various
reasons, including fun, ease to synchronize, testing, a.s.o).

[1] http://impressive.net/archives/fogo/[email protected]

This was rather easy to do.

1) writing a script to run mbox2maildir on all my mboxes
2) set mbox_type=Maildir into ~/.muttrc
(so that it creates maildirs and not mailboxes)
3) adding '/' to procmail rules


Just wanted to add my conclusions to :

> 2. Will MUAs recognize the maildir format?
>   Mutt[5] 1.2 can access maildir folders.

Indeed, it can access maildir folders. Still, maildir support is not as
good as mailbox support. The following are not really bugs, but can be
annoying.


* 1 : when in "open mailbox" mode, pressing "c" again will put
you in "Chdir to" mod. Whatever you do, you'll end up with an error,
most likely "not a mailbox". Anyway, why the hell did you press "c",
heh?


* 2 : default unread maildir
In a mailbox configuration, once you have read some mail in one box, it
is considered as read, and won't show up as having new mails, even if
you have left some unread mail in it. It won't show up as the default
unread mailbox, either.

In a maildir configuration, you have to read *all your new mail*
in a maildir, and if not it will always show up as the default
"next-to-be-read".

Cool behaviour would probably be somewhere between those two. Being able
to put aside some mails you want to read later, and being reminded it
once the rest has been read.

What I do is that I copy all these "I'll read this later" mails into
a special maildir (thus marking them as read). When I have time for
them I process the special directory fifo-style, and remove the mails
when done. The only drawback is that the original mail is not marked as
"replied" (r) in its actual maildir.


--
Olivier

Re: Moving to maildir folders

Replies:

  • None.

Parents:

On Tue, Feb 20, 2001, Olivier Thereaux wrote:
> after reading hugo's mail[1], I decided to switch to maildirs (for various
> reasons, including fun, ease to synchronize, testing, a.s.o).

Excellent. I will have to do this at some point too, as soon as I have
figured out how to make it work with IMAP.

[..]
> Indeed, it can access maildir folders. Still, maildir support is not as
> good as mailbox support. The following are not really bugs, but can be
> annoying.
>
[..]
> * 2 : default unread maildir
> In a mailbox configuration, once you have read some mail in one box, it
> is considered as read, and won't show up as having new mails, even if
> you have left some unread mail in it. It won't show up as the default
> unread mailbox, either.
>
> In a maildir configuration, you have to read *all your new mail*
> in a maildir, and if not it will always show up as the default
> "next-to-be-read".

I have seen something like that. I have reported a bug[0] about
trashed emails, but I wanted to report this behavior too. You should
probably send another message there with your comments. Well, my
problem is that I could not leave a folder until I had read *all* the
emails in it because it would otherwise always show up in the change
folder prompt as a folder with new mail. I should try with a newer
version of Mutt.

Note that I discussed on mutt-dev new mail detection in the past. The
problem is that, even in mbox format, it doesn't work that well: Mutt
uses the folder's timestamp (which is a cheap test) to detect new
mail. The problem is that it doesn't always work, especially over NFS.
This is why I always compile Mutt with the buffy size option which
reads the folder to see if there is new mail.

> Cool behaviour would probably be somewhere between those two. Being able
> to put aside some mails you want to read later, and being reminded it
> once the rest has been read.

I think that this can be achieved with the mark_old option, although I
am not sure about it. I tried to use it, but I wasn't that happy,
although I can't remember why.

 0. http://bugs.guug.de/db/41/412.html

--
Hugo Haas <[email protected]> - http://larve.net/people/hugo/
Asleep at the switch? I wasn't asleep, I was drunk! -- Homer J. Simpson

HURL: fogo mailing list archives, maintained by Gerald Oskoboiny