OpenBSD Journal

Testers needed for OpenBSD online backup system

Contributed by jose on from the backups dept.

Ben writes: "I've just posted this to misc@openbsd.org...

I'm developing an online backup system -- sort of like rsync, but all the data on the server is encrypted and it allows access to old versions and deleted files. It's now ready for other people to try it out:

http://www.fluffy.co.uk/boxbackup/

If you're interested in automatic backup to a central server, then please give it a try and let me know how you get on. I don't think it's quite ready for production use yet, but I have been using it on a number of systems for a while.

If you don't have a suitable server, I may be able to offer an account on my server in exchange for feedback on the system.

I would also appreciate people looking over my use of cryptography, to make sure it is as secure as it should be. There are some programmers notes in the distribution archive which will help to evaluate this.

It's primarily written for OpenBSD, but I have have done initial ports to some other platforms. However, getting it to run well on OpenBSD is my primary concern, hence asking for testers in this forum.

I am releasing it under a modified BSD license, and intend to change this to a more conventional one as soon as I can. (see the web site for details)

Thanks,

Ben"

(Comments are closed)

Comments

  1. By RC () on

    It's really nice to see something like this. I often wondered why all the backup utilities around were still holding on to the old tape-like backup senarios.

    This might make things a lot easier than my selection of scripts (using ssh and rsync). The one thing I'm not sure about is encryption... It uses extra space, and isn't all that necessary when I've already got physcial security. I hope that's easy to disable.

    Comments

    1. By sthen () on

      If you want something without encryption on the server, there's always rdiff-backup .

      Comments

      1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

        And as a complete alternative, Duplicity.

        http://www.nongnu.org/duplicity/

        Although there are big differences between the two systems which may make one more suitable than the other for your application. I'll be writing up a comparison later today.

    2. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      Encryption doesn't add that much of an overhead. The data is compressed before it's encrypted. The main thing which inflates the space used is the relatively small chunks of data being compressed, as there's less data for the compression algorithm to take advantage of.

      However, it's not bad. Backups are generally between 50% and 75% of their space on disc. Depending on your data, of course.

      There is no way to disable it.

      Comments

      1. By Anonymous Coward () on

        im sure you have thought of the fact that all backups becomes useless if the encryption key is lost

        me keeps unenecrypted backups

        Comments

        1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

          I have. The documentation and the configuration scripts remind users repeatedly that they need to back this up.

          However, you only need to conventionally back up one piece of information once, not a huge amount of information at regular intervals.

  2. By submicron () on

    Ben,

    I'm really glad to see someone doing work in this area. This is something that I think a number of us do with homebrew scripts right now. I'll definitely start playing with this over the weekend.

    Comments

    1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      I'm intending to quite rapidly get the rest of the required features in there and shake out any issues/bugs, so feedback is welcomed!

      I have a mailing list for discussions and annoucements, details on the site shortly.

      It's good to see the enthusiasm for this project!

    2. By P. Pruett () ppruett@webengr.com on http://www.cocoavillagepublishing.com/development/

      Here is a HOME BREW example, using pax w/ cpio to accept long file & path names. Usefull for a quic
      backup of must have stuff, like right before trying a remote upgrade. have notes on this at
      http://www.cocoavillagepublishing.com/development/tools/openbsd/tips/upgrading/


      (2 things to watch out for)
      (one - watch out, unlike tar pax will use the / in the path name when restoring unless you use the -s option!!! and you will restore to same place!!!)
      (two - if cut/paste from this post watch out for were line was cut during wrap)


      PAXDATE=`date +%Y%m%d%H%M`

      ### no compression on remote, compress afterwards
      ssh root@remotecomputer "pax -w -x cpio -b 5120 /etc /root /var /home /usr/local /usr/share" | gzip > /storage/remotecomputer.$PAXDATE.pax.gz

      ### no compression on remote, compress afterwards
      ### with -s option to remove root /
      ssh root@remotecomputer "pax -w -x cpio -b 5120 -s ',^//*,,' /root /var /home /usr/local /usr/share" | gzip > /storage/remotecomputer.$PAXDATE.pax.gz

      ### compression on remote
      ssh root@remotecomputer "pax -wz -x cpio -b 5120 /etc /root /var /home /usr/local /usr/share" > /storage/remotecomputer.$PAXDATE.pax.gz

      ### compression on remote
      ### with -s option to remove leading /
      ssh root@remotecomputer "pax -wz -x cpio -b 5120 -s ',^//*,,' /etc /root /var /home /usr/local /usr/share" > /storage/remotecomputer.$PAXDATE.pax.gz

  3. By deem0n () dima@yasp.com on http://www.yasp.com/

    http://www.bacula.org/
    it hink it is good idea to add security features to that one, instead of inventing something again.

    Comments

    1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      From home page: "However, if you are new to Unix systems or do not have offsetting experience with a sophisticated backup package, we do not recommend using Bacula as it is much more difficult to setup and use than tar or dump."

      Box Backup is designed to be really easy to install and use, yet secure.

      Bacula I think solves different problems. There is room for more than one solution -- I wrote this because nothing fitted what I needed to achieve.

    2. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      Take a look at

      http://www.fluffy.co.uk/boxbackup/comparison.html

      (posted since you wrote your comment)

  4. By andre () on

    I hope you can switch to a free license as soon as possible.

    Comments

    1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      Absolutely. I do not want this clause, but given the fact that this project is my day job, funded by myself, I do not feel it prudent to switch to one quite yet.

  5. By Anonymous Coward () on

    http://www.asiaosc.org/article_26.html

    using some servers

    Comments

    2. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      Actually, now I look at it again, the backups don't appear to be encrypted on the internet server.

      It's only the connection to the server which is encrypted.

  6. By medium sized potato () on

    Ive been working on a backup system too, except the server end is fairly simple, and is designed to work with win32 clients.

    Basically they send in a login (non-system login), then start sending the files. The files and dirs get thrown as they are into /backupdir/dd_mm_yy/username/.

    The /backupdir gets shared read-only over samba, so that people can pick out the files they want.

    Comments

    1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      The client could be ported to Win32 very easily, and a graphical UI should be no problem either.

      How far have you got? If you'd like to think about porting this, then email me. (says he, being very optimistic)

      Comments

      1. By medium sized potato () on

        Well ive written a client backend thingy which loads which dirs/files to backup from a normal text file and throws them at the server.

        I have only got the GUI ( grrrr, I loath writing win32 GUI's ) to write and its a goer.. I have a number of people with dollars who want it done, so I need to.

        But there are things I want to improve on, so I might have a peek at your stuff and see if its worth just writing a win32 client backend and putting the existing GUI in front of it.

        Dont hold your breath though :)

        Comments

        1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

          Sounds good.

          If you do, I'd be interested in any comments on code quality you might have. (even if you don't use it)

  7. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

    I've just updated the web site with a comparison to other backup systems, and a fuller description of the system.

  8. By Adrien Kunysz () a_kunysz@y!.com on http://krunch.servebeer.com/~krunch/

    From the site:
    all data is encrypted
    Only changes within files are sent to the server, just like rsync

    If you encrypt a file, change the original then encrypt it again, the two encrypted files will be completly different. Uploading changes isn't that much efficient. Or did I miss something ?

    Comments

    1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      You work out changes in the unencrypted version, and encrypt those changes.

      In practise, this means you split the file up into many blocks, each of which is individually compressed and encrypted. Checksums are stored and processed separately, so the client doesn't need to download and decrypt them to do the rsync-like comparison.

      It's surprisingly efficient -- almost as good as rsync itself.

  9. By Anonymous Coward () on

    If I have an alternative between software X & Y, and Y is written in C++, I try very hard to run X instead. Why? Because while C++ is supposed to be a standard and supposed to be portable, it's still not quite there yet, and one more huge headache for me the sysadmin in a multi-platform environment. Given that there are viable remote backup solutions that are written in portable C (rsync being a great one though it lacks some of the security features), I'm simply not going to look at your C++ program. Nothing personal - it might be a great effort and I applaud you for soliciting feedback - but it will take me a lot of pain and anguish to get the thing to comple correctly on my different OSs, and that's already a major strike against you.

    Please find a way to rewrite it into thoughtfully portable C if you want to really be multiplatform.

    Comments

    1. By Ian McWilliam () i dot mcwilliam at uws dot edu dot au on mailto:i dot mcwilliam at uws dot edu dot au

      Oh, Please...... How many of us working on ports in the PORTS tree coome across the "highly Portable" C code. There is just as much work porting "C" code as their is any other lanugage across different OSes. What's usually more of a headache is sysadmins who lack skills and complain about it.

    2. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      I wonder if you could be a bit more specific about which features of C++ cause portability issues?

      I'm not convinced that C++ would make any program less portable. It's how you use the underlying OS which causes portability problems, and since C++ uses the C API to do this you have exactly the same issues with C and C++.

      I could see that there might be issues revolving around some newer C++ features or with STL implementations, but these are things of the past. If your compiler installation is less than four years old, I would be surprised if you had problems.

      There are distint advantages in using C++. Done well, it can decrease development time and increase correctness and reliability.

      So, what things have caused you problems?

      Comments

      1. By Anonymous Coward () on

        It's primarily the libraries and also the compiler strictness. GCC in particular has gone through a lot of C++ growing pains. It has rather often been the case that good practice at time x becomes a syntax or type error at time (x+1yr).

        This is a C++ compiler & library maturity issue, which in turn lags the spec maturity and all this reflects the spec's complexity. Maybe in a few years this will be a distant memory of pain long passed, like the move from K&R-with-extensions to the original ANSI C.

        But today, my comment stands. Arbitrary C++ programs off the net are far less likely to compile out of the box than arbitrary C programs. By a very noticeable margin. And the best intentions of the authors might not be able to solve the problem because the compile environment is what's causing the portability problems.

        I could also go on for ages about why C++ is bad as a language, but that's a religious debate that belongs in a better forum than this.

        Comments

        1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

          OK, I see what you're saying, and perhaps on some platforms with older compilers and libraries this might be a problem. (Although nothing un-fixable.)

          For this particular project, which originally targetting just OpenBSD, the use of C++ gives a sensible compromise between ease of authoring and ease of porting.

          I'm not looking to conquer the world, just solve a problem for myself. Being able to solve other people's problems on other platforms is just a nice bonus.

          Anyway, if it was listed as compiling on all the platforms you were interested in, I hope you'd take a look at a C++ program.

  10. By zerash () zerash@metawire.org on http://www.metawire.org/

    I recently implemented Box Backup at Metawire.org, and I must say it runs great. I've been using rsync, with a ton of scripts for my backup solution till now - which, yes, did work but often would be too slow to be a good data backup implementation at all. With 3000 users and data being modified on a second-per-second basis, you opt for anything out there that might increase the speed or performance.

    Box Backup definatelly does that and more.

    <3

    -zerash

    P.S. Ben did not pay me to write this :)

    Comments

    1. By Ben Summers () on http://www.fluffy.co.uk/boxbackup/

      Thanks for your kind words! I'm glad it's being useful.

      Make sure you pick up the latest version.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]