Backup Redundancy Etcetera

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Backup Redundancy Etcetera

David Walker-16
Hey.

Currently my backup regime is woeful.
I have years worth of work on a Windows machine and some stuff
scattered across OpenBSD machines.

I'm thinking of building a machine (the file server) to provide some
backup and central storage.
I'll probably try and get my head around softraid for redundancy
redundancy on the file server and I'm looking at these ideas for data
transfer ...

Being able to push data to the server manually from Windows and other
operating systems over the network. SSH or IPsec or similar is my idea
here.

Having some mechanism where I can pull onto the server from the
clients at selected times or poll the machines for changes and update
the server or something.
I have no experience here and I'm thinking about acronyms like NFS,
rsync, etcetera.

This is for a small number of machines and low rate data changes but
if I can find something that's in base, scalable, robust, secure,
simple, quick ...
:]

Please give me some recommended acronyms, man pages, etcetera.

Best wishes.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Nico Kadel-Garcia-2
On Mon, Feb 6, 2012 at 4:10 AM, David Walker <[hidden email]> wrote:
> Hey.
>
> Currently my backup regime is woeful.
> I have years worth of work on a Windows machine and some stuff
> scattered across OpenBSD machines.

Uh-oh.

> I'm thinking of building a machine (the file server) to provide some
> backup and central storage.
> I'll probably try and get my head around softraid for redundancy
> redundancy on the file server and I'm looking at these ideas for data
> transfer ...
>
> Being able to push data to the server manually from Windows and other
> operating systems over the network. SSH or IPsec or similar is my idea
> here.

Pull through CIFS mounting, don't try to pull over SSH. (See the old
thread at http://fixunix.com/ssh/73787-mcafee-cygwin-ssh.html .)

Also, running rsync on a Windows box is..... fragile, due to the way
Windows locks processes when they try to "open" a file that is "busy".
It makes rsync very fragile because the set of such files is almost
impossible to pre-identify and exclude, and some of them are really
important, such as Outlook backups.

That said, there's a very useful toolkit called "rsnapshot" that I've
been using for years which is very flexible and can easily be targeted
at CIFS shares. I've been using it on numerous UNIX and Linux systems,
including OpenBSD, quite effectively.

> Having some mechanism where I can pull onto the server from the
> clients at selected times or poll the machines for changes and update
> the server or something.
> I have no experience here and I'm thinking about acronyms like NFS,
> rsync, etcetera.

The one you want is "CIFS", where the BSD system can mount authorized
shares from the Windows boxes using the Samba software.
> This is for a small number of machines and low rate data changes but
> if I can find something that's in base, scalable, robust, secure,
> simple, quick ...
> :]
>
> Please give me some recommended acronyms, man pages, etcetera.
>
> Best wishes.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

David Coppa
On Mon, Feb 6, 2012 at 3:43 PM, Nico Kadel-Garcia <[hidden email]> wrote:

> The one you want is "CIFS", where the BSD system can mount authorized
> shares from the Windows boxes using the Samba software.

OpenBSD does NOT have cifs support

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

James Shupe-4
In reply to this post by David Walker-16
On 02/06/2012 03:10 AM, David Walker wrote:
> Hey.
>
> Currently my backup regime is woeful.
> I have years worth of work on a Windows machine and some stuff
> scattered across OpenBSD machines.
>

You might want to look at Bacula.

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

David Walker-16
In reply to this post by David Coppa
Thanks for the replies.

I should have stated I'm after something I can understand at a block level.
There are only a few datapoints I care about:
* the /etc from a few internal and external OpenBSD machines.
* a few other *conf* areas like /var/named and so on from external machines.
* either /var/www/virtuals from an external machine or from the
Windows machine they were built on.
* some personal data from the Windows machine.

All that stuff changes little (especially the OpenBSD machines).
If I lose a day or so from the Windows machine that's fine.
So simple is good.

I've read through the ideas and something like dump looks suitable.

     dump - filesystem backup

     -f file
             Write the backup to file; file may be ...
             ... an ordinary
             file ...

This suggests I can mount a remote partition via NFS and dump to a file there.
Is this correct?
Can I do this via SSH also?

The only other question mark is doing something similar for the
internal Windows machine.
I could do this manually via ftp but I suspect that will result in it
happening far too little.
As far as I understand it, Microsoft supply an NFS client via the
resource kit and it looks easy to "at" and script as long as it's
interoperable and Microsoft read the RFCs ...

Best wishes.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Anonymous-3
In reply to this post by David Walker-16
> Hey.

Yo.

> Currently my backup regime is woeful.
> I have years worth of work on a Windows machine
                                  ^^^^^^^

> and some stuff
> scattered across OpenBSD machines.
>
> I'm thinking of building a machine (the file server) to provide some
> backup and central storage.

Solaris
ZFS

> Being able to push data to the server manually from Windows and other
> operating systems over the network. SSH or IPsec or similar is my idea
> here.

Windows is a weakspot since it is so bad and has few standard tools. You
could probably script Filezilla to SSH what you want to the file server.
Everybody else can simply mount NFS shares, dead easy with ZFS. Or you could
rsync from non-Windows systems to Solaris.

>
> Having some mechanism where I can pull onto the server from the
> clients at selected times or poll the machines for changes and update
> the server or something.

You can script cron jobs to rsync from everywhere but on Windows.

> I have no experience here and I'm thinking about acronyms like NFS,
> rsync, etcetera.

NFS is better for sharing in real time. For backups rsync is hard to beat
but Windows is a weak point as mentioned by other posters.

>
> This is for a small number of machines and low rate data changes but
> if I can find something that's in base, scalable, robust, secure,
> simple, quick ...

Solaris
ZFS

> Please give me some recommended acronyms, man pages, etcetera.

PLEASE check the Solaris HCL and possible [hidden email] before
building a file server. If you pick the wrong components, ZFS will hurt you
badly. If you pick the right components you will be so happy.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Stuart Henderson
In reply to this post by James Shupe-4
On 2012-02-06, James Shupe <[hidden email]> wrote:
> On 02/06/2012 03:10 AM, David Walker wrote:
>> Hey.
>>
>> Currently my backup regime is woeful.
>> I have years worth of work on a Windows machine and some stuff
>> scattered across OpenBSD machines.
>>
>
> You might want to look at Bacula.

Yes, Bacula works well for Windows, it has a proper client which runs
as a Windows server and can do VSS. Initial config of Bacula is not very
fun, but once that's done you don't have to touch it too much.
webacula is also in ports which is a nice web UI, especially good
for status displays.

There is also backuppc which can do backups using samba as a client,
some people might prefer this, personally I don't.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Nico Kadel-Garcia-2
In reply to this post by David Coppa
On Mon, Feb 6, 2012 at 10:21 AM, David Coppa <[hidden email]> wrote:
> On Mon, Feb 6, 2012 at 3:43 PM, Nico Kadel-Garcia <[hidden email]> wrote:
>
>> The one you want is "CIFS", where the BSD system can mount authorized
>> shares from the Windows boxes using the Samba software.
>
> OpenBSD does NOT have cifs support

Then what do you call Sharity?

    http://openports.se/net/sharity-light

It might not be a built-in, but it works quite well according to the
various Google reports.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

David Walker-16
In reply to this post by David Walker-16
On 07/02/2012, Nico Kadel-Garcia <[hidden email]> wrote:
> On Mon, Feb 6, 2012 at 4:10 AM, David Walker <[hidden email]>
> wrote:
>>
>> Currently my backup regime is woeful.
>> I have years worth of work on a Windows machine and some stuff
>> scattered across OpenBSD machines.
>
> Uh-oh.

I know.
I do have "hard" copies of some stuff (drives on shelves, etcetera)
but I need to "cloud" it a little more and in the process get more
methodical (instead of me forgetting).
Fortunately I have no problem losing any of these machines and
starting from scratch - I don't need drive images or anything, the
data I care about is in a few specific areas.
For instance the web server, I mainly care about the web sites of
which I have multiple copies.
I also have a copy of the Apache *conf and I probably have a copy of
the /etc changes (rc.conf.local, pf.conf, so on).
In a worst case I can re-install from scratch, adjust /etc and copy
Apache *conf (or re-write them in half an hour) - all that's not
practically rebuildable is the websites themselves.

Anonymous <cripto () ecn ! org> wrote:
> Solaris
> ZFS

I've heard of it (ZFS) but here's the thing, I struggle enough keeping
up with Wndows and OpenBSD I don't want to put another system into the
mix.

> Being able to push data to the server manually from Windows and other
> operating systems over the network. SSH or IPsec or similar is my idea
> here.

> Windows is a weakspot since it is so bad and has few standard tools.

Especially open protocols and secure.
You either accept and embrace Active Directory or install third party
software or stay simple.
Fortunately the Windows machine is internal so insecure is okay.

> You
> could probably script Filezilla to SSH what you want to the file server.

Good idea.
I'll probably end up either installing the Microsoft NFS client and
scripting that or use the bog standard ftp client and script that.

> You can script cron jobs to rsync from everywhere but on Windows.
> NFS is better for sharing in real time. For backups rsync is hard to beat
> but Windows is a weak point as mentioned by other posters.

I'm looking at that now.
Part of the reason I want to use base is so that the curve in getting
a machine back up is easy.
It's kind of what I was looking for but the overhead probably isn't
worth it in my situation.

Again thanks for all the replies (including off-list).
Again I only want to backup data (which is really limited to the
Windows machine) and configuration information (which is easily
quantifiable and changes infrequently) - simple is probably best. The
scenario is so simple that installing software is possibly creating
more difficulty.

I'll try scripting NFS maybe in combination with dump on the OpenBSD
machines and see how that goes.

Best wishes.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

David Coppa
In reply to this post by Nico Kadel-Garcia-2
On Tue, Feb 7, 2012 at 6:42 AM, Nico Kadel-Garcia <[hidden email]> wrote:
> On Mon, Feb 6, 2012 at 10:21 AM, David Coppa <[hidden email]> wrote:
>> On Mon, Feb 6, 2012 at 3:43 PM, Nico Kadel-Garcia <[hidden email]>
wrote:
>>
>>> The one you want is "CIFS", where the BSD system can mount authorized
>>> shares from the Windows boxes using the Samba software.
>>
>> OpenBSD does NOT have cifs support
>
> Then what do you call Sharity?

It's userland implementation.

>    http://openports.se/net/sharity-light
>
> It might not be a built-in, but it works quite well according to the
> various Google reports.

Do not trust google: sharity-light is a crap

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

James Shupe-4
In reply to this post by David Walker-16
> I'll try scripting NFS maybe in combination with dump on the OpenBSD
> machines and see how that goes.
>
> Best wishes.
>

Seriously, look at Bacula. It'll do a better job and be less headache.

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Anonymous Remailer (austria)
In reply to this post by David Walker-16
> Anonymous <cripto () ecn ! org> wrote:
> > Solaris
> > ZFS
>
> I've heard of it (ZFS) but here's the thing, I struggle enough keeping
> up with Wndows and OpenBSD I don't want to put another system into the
> mix.

Understood. Unfortunately or fortunately however you look at it OpenBSD
doesn't have ZFS. But FreeBSD does. That could be another option with less
of a learning curve than Solaris which admittedly is steep. Another thing to
consider is a prebuilt NAS appliance based on FreeBSD or OpenSolaris. There
are numerous ones out check distrowatch.com

What ZFS does for you aside from offering pretty high quality software RAID
and other redundancy/protection from data loss is give you really nice
management features like being able to do quotas and resize filesystems and
compress (and with Solaris 11 even encrypt them) all from one central
management interface instead of external or add-on tools. It's one stop
shopping. It also makes NFS and SAMBA less painful since you don't have to
play around with the normal share tables and portmapper stuff (not THAT big
of a deal but not zero) you can just turn features on or off at the ZFS
filesystem level. It's really ideal for a backup or NAS appliance. Again you
must have known good hardware from the disks to the backplane to the RAM or
ZFS will ruin your week or even your whole month. When it works, it
works. When it doesn't, oh shit.

> > You
> > could probably script Filezilla to SSH what you want to the file server.
>
> Good idea.
> I'll probably end up either installing the Microsoft NFS client and
> scripting that or use the bog standard ftp client and script that.

The problem is the M/S NFS client only works on certain versions of Windows
and not others. Even on the versions it is supposed to work on it doesn't
always work. I have an XP Pro box that SFU refuses to install on.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Bentley, Dain
In reply to this post by David Walker-16
I second Bacula. It runs on pretty much any OS and has tons of options and is
very configurable. You could run it on an OpenBSD server and back up you
windows and OpenBSD clients. If you have enough disk space back up your
clients to disk and migrate to tape for offsite. The windows client is also
stable and you can be very glandular with you config.

Regards,
Dain Bentley

-----Original Message-----
From: Anonymous Remailer (austria) [[hidden email]]
Received: Tuesday, 07 Feb 2012, 4:01am
To: [hidden email] [[hidden email]]
Subject: Re: Backup Redundancy Etcetera

> Anonymous <cripto () ecn ! org> wrote:
> > Solaris
> > ZFS
>
> I've heard of it (ZFS) but here's the thing, I struggle enough keeping
> up with Wndows and OpenBSD I don't want to put another system into the
> mix.

Understood. Unfortunately or fortunately however you look at it OpenBSD
doesn't have ZFS. But FreeBSD does. That could be another option with less
of a learning curve than Solaris which admittedly is steep. Another thing to
consider is a prebuilt NAS appliance based on FreeBSD or OpenSolaris. There
are numerous ones out check distrowatch.com

What ZFS does for you aside from offering pretty high quality software RAID
and other redundancy/protection from data loss is give you really nice
management features like being able to do quotas and resize filesystems and
compress (and with Solaris 11 even encrypt them) all from one central
management interface instead of external or add-on tools. It's one stop
shopping. It also makes NFS and SAMBA less painful since you don't have to
play around with the normal share tables and portmapper stuff (not THAT big
of a deal but not zero) you can just turn features on or off at the ZFS
filesystem level. It's really ideal for a backup or NAS appliance. Again you
must have known good hardware from the disks to the backplane to the RAM or
ZFS will ruin your week or even your whole month. When it works, it
works. When it doesn't, oh shit.

> > You
> > could probably script Filezilla to SSH what you want to the file server.
>
> Good idea.
> I'll probably end up either installing the Microsoft NFS client and
> scripting that or use the bog standard ftp client and script that.

The problem is the M/S NFS client only works on certain versions of Windows
and not others. Even on the versions it is supposed to work on it doesn't
always work. I have an XP Pro box that SFU refuses to install on.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Bentley, Dain
In reply to this post by David Walker-16
Damn auto-correct



Regards,

Dain Bentley



-----Original Message-----

From: Josh Grosse [[hidden email]]

Received: Tuesday, 07 Feb 2012, 10:22am

To: Bentley, Dain [[hidden email]]

Subject: Re: Backup Redundancy Etcetera



"Bentley, Dain" <[hidden email]> wrote:



>........and you can be very glandular with you config.



You have to hate auto correction software.  :)

--

Sent from my phone.  Please excuse any idiotic automated word choices.  It wasn't me.  Honest.


Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Aaron Mason
On Tue, Feb 7, 2012 at 10:07 AM, Anonymous <[hidden email]> wrote:
> PLEASE check the Solaris HCL and possible [hidden email] before
> building a file server. If you pick the wrong components, ZFS will hurt you
> badly. If you pick the right components you will be so happy.
>

This I know well - I was left with a file server that maxed out at
12MB/sec.  Oh, and a dead hard drive which Seagate were more than
happy to replace.

There is a prebuilt appliance by the name of EON - it was based on
OpenSolaris before Oracle killed it, now it's based on
OpenIndiana/Illumos.  There is a very good starting guide to get you
on your way, with CIFS shares and the like.  Instructions are quite
similar to do NFS exports, even iSCSI if you're that way inclined.

But as Mr. Anon says, choose your hardware carefully.  Getting it
wrong can be disastrous.  My EON-based file server is a Core2Duo 6400
w/ 4GB RAM on a Gigabyte G41MT-ES2L and 4 2TB WD Green drives which
collectively pull 177MB/sec, though the onboard Realtek NIC maxes out
at 88MB/sec it still beats the shtick out of the 12MB/sec I was
getting before.

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

bofh-6
On Tue, Feb 7, 2012 at 11:57 PM, Aaron Mason <[hidden email]>
wrote:
> But as Mr. Anon says, choose your hardware carefully.  Getting it
> wrong can be disastrous.  My EON-based file server is a Core2Duo 6400
> w/ 4GB RAM on a Gigabyte G41MT-ES2L and 4 2TB WD Green drives which
> collectively pull 177MB/sec, though the onboard Realtek NIC maxes out
> at 88MB/sec it still beats the shtick out of the 12MB/sec I was
> getting before.

I would avoid "green" drives like the plague.  Check out the SMART
status on them and look at the drive park statistic among others.
Look at how high the number is, versus what the life time recommended
number...


--
http://www.glumbert.com/media/shift
http://www.youtube.com/watch?v=tGvHNNOLnCk
"This officer's men seem to follow him merely out of idle curiosity."
-- Sandhurst officer cadet evaluation.
"Securing an environment of Windows platforms from abuse - external or
internal - is akin to trying to install sprinklers in a fireworks
factory where smoking on the job is permitted."  -- Gene Spafford
learn french:  http://www.youtube.com/watch?v=30v_g83VHK4

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Aaron Mason
On Wed, Feb 8, 2012 at 4:17 PM, bofh <[hidden email]> wrote:

> On Tue, Feb 7, 2012 at 11:57 PM, Aaron Mason <[hidden email]>
> wrote:
>> But as Mr. Anon says, choose your hardware carefully.  Getting it
>> wrong can be disastrous.  My EON-based file server is a Core2Duo 6400
>> w/ 4GB RAM on a Gigabyte G41MT-ES2L and 4 2TB WD Green drives which
>> collectively pull 177MB/sec, though the onboard Realtek NIC maxes out
>> at 88MB/sec it still beats the shtick out of the 12MB/sec I was
>> getting before.
>
> I would avoid "green" drives like the plague.  Check out the SMART
> status on them and look at the drive park statistic among others.
> Look at how high the number is, versus what the life time recommended
> number...
>

I would too if it were an enterprise setup.  This is but a home setup
built on a limited budget.

>
> --
> http://www.glumbert.com/media/shift
> http://www.youtube.com/watch?v=tGvHNNOLnCk
> "This officer's men seem to follow him merely out of idle curiosity."
> -- Sandhurst officer cadet evaluation.
> "Securing an environment of Windows platforms from abuse - external or
> internal - is akin to trying to install sprinklers in a fireworks
> factory where smoking on the job is permitted."  -- Gene Spafford
> learn french:  http://www.youtube.com/watch?v=30v_g83VHK4
>



--
Aaron Mason - Programmer, open source addict
I've taken my software vows - for beta or for worse

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

bofh-6
On Wed, Feb 8, 2012 at 12:34 AM, Aaron Mason <[hidden email]>
wrote:
>> I would avoid "green" drives like the plague.  Check out the SMART
>> status on them and look at the drive park statistic among others.
>> Look at how high the number is, versus what the life time recommended
>> number...
>
> I would too if it were an enterprise setup.  This is but a home setup
> built on a limited budget.

Non-green drives are at the same price of green ones.  Maybe not by
the same manufacturer, but I'm partial to Hitachis.  Cheap and fast
and good (so they got bought out by WDC, and it'll be crappy soon).

--
http://www.glumbert.com/media/shift
http://www.youtube.com/watch?v=tGvHNNOLnCk
"This officer's men seem to follow him merely out of idle curiosity."
-- Sandhurst officer cadet evaluation.
"Securing an environment of Windows platforms from abuse - external or
internal - is akin to trying to install sprinklers in a fireworks
factory where smoking on the job is permitted."  -- Gene Spafford
learn french:  http://www.youtube.com/watch?v=30v_g83VHK4

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

Aaron Mason
On Wed, Feb 8, 2012 at 4:39 PM, bofh <[hidden email]> wrote:

> On Wed, Feb 8, 2012 at 12:34 AM, Aaron Mason <[hidden email]>
> wrote:
>>> I would avoid "green" drives like the plague.  Check out the SMART
>>> status on them and look at the drive park statistic among others.
>>> Look at how high the number is, versus what the life time recommended
>>> number...
>>
>> I would too if it were an enterprise setup.  This is but a home setup
>> built on a limited budget.
>
> Non-green drives are at the same price of green ones.  Maybe not by
> the same manufacturer, but I'm partial to Hitachis.  Cheap and fast
> and good (so they got bought out by WDC, and it'll be crappy soon).
>

Not in Australia, and not Seagate, the only brand I will trust these
days.  A 2TB Green drive is AU$135, a 1TB non-green is $155.  Oh, and
the drives were bought second hand off a guy who (stupidly as he
admits) bought them for a hardware RAID setup and had to take them out
because the raid kept falling over as the drives went into sleep mode.
 Not a problem with software RAID/ZFS.

--
Aaron Mason - Programmer, open source addict
I've taken my software vows - for beta or for worse

Reply | Threaded
Open this post in threaded view
|

Re: Backup Redundancy Etcetera

bofh-6
On Wed, Feb 8, 2012 at 1:02 AM, Aaron Mason <[hidden email]>
wrote:

> Not in Australia, and not Seagate, the only brand I will trust these
> days.  A 2TB Green drive is AU$135, a 1TB non-green is $155.  Oh, and
> the drives were bought second hand off a guy who (stupidly as he
> admits) bought them for a hardware RAID setup and had to take them out
> because the raid kept falling over as the drives went into sleep mode.
>  Not a problem with software RAID/ZFS.

Seagate is one brand I will not buy nowadays.  After their 1.5/2TB
fiasco, and more recently with one specific line of drives, so bad
that they got a huge number of 1 or 2 stars at newegg.  When 3TB
drives were in the $300 range, after the flood, that one specific
model was still at $150 or so - word was they were just trying to push
the inventory out and screw the customers over.

It's almost as if the Maxtor side took over.


--
http://www.glumbert.com/media/shift
http://www.youtube.com/watch?v=tGvHNNOLnCk
"This officer's men seem to follow him merely out of idle curiosity."
-- Sandhurst officer cadet evaluation.
"Securing an environment of Windows platforms from abuse - external or
internal - is akin to trying to install sprinklers in a fireworks
factory where smoking on the job is permitted."  -- Gene Spafford
learn french:  http://www.youtube.com/watch?v=30v_g83VHK4