computer failure. backup routine suggestions

[Samir Patel]

Looks like my crap computer finally failed. Any backup routinr suggestions for going forward anyone? I tend to make copies of very important files every few months but never on a schedule. Going forward I need to come up with a schedule. Any tips everyone else is using?

 

[Andy White]

Buy a small server (or NAS) and have a secondary DAS stuck into that.

This way you can schedule backups and it will change the way you work ad look at data.

Then, only use your machine to install and run software and use the system drive as a cache. If you run a second fast internal drive, mirror that data at the end of a session to the NAS.

Basically, the NAS becomes your central data store (ie a big hard drive that is not part of your computer) - run it as Raid 1 (it will half the drive capacity, but, on a 2 disk system, all your data will be copied on teh second drive so it's simple to add a new drive if one fails and this will rebuild the array). The DAS unit sticks in the back of the NAS as a second back up system, where you schedule a time for it to make a complete backup.

Using a simple set up like this (it can even be done on a wireless network with your Router as the hub), you administer the NAS from a browser (or network login) as it's a small computer system itself.

Advantage is you can then network all machines int this - say you get a new computer and then a laptop or tablet etc... and, rather than duplicating data, you simple add the machine to the network and get working.

This also means, when a machine goes down, you don't have that 'kapow' moment... as all your data stays safe and in the same place. All you do is put re-install your software on a new machine and you're good to go. Of course, you could even store machine 'snapshots' on the NAS... and this wouldn't take long, as your machine(s) are only your interface to the data (which is held on the NAS).

It's not that expensive to do the above - and easy to put together on PC or MAC... basically plu and play (years ago, you'd need to know a lot about networks, now it's all point and click The only thing I'd add would probably be to sit the server/NAS and DAS behind an UPS, incase of power spikes or failure.

 

[Samir Patel]

Thanks Andy. Looks like I also need to finally have a computer for general web browsing, email and another one for only cs6, red, resolve to avoid that kapow moment

 

[David Kuhn]

I like triple redundancy if I can get it... for my home systems:

Clients (Wife's machine, Kids's machine's, and my desktop system):
Raid1 if possible, but not always (laptops)
All important data stored on File Server.
(these are 'replace or rebuild clean if broken' sorts of systems)

Hyper-V Server:
-Raid1 for Boot
-Raid6 for data
-Boot partition backed up daily to Data Protection server (with a separate Raid6 array).

Server VM's:
-Entire VM backed up daily to Data Protection server.

For my cluster at work (VMWare) I've got the same basic setup... but the primary array is EMC Enterprise class (instead of 'do it affordably' class), The backup array is one of the higher end Thecus units (small business class I'd say). Triple Redundancy again... just with another 0 on the price tag - and IT takes care of the desktops so I don't need to - and I have hot spares on all the arrays too... disks are cheap.

 

[Andy White]

Thanks Andy. Looks like I also need to finally have a computer for general web browsing, email and another one for only cs6, red, resolve to avoid that kapow moment

Yup - it helps to separate work from play/casual browsing - the stress you get when a machine goes... and then finding you didn't backup the data you need or the last backup was 3 month ago (etc) just adds grief that can be avoided.

We all work with computers for pretty much everything we do - so backup and data management is important

I'm on my 'web' laptop now (I use it only for surfing and sometimes testing out trial versions of software, so I can see how they run without knackering a workstation). My workstations are rendering some CGI shots and I have a couple of laptops processing some mocap data... all are on the network and writing to a central store (my main NAS). I'm considering something like a Surface2pro as my next web/test machine.

 

[Gavin Greenwalt]

I have a multi-pronged approach. For the first 500GB I have OneDrive from Microsoft. This is my "live" documents drive so that I can access everything active between my PC at home, work, Surface and phone.
OneDrive

For my near-line storage I simply use an external enclosure with 3TB drives. Every other night I use SyncBack SE (free) to update the attached storage/NAS. RAID makes for a second/third copy depending on if it's in the OneDrive folder.
http://www.2brightsparks.com/syncback/compare.html

For a third/fourth level of security I also use Backblaze. Backblaze is unlimited storage for $50 a year per computer. Backblaze.com

As I see it... if my computer fails and it's a file I use a lot (documents, invoices etc) it's on OneDrive. If it's not on OneDrive or somehow OneDrive died, it's also on my external storage. If my external storage and computer burned up in a fire and OneDrive also lost a file then I have Backblaze. It seems paranoid but I have gotten to the third level of backup before to find a file that was accidentally deleted (and therefore not mirrored to OneDrive). I wouldn't ever consider 2 copies of data sufficient. I also ensure you have at least one off-site system. And make sure you have one system which indefinitely holds files (aka doesn't synchronize deletions).


In the studio we have a slightly different system. On the File Server we're RAID 5. The File Server has Shadow Copy enabled in Windows Server so if you delete a file or write over it, you can just retrieve it going back several versions (This saves us all the time). Everything is nightly Mirrored to a second system. Also every night the changes are backed up to an external Drive and walk off-site. So if the office burns down there's a new version of everything on an external portable drive.

 

[Wayne Morellini]

There is an backup issue I read over at xxcopy or XXclone. That due to buffering and caching in the os and drive system, verification gets partly done on buffered data rather than the actual written data, to make it worse schemes use the checksums only where there are compensating errors that can slip through with the same checksum result. So dealing with terabytes of data with the yearly re-backup process some people use corruption can get through. I have being meaning to research for software that comprehensively does a word for word comparison verification of actual data flushed to the disk. Does anybody know of true solutions?

 

[Gavin Greenwalt]

Does anybody know of true solutions?

Multiple Redundancies. If there is a checksum error on one out of 10,000 files (extremely unlikely but I guess possible). The chances of that occuring on a second copy using a separate copy application on a different day on the exact same file is highly highly unlikely. Most of our studio archive is image sequences. Layer and layer of image sequences. Millions of files. We've never had a bad copy. But if we did have a corrupt copy, we've got another one on a static drive physically separated from the other copies and not vulnerable to a software bug deleting anything on them.

The only time I've EVER had a copy error is with REDMags. And I've had tons (a few dozen, but relative to none from millions way too many). RED's REDMAG readers are crap and I assume now that anything I copy from a REDMag has a few corrupted frames so I must make more than one copy before formatting a mag. It might be because they're using FAT instead of something modern and reasonable file system like exFAT, NTFS, HFS etc.

 

[Wayne Morellini]

I need bit accuracy myself, but I notice people in the past saying transferring their footage archive to new media each year is the safe way to do it. On intra footage you may loose a run of pixels, but on inter footage (we are not talk just cinema camera archives) that might be a bit worse. Maybe a better checksum system is now in play, as you indicated, in the new file systems, I definitely was hoping to replace it in my is design. Still well done software that dots the T's and crosses the I's would be good, after all, a compensating error would come back as valid between the raid drives and backup which would just copy it, unless the raid drives are set to cross the t's and dot the I's.