Your Storage and Backup System and Strategy: Chime In.

[Anson Fogel]

Much discussed but sporadic, limited and spread out here. Let's put it in one place. We've got lenses covered here exhaustively, but this subject is just as critical to my daily work and to a successful RED workflow - we all know how storage intensive RED is. Let's get the spectrum from small one man shops to big post houses. So, please post:

Your online storage systems with specifics, costs, and specs
Your offline storage systems with specifics, costs and specs
Your backup systems with specifics, costs and specs
Your backup practices with schedules, times, uses, etc
THEN discuss the pros and cons as you see them for your system.

I know, its a lot to ask, but the details matter and it would be great to get them all on one place, sort of a state of the union for 2010 given the rapid state of change here.

I'll start in the next post.

 

[Anson Fogel]

Our storage and backup system and strategy

Our storage and backup system and strategy

Forge is a small shop, 2 of us plus assistants and other frequent collaboraters. Shooting on in house RED and occasionally 5D, 16mm, Sony for commercials, promos, web, corporate and some for fun.

Primary Editing and Post Machine: Mac Pro later 08 dual quad 3.2, 16GB ram, dual 300GB raptors RAID0 as system drive, not used for any storage other than apps, etc

Online Storage: 1 Caldigit HDOne running RAID5, early 2009 model, updated all drives from 8 ea 500GB to 8 ea 1.5 GB hitachi drives fall 09, total 10 TB usable storage, about 400 MBPS in Aja test.

Offline Storage: 2TB MyBook FW800 drives - currently 8 - 4 in studio, 4 offsite in fire safe mirroring the primary 4. Secondarily, 2 ea 1TB mybook Fw800 for system drive backup using time machine with one offsite mirror.

Strategy/Schedule: Backup entire online Caldigit array once a week (give or take, depends on activity) manually to multiple FW drives, rotating the 4 offsite and onsite every 3-6 weeks, while also checking for health. PITA. Backup recently completed or heavily worked projects to the FW drives more often.

Pros/Cons: Ongoing and then significant problems with CalDigit array dropping drives, but until recently has worked well save the frequent drive locks. The backup system is clumsy, manual, and time consuming. LTO solutions probably coming soon to us, but w/800GB tapes and the cost of robotic loaders, not really helpful with respect to time and hassle, only data integrity. Just may have lost 100 hours of work due to failure of more than one drive and my failure to backup the 2 projects as we worked on them. Considering dual raid arrays mirroring each other all the time, and a less frequent backup schedule to LTO tape for archiving and offsite. Time machine for system drive works like a champ. Would love to hear others solutions to the backup process.

 

[Peter Osinski]

I am going to start off by saying that I am aware my set up is less then top of the line, but this is all a result of my budget as of yet

Primary Editing System: 2008 Model Mac Pro 2x2.8Ghz quad cores w/ 2 (soon to be 16) GB of RAM booting OSX snow leopard and apps from external fw800 drive

online storage: 4 x 2TB drives in internal Bays Raid0 setup by apple software raid 8TB Total ($600)

offline storage: Projects are backed up to drives of appropriate size (whatever will fit total project footage and project files + scratch disks )

Schedule/Strategy: Backup projects to individual project drives from online storage, once project is finished move backup drives offsite, which leaves one copy on online, and one on offline. When Online gets too full I back up the project onto another harddrive and keep that on site, I then delete the project from online storage. In the end I will still have 2 copies of every project, 1 offsite at all times.

 

[Bruce Allen]

My storage system:

archival / work space:
- 8x2TB drives in RAID6 - highpoint 4300 series controller - approx $3000

current feature film grading project:
- 4x2TB RAID5 periodically mirrored to another 4x2TB RAID5 (plus most important 2TB is backed up to workspace RAID6) - $1100 each

transfer / temp:
- 5 x firewire / USB drives (.75TB - 2TB) - $1000

backup:
- 10 x firewire / USB drives of misc sizes (have copies of what is in archival / work space) - offsite - $1500 (some bought years ago)
- plus crucial stuff stored AGAIN on a PC with software RAID5 (6TB) - offsite - $1000 (again, bought a long time ago)

This is for personal projects - at my day job they of course have their own storage system and backups.

Bascially, my strategy is:
- availability (have everything on drives, which I check regularly so that I know that they're working)
- redundancy (everything important is on at least 2 different physical chassis)
- cheap (essential for 3-fold or 4-fold redundancy which is what I have for my important stuff)

Haven't lost anything for over 10 years.

Because everything is so cheap I can afford to just keep buying bigger and better RAID systems and copying the important data over. I will probably buy a new RAID in the next 9 months or so, but am just waiting for:
- the 3TB drives that Drobo accidentally announced
- possible innovations re USB3 or 6gb/s SATA that give me high performance for cheaper

Personally, I've never seen a restore from tape go smoothly (I've seen 4 Xserve RAID failures at different companies I've worked at and they have ALL lost substantial chunks of data when trying to restore from LTO). So I prefer to just rely on multiple sets of hard drives.

Also, I make sure that the hard drives in my RAID are from different manufacturing dates.

Bruce Allen
www.boacinema.com

 

[michael zaletel]

(4) - G-RAID 4TB eSata RAID Drives
(32) LTO-4 Tapes
(40,000) Iomega 100MB Zip Drives - J/K

-michael zaletel

 

[Rainer Fritz]

Mass Storage Fibrenetix Pixi extreme 9 or 12TB in RAID6 sustained 650MB/s via SAS ATTO R380 controller... cost about incl. controller 8000,- EUR

for set backup SK-3500 enclosure with hitachi enterprise drives 1TB or 2TB via esata.

LTO-4 HP 1840

lg
rainer

 

[Anson Fogel]

Highpoint question

Highpoint question

Bruce,

What enclosures are you using with the HighPoint controllers? Assume you are using the Hitachi drives...any problems? Is the enclosure super loud?