Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Storage and archival options

Christoffer Glans

Christoffer Glans

Well-known member
Joined
Jun 11, 2007
Messages
4,993
Reaction score
36
Points
48
Age
42
Location
Stockholm, Sweden
By now, it's no mystery that Komodo will require more disk space for the material that we shoot. For anyone who views Komodo as their new A-cam rather than a crash cam, this will probably be the biggest challenge going forward.

What might be good options going forward? Good drives that are cheap, auto-backup type drives? Archival options for RAW material and so on?
Even though this is always something important to discuss, I believe it to be more so going forward with Komodo as an A-Cam.

Thoughts?
 
Storage is cheaper now than it was in the past. Anyone already shooting with other 4K+ cameras already knows what they’re getting into. So same as before, reasonably sized RAIDs/ SAN systems for online and near-line storage. Bare HDDs for bulk storage, supplement with LTO tape for critical data if or as budget allows. That’s pretty much our current strategy. Good backups are a lifesaver. And eventually everything needs to be moved to long term storage if you want to keep it. Backup and archival is more a strategy than about a specific technology. Whatever methodology you choose, plan to revise that every now and then. Plan to revisit those backups and archives every so often and migrate them forward to newer tech. My oldest backups are on LTO-4 tape and newest on a combination of HDDs and LTO-6/ LTO-7 and I will pull them forward at some point here. Probably when moving to LTO-9 or other archival solution that I decide on in the next year or three.

I don’t shoot anywhere near as much these days as I did in the RED One and EPIC times, but I stil produce just as much or more data with all that I do. Camera files have grown, other image and app data has grown... This past year, 10~12 TB HDDs have been the sweet spot for bulk storage and my RAIDs/ SAN systems. I’m all SSD storage for onboard and direct-attach on my workstations for performance and increased reliability.
 
Yes, this is what I mean. A thread about what solutions that are good, both strategies and also what products that are good. It's a topic that I think would be good for those planning workflows.

What do people use? What strategies are good etc.?
 
My story is the same as a lot of others. Started way back in 2006 with some Western Digital externals. Moved up to a 4 bay RAID 5, then got a second RAID array to back up my first one.

Right now I've got a Synology 1819+ in RAID 6 with 12TB disks, which back up nightly to a local RAID array and over the internet to a Synology 420j at my parents' house. This gives me triplicate backup with redundancy on all three devices, with about 700-900MB/s speeds out of the Syno. I would recommend a DS420 or 920 to begin storing things, but if you want to edit off of it, you'll want at least 8 drives with a 10Gbe network. You need the 8 drives to be able to saturate the bandwidth of the 10 gig connection.

You could go with T5 SSDs, but that's a single point of failure, though quite unlucky. You could go with an SSD or NVMe RAID, which would provide extraordinary performance, but might be a bit overkill for an owner/operator, and certainly isn't the most affordable option.

Curious to hear what everyone else's setups are.
 
LTO is the only real option for archiving footage. Storing on spinning drives is an accident waiting to happen.
We have been using LTO for over 11 years now, and it has changed our life. We have over 800TB of footage on tape, and being able to search a database and restore a project from years ago is a game changer. We have nightly backups of the entire SAN happening each day and separately we archive footage and projects to LTO. Far cheaper and more manageable than drives, and we add a small archive fee to every project to offset cost. Plus bill for restores from our archives.

and I'll just add that during the initial COVID shut down, life would have been miserable with any other solution. We edited new commercial campaigns during shutdown for two clients that were completely assembled from past campaigns. The ability to quickly find and restore those assets from LTO made that possible.

The upfront cost may seem high, but its really not...and once you make the change, you will only regret not doing it sooner.
 
It really depends more on if you are an individual managing your own data or part of a small or large studio. If it's the latter then it really should be a dedicated person to handling it all. If you are an individual a good modern machine with TB3, a good TB3 storage array, and a good online backup solution with a fast internet connection, is infinitely easier and more accessible than LTO options.

I have an array for active projects and other that acts as an archive. Both get backed up to Backblaze over a 1Gb/1Gb connection for $55/year. I personally use the G-Tech Shuttle TB3 arrays and have yet to have any issues with them with them on 24/7. I like having 2 separate arrays so if one goes down for whatever reason there is another ready to go. This is just a personal non-professional solution that I use for my own video and vfx projects. If you are working on projects with data security issues then this might not be viable.

Obviously for a small or larger studio with multiple people you need to start looking at a dedicated SAN setup and someone to manage it and might not be able to utilize off the shelf backup solutions for security reasons. If you never check your LTO archives or do anything with them, it is much more useful to just keep an online archive and pay for the storage that you need to do that. LTO definitely has it's uses but having an online archive is much more useful if you can afford it. The setup I use professionally for work deals in PB not TB and 100Gb connections, so that is an entirely different solution with an entire team dedicated to it.
 
Last edited:
Hi, as a new Red user with Komodo being my first purchase can anyone tell me what’s a micro budget storage solution? I have angel birds 2tb but need to know after recording where should I put it to edit.
 
shaun bless said:
Hi, as a new Red user with Komodo being my first purchase can anyone tell me what’s a micro budget storage solution? I have angel birds 2tb but need to know after recording where should I put it to edit.
Click to expand...

For post-production you would need a RAID. I have three internal normal 7200 RPM drives put in a SATA RAID0 to maximize speed. But you also need to back your material up. There are some slow but large drives. I have a 18TB drive where I put all my material until it's full, then buy a new one.
Handling material is a big part of shooting with Red cameras, so you need a budget for this as well.

I'm thinking of buying a LTO-8 or LTO-9 solution for all material I don't use. Buying drives that could fail has been my way for a long time because LTO is expensive. But now that I have a lot of material piling up a lot of drives, it isn't really a good option for me anymore. Buying a LTO-8 recorder will set you back around $5000, but will save money in the long run. Hopefully LTO-9 won't be more expensive and might drive cost of LTO-8 down a bit. Bottom line, it will be a good choice to have LTO for long term and a stack of normal drives for day to day work that you replicate between drives (3 copies on 3 HDD for each file)
 
Michael Tiemann said:
Not really. I have some LTO-5 and LTO-6 drives, but am waiting for LTO-9 to refresh everything.
Click to expand...

Do you have any idea what the cost for LTO-9 would be? Does the price usually just replace with every new generation? LTO-8 price will be the LTO-9 price when it's released etc.
Any good brand recommendations for LTO recorders and tape?
 
Been there, done that. All the options for 30+ years.

Right now for Archiving LTO is the only valid choice. As much as it pains me, old fashioned tape is tried and tested. I've been through SMR archive drives and god knows what.

I tend to stay behind the LTO curve a bit, right now i'm on LTO 7 and it's a nice balance between cost and storage space. I have a couple of NAS, one is 10gbe and the machine that the drive is plugged into can pull data off at the archive speed. For big files it's actually pretty fast - for thousands of small images/frames it can be quite a bit slower.

At the moment i'm using LTFS as it appears fairly standard and it's quite user friendly to use. Mount the tapes like a drive and drag on and off. Having said that i'm running a separate cataloguing software so i can quickly find files without needing the tapes.

I've had near misses with the synologies. I've had drive failures and multiple drive failures. I'm running Raid 6 and even so a couple of close calls (always use NAS specific drives!!)

I am just me as well, but still generate a shed load of archives and footage!

cheers
Paul
 
Half of my LTO catalog is written with BRU PE. I want to use the LTFS system moving forward. What are the alternatives to “PreRoll Post” by Imagine Products for a PC user. Looking for simple clean user interface and ability to search file names for off line tapes.
Any recommendations would be appreciated. Thanks
 
shaun bless said:
Hi, as a new Red user with Komodo being my first purchase can anyone tell me what’s a micro budget storage solution? I have angel birds 2tb but need to know after recording where should I put it to edit.
Click to expand...
If you are looking for the most easiest solution: go for single drives, for storage and for work also single fast, like ssd's.
 
Mark van Coller said:
Half of my LTO catalog is written with BRU PE. I want to use the LTFS system moving forward. What are the alternatives to “PreRoll Post” by Imagine Products for a PC user. Looking for simple clean user interface and ability to search file names for off line tapes.
Any recommendations would be appreciated. Thanks
Click to expand...
I beta tested Bru for Windows late last year. Obviously they could never make it work, but the biggest problem was that installing it on Win 10 wiped out LTFS functionality. Archiware P5 desktop was free for a few months last year but it is not writing to LTFS reliably and consistently. Probably if I paid for their support they might trouble-shoot and fix it but the following solution is working fine for me.

I've been using this program https://www.wincatalog.com as a poor man's MAM and it's brilliant. Be aware that Windows will not preserve original time stamps when copying directly to LTFS, YOU MUST 7ZIP them if you want to preserve original timesamps. TerraCopy works well for automating the archival process. 7ZIPing your files before archiving will also remove the issue of spaces in filenames and unrecognized characters not being recognized when you retrieve your LTFS files.
 
You must log in or register to reply here.
Back
Top