Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

$.01 per Gig a month storage online with Amazon Glacier

Jarek Zabczynski said:
$120/year for a TB isn't bad I suppose. Then again, you can take that same $120, get a 2TB hard drive and keep it at a friend's / relative's house.
Click to expand...
Yes but Amazon Glacier won't crash one day soon.
 
For a 5k transcode from a DI to 2k, would be a simple move from Glacier to S3 then you could kick off spot instances ... would be nice to processing at 5000 frames per second (i.e. linux compositing/color correction/etc - as long as each frames were independent in time). This type of setup will be awesome for 6k when that happens (will be funny processing 6k a hundred times faster then a 2k pipeline).
 
How would you upload and get easy access to all these Terabytes online? I have problems just trying to upload a few gigs to my FTP for others to access....how fas of connections are there out there to allow for online data storage of this size? This all said, i have a library of about 15TB footage....and online storage does not seem practical.
 
Johnny Friday said:
How would you upload and get easy access to all these Terabytes online? I have problems just trying to upload a few gigs to my FTP for others to access....how fas of connections are there out there to allow for online data storage of this size? This all said, i have a library of about 15TB footage....and online storage does not seem practical.
Click to expand...

Easy way would be to just to ship amazon a few of the 12 Terabyte g-speed es external disks. They can receive those via shipping, stage them in s3 then move the data to glacier for when you need it. I would think this would be real nice for you guys that shoot in crazy locations(that way you don't even need to have a decent network connection). For Silicon valley Indie's, is more of a cable modem upload in 1 gigabyte chunks during the night (there are some ways of doing that so they can handle interruptions) ... but this is kind of a hassle. For production houses, the direct connect links run around $10k per month for 10 gigabit high reliability(i think there is distance between hops related cost, so you'd have to price that for your area [i haven't personally done this]).
 
PatrickFaith said:
Easy way would be to just to ship amazon a few of the 12 Terabyte g-speed es external disks. They can receive those via shipping, stage them in s3 then move the data to glacier for when you need it.
Click to expand...

Glacier is the long term storage not S3. I'm not sure if you have to move the files to S3 to download them or not. I need to dig a bit deeper into what they are offering. I've used AWS since they started. This is huge news.
 
Brandon J.F. said:
Glacier is the long term storage not S3. I'm not sure if you have to move the files to S3 to download them or not. I need to dig a bit deeper into what they are offering. I've used AWS since they started. This is huge news.
Click to expand...
Yep ... I was thinking moving data between elastic cloud map/reduce(hadoop) and glacier between major jobs (i.e. glacier when i didn't need to run anything for a month, then move from glacier to elastic cloud when i need to process streams). Give a yell on how it works out.
 
so far none..
Its been running on my mac pro to macbook pro for about two months. Haven't gotten a single corrupted file, transferred over 1TB of files not worried about thief cause optional encrypting with personal pin
 
Amazon is more for doing parallel processing on the data, with amazon glacier being a way to store things between major processing runs(since it's such a hassle moving that much data around on the network).
So if your thinking of doing a film trilogy of something at the level of avatar, but on 4k or 6k, amazon allows an indie to setup a processing film series just like the big guys (i.e. like a ILM data center). The catch on the cloud side is it requires excellent linux and digital computing skills. Also if your running 24x7 with continuous loads - the spot pricing can get expensive. For example it is relatively cheap to spin up 5000 servers, load some data from s3&tape, then de-allocate the servers after 24 hours. But keeping 5,000 servers running continuously over a year ... that's when you need to consider segmenting processing between the cloud and internal racks. So for the part of your load that is getting hit hard over time, something like editing/compositing/vfx it might make sense having a internal rack of servers that have nvidia 690 boards in them for that subset of work. If your dealing with a short, or a single feature that doesn't have a lot of vfx/sfx, then i'd think all the cloud stuff would be overkill.

Crash plan I see as a backup solution only, the catch being how long it takes on the network to access the data.
 
AWS and Glacier

AWS and Glacier

We ran a trial of Aspera On Demand - which is Aspera Faspex running on AWS (Amazon Web Services) - a virtual machine., with, theoretically, a very fast connection to the internet. We also ran Aspera 'Shares' on another virtual machine at the same time. Without even being charged for the Aspera License (which is not cheap, but we were doing a trial, so it was free) my AWS bill for two machine instances, in only one region, for only ONE month was something like $350.

For that price, I can rent a pretty fast server, probably with raid 1 drives, sitting on very fast fiber, somewhere in some building in downtown LA. We only moved a few gigs, 10 or 20 at most, worth of media during that month. Further, the connection wasn't any faster than much, much, much, cheaper store-and-forward type services (yousendit, etc.) In most cases, it was actually slower.

Left a very bad taste in my mouth for using Amazon for data storage or retrieval or processing or anything. It would seem there's a business there, since people are using AWS, but I can't see it as being cost effective for archiving or mass storage or large-file distribution.

For $120, get someone to make you an LTO.

Just my .02
 
You must log in or register to reply here.
Back
Top