Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

The über SAN

J

jamesratnarajah

Member
Joined
May 6, 2007
Messages
22
Reaction score
0
Points
0
Hi Guys,

Just wanted to get your thoughts on something. I am planning for the future in terms of what it will take to make a comfortable and expandable post environment for the Red workflow. My thoughts in this post focus on the SAN that will need to be constructed to serve say 5-6 workstations with the sustained read/write speeds needed to edit 4K files.

While doing research I stumbled on an HD Expo 06 interview with PLASTERcity Post director, Michael Cioni. In this interview he talks about how, at his facility, they have created a 'frankenstien type' SAN. He is referring to how they didn't purchase one 'off the shelf system' but created it themselves from bits and pieces from many different companies. The interesting thing, however, was that he stated that they have achieved a sustained data rate of 1.5GB/s to their 10 or so Final Cut Stations :blink: if I was able to achieve that kind of speed at my shop, it would be more than enough bandwidth to edit Red 4K files!

Now this may not be anything new to certain people here but my question is; has anyone else seen this kind of performance in SAN that serves roughly 10 computers? If so, what did you do to get there? I hope this hasn't been covered before in a previous post, but I thought this post could be handy for others, as well as myself, to gain understanding into creating the perfect über SAN.

What do you guys think?

James
 
jamesratnarajah said:
While doing research I stumbled on an HD Expo 06 interview with PLASTERcity Post director, Michael Cioni. ... The interesting thing, however, was that he stated that they have achieved a sustained data rate of 1.5GB/s to their 10 or so Final Cut Stations :blink: if I was able to achieve that kind of speed at my shop, it would be more than enough bandwidth to edit Red 4K files!
Click to expand...

Without having a chance to talk to Michael and see exactly what he means, here are some thoughts:

1) Doing some quick math, that means 153MB/s for each FCP station. What are 10 FCP stations doing at the same time that requires that bandwidth?

2) There is no rocket science here. I certainly believe that he can aggregate bandwidth through a SAN for 1.5GB/s sustained. But there is a huge difference between aggregate bandwidth across multiple clients, and single-stream bandwidth to a single client. One is much easier than the other. :)

This is only important because a lot of the discussion on this forum revolves around sustained reads at speeds necessary to support 4K RGB at 24fps. For Full Aperture 4K, that's about 1.2GB/s sustained to a single client.

And Michael may have accomplished this! Just pointing out that one is very different from the other, and since PlasterCity is an FCP shop, and FCP can't do 4K right now, I'm guessing that he is talking about aggregate bandwidth. :)

Lucas
-----
Knows more about SANs than he ever really wanted to
ASSIMILATE, Inc.
LA, CA, USA
 
jamesratnarajah said:
Hi Guys,
While doing research I stumbled on an HD Expo 06 interview with PLASTERcity Post director, Michael Cioni. In this interview he talks about how, at his facility, they have created a 'frankenstien type' SAN. He is referring to how they didn't purchase one 'off the shelf system' but created it themselves from bits and pieces from many different companies. The interesting thing, however, was that he stated that they have achieved a sustained data rate of 1.5GB/s to their 10 or so Final Cut Stations
Click to expand...

That number sounds like aggregate bandwidth, in other words, 150MB/sec to each station if every one is saturating the pipe. However, there's nothing magical about achieving more bandwidth on a SAN. It's all about the number of spindles, striping between LUNs, and the type of interconnection (i.e., FibreChannel and the way you design the fabric switch). At the same time, SANs are not for do-it-yourself individuals. They require a degree of knowledge and expertise for both installation and administration, especially if you're looking for very high bandwidths, as required by digital image files. If one is thinking about a SAN installation for an imaging facility, they would do well to hire a consultant/system designer to spec it out and deploy it.
 
Hey guys,

I think I understand now. So if I hired the right person to consult and get the right equipment for my shop, I would be able to achieve sustained rate of 1.2~1.5GB/s to each station in 5-6 station shop? Going to be using FCP to edit the 4K file once Redcode is supported.

James
 
hi james,

no, not really, getting "1.2~1.5GB/s to each station in 5-6 station setup" sums up to about 6-9 GB/sec. And thats not so easy to achieve, even by specialists. Switches, connections and raids have limits. You've got to bundle/team them an this is only possible with the really bug stuff. Its very difficult to aggregate to those numbers.
Somebody may correct me, but the number of posthouses out there able to do that (e.g. more than 4 clients with a simultaneous throughput in 4k form a SAN) is small and those houses are big or very specialised. 2 years ago when I retired from systems integration, I believe there have been about a handful in Hollywood. Surely this number raised, but still this a very expensive game. Try to alter your concept. Where and why do all those poeple have to edit 4K uncompressed ?
Or if on the other hand you're talking about RED 4K, its a different game. No deal at 30 MB/sec. Let them play with RED-4K stuff all at the same time.
For uncompressed: Try the avid-game. Edit in "low-res-HD-proxy" (aka ProRes). Should be more than enough.
Finish in 4k on 1 or 2 machines if you believe thats necessary. Still effort enough to get 2.4-3 GB/sec.
 
Hey augenmensch,

Thanks for your feedback. SAN's like this are all new to me so your feedback is helping me understand a lot more. If I could explain a few things that may make my ultimate goal easier to explain.

I would like these 5-6 stations to be able to edit RED 4k and other formats such as HDcam, DVCPRO HD, and most SD formats. As my posthouse will specialise in feature work, for indie film and TV, I would like our stations to be able to handle all of the mentioned formats comfortably. So I guess it's just a case of finding the max data rate required and getting this speed to all the stations?

The plan is to have these 5-6 stations as edit bays where the editor can work with the director. So for RED 4K, we would then run an EDL through redcine and take the full res 4K files to a Scratch or iQ system for conform. So I believe it's necessary for the 5-6 stations to have the data rate needed for RED 4K and all the other formats I mentioned.

What speed would this need to be? Do yu think this is achievable on the scale I require? Forgive me if my qustions seem very primitive.

James
 
not primitive at all. As I said I went out of this, am just in it by interest now.
But what you require is manageable "easily".
Your setup requires a sum (aggregated throughput) of worst case 40MB/sec. per workstation. A model that I like, because it can be realized without a SAN or fat SAN architecture. And FibreChannel is what makes it expensive.

If you only need GigE (thats what I think you do) a big storage with a fast connection should be the base. We got 37-42 MB/sec. on G5 Macs from a self-built NAS. However that NAS better be good to support 4-6 workstations simultaneously and a fast GigE Switch.
The following situation might be worth thinking:
A NAS with proper RAID-setup and 10-GigE NIC. GigE-Switch with 10GigE Uplink to feed your stations. All stations should support jumbo-frames on GigE. Then - in my thinking - it could be nice.
Thats won't bring you what a SAN can, but way cheaper. But don't forget, its a NAS and you should properly test it. Our NAS went slow when too many clients accessed it. Having a proper machine with enough throughput behind it is crucial.

On the other hand, you should have a look at a Terrablock.
Don't know if this kind of advertisement is appropriate here. But you get a lot for 15k. There are clear limitiations, like dedicated bandwidth cannot be allocated and user rights mangement is limited. (Usually within a SAN architecture the metadataserver is responsible to transmit and assign access rights on a per-file basis - Terrablock can work with access rights per user and volume/partition only. Meaning you could work with 2 edit stations on THE SAME project with a proper SAN (Capture into the project with one and edit within the same from another workstation). Currently impossible with Terrablock AFAIK. One reads, one writes. BUT: awesome performance for the price, small, manageable, "built-in-switch". Have to look though with its limitations mentioned, if it is suitable for you.

Ask somebody like Lucas here on the board if they are willing to help you integrate Scratch. Either with a NAS which is fairly easy to integrate or, with a Terrablock. They have experience with Terrablock which does about 300MB/sec. even on the middle model.
Or have Scratch with a dedicated storage and export your RedCine output (dpx sequence) directly to that over night or whatever it may take. Next [available] day start your work within scratch.

offhollywood said:
we will be testing MetaSan in the fall - some pretty cool advantages -
Click to expand...

MetaSAN is an interesting architecture. We used it on a TV series (April/June/July). And its distributing the metadata in a peer-2-peer model, not over a separate server. When we had our crashes once a week I thought they'd better use one... but it makes a MetaSAN based setup cheaper than the competition.
But don't expect it to give you troublefree life. Its still a SAN. And you can't (let's say shouldn't) have the same FCP project open on 2 edit stations. MetaSAN is not capable to handel that properly. Expect edit seuqence version losses (Older one saved over newer one - newer one gone).
 
Hi augenmensch,

I think I understand, going to have to research the information that you've given me. So do you think that a 40MB/s speed to all stations is enough to 'handle', HDCam, DVCPRO HD and RED 4K (obviously not at the same time).

James
 
augenmensch aka therock said:
... Ask somebody like Lucas here on the board if they are willing to help you integrate Scratch. Either with a NAS which is fairly easy to integrate or, with a Terrablock. They have experience with Terrablock which does about 300MB/sec. even on the middle model.
Or have Scratch with a dedicated storage and export your RedCine output (dpx sequence) directly to that over night or whatever it may take. Next [available] day start your work within scratch.
Click to expand...

We have several customers using Terrablock with great success. At one particular install, we have 3 SCRATCH systems + 1 Media Composer + 2 Shake stations + all tied to 2x Terrablock 24Ds.

A few weeks ago, I was talking on this list about being at a particular site where I was getting 1.3GB/s (gigaBYTES) off of a single client. That was with 2x Terrablock 24Ds.

Terrablock does have some limitations when compared to systems like CXFS and Bright/ADIC, but it also has some advantages and typically comes in at a fraction of the price of these other systems.

Cheers,

Lucas
-----
ASSIMILATE, Inc.
LA, CA, USA
 
Hi Guys,

Seems as though Terrablock is a seriously viable solution. Going to try write something up, in terms of specs that I require, and see what setup I'm going to need!

Thanks to all of you for your help!

James
 
this is what happens, when one writes too long on a post.. double info. Whatever, here's my superfluous work...:-)

HI James,

no - I dropped HDcam on my answer, sorry.
For HDcam you need a mere 130 MB/sec, usually a solid over it, lets say 150MB/sec. No way on GigE.
My proposed solution just got a big crack, sorry.
Err... no! Not necessarily.
Lets get back.. HD, uncompressed... 8bit, YUV....
So, if you're not directly talking about SR and 444 and so on... we might get ProRes422 back in to the equation..... then we are at less than 40MB again.

Is that a possible solution ?
There are 2 variants of ProRes, lossy and lossless I believe.
Maybe somebody else knows and has a little experience with it. The lossless compression is VBR I think and supposed to run from a 2.5" disc...

Experience anyone ? Mark ?

luki said:
...about being at a particular site where I was getting 1.3GB/s (gigaBYTES) off of a single client. That was with 2x Terrablock 24Ds.
Click to expand...

Hi Lucas, James:
The 24 is the bigger model, this sounds definitely worthwhile thinking.

An idea might be to start off with a single 24D. If need for capacity and speed rises, you double the package and do the stripe (what I assume Lucas' mentioned install consists of).
Call Facilis, get a quote.

We had our Terrablock running in both SAN and NAS.
Bascially it was a 2 client setup (with a 3rd to come, but never did), where the 2nd client was a server, acting as the NAS appliance.
Small, but performant and more stable than I thought (in fact no real problems during 2 years). What a pity it got stolen. (If you believe it, or not, yeah: stolen!)
 
Hi augenmensch,

To be honest I would like the edit stations to edit HDCAM 4:4:4. I would consider capturiung to ProRes possibly for broadcast features but wouldn't like to do it for feature work. So I would need the 150MB/s speed to all stations, and that will handle everything from HDcam 'downwards' (in terms of data rate, not quality).

After looking at the Terrablock solution, what I'm thinking of doing is having a SAN for the control room and edit stations. A SAN for the audio dept, and a direct attached 24D (or 2) to the scratch station. All I would need to figure out is a way of getting the files between the different SANs.

What do you think?

James
 
jamesratnarajah said:
To be honest I would like the edit stations to edit HDCAM 4:4:4.
Click to expand...

There is no such thing as 4:4:4 HDCam. Only HDCam SR does 4:4:4.

After looking at the Terrablock solution, what I'm thinking of doing is having a SAN for the control room and edit stations. A SAN for the audio dept, and a direct attached 24D (or 2) to the scratch station. All I would need to figure out is a way of getting the files between the different SANs.

What do you think?
Click to expand...

I think you really need to talk to a systems integrator who has experience with putting together shared storage systems. What you're talking about is possible (hint: have a specific workstation be a client on multiple SANs), but it's expensive, complex, unnecessary, and largely defeats the purpose you're trying to serve. It also involves multiple technologies, not something you really want to do when putting together a shared system. Take my advice: contact a systems integration consultant and let them put together something for you. They can do it a lot better than you can (and probably get better vendor prices as well).
 
We're using an Apple SAN.

Apple has a special group for professional organizations offering substantial discounts and terrific technical support. I had never heard of Apple discounting anything, but now we buy all our gear, even Powerbooks, under list.

While there are likely less costly solutions out there, I'm more comfortable with a SAN that is sure to work with all future updates to Final Cut. I really like a single source; no chance of finger pointing and, "it's not our fault."

We have four edit suites and two audio rooms with no problems whatsoever.

Good shooting and best regards to all,

Leo
 
Cool than that's what I'll do...... now does anyones know any good systems intergrators in the UK?

James
 
You must log in or register to reply here.
Back
Top