Flash Incompatibility

[Chris Kenny]

There is no memory card standard which reliably handles the data rates Red needs. This isn't a matter of Red putting a standard port on the camera, or a proprietary version of basically the same thing. REDFLASH is almost certainly going to be a sort of RAID using multiple memory chips/cards. The interface between the camera and the digital magazine is going to be something which supports that. There is no generic version of that.

 

[Stokestack]

But as I see it coming, the RED FLASH MEMORY INTERFACE for the camera will be plugging in to the RED-DRIVE SATA connector... SATA connector on the camera body is modified to supply the power since the original SATA spec do not have provision to deliver power to the SATA device.

Is the existence of this port on the Red confirmed?

And I couldn't believe eSATA failed to provide for power, considering it's a standard from 2004. What a ridiculous omission. People never learn.

 

[Jeff Kilgroe]

There is no memory card standard which reliably handles the data rates Red needs. This isn't a matter of Red putting a standard port on the camera, or a proprietary version of basically the same thing. REDFLASH is almost certainly going to be a sort of RAID using multiple memory chips/cards. The interface between the camera and the digital magazine is going to be something which supports that. There is no generic version of that.

...Although, several new FLASH products were just shown at CES that could handle the data rate.

IMO, the more I have thought about it the last two days, I think RED should adopt an ExpressCard interface and use standard EC storage media. Then we could pop the card right out of the RED FLASH unit and put them right into a Macbook Pro or newer PC notebook or other system that has an ExpressCard slot. Essentially taking Panasonic's P2 design to the next level. Also don't cripple the standard by making the cards and/or interface proprietary like P2. There will be lots of EC storage devices based on FLASH memory and many of those due out over the next several months will have transfer rates plenty good enough. SanDisk showed both CF and EC media at 32GB and 64GB at CES that had write speeds in excess of 30MB/s. Read speeds were nearly 50MB/s. Off-the-shelf FLASH products that can cope with RED's I/O requirements are literally a couple months away.

 

[Jeff Kilgroe]

Now, out of curiosity, while we're on the subject of FLASH and mini-RAIDs and all that. What file system(s) will be used on the RED FLASH media and the RED DRIVE? Anyone at RED able to comment on that yet? Has it even been decided? Hopefully not FAT32 so we don't have to put up with multiple partitions on RED DRIVE. NTFS is great, but OSX can't write to NTFS volumes and HPFS requires third-party software on Windows. Hmmm... I guess the FLASH and DRIVE units could use a common format and then provide translation for one platform or the other. ZFS would be cool to use, but I'm not sure on the licensing from Sun... OSX 10.5 will support ZFS. Windows does not natively... Hmmm.

 

[Chris Kenny]

The file system question is a good one.

ZFS would be cool, because it checksums everything transparently, and in REDFLASH, where there will probably be a sort of mini-RAID setup with multiple chips/cards internally, you could even conceivably use RAID-Z for redundancy. It would also be really slick if you could use ZFS features on external RAID arrays. Just hook up a box of SATA drives (with a port multiplier), and set up any combination of striping/mirroring/RAID-Z across them. I suspect, however, that the overhead and complexity of ZFS might be rather high for an embedded system.

NTFS isn't writable out of the box in OS X, but is this a huge deal in this application? The only real consequence of this would be that you couldn't reformat drives/cards on OS X, but the camera could presumably format them, so this wouldn't be a huge problem, right? On the other hand, I'd be really afraid to erase a digital mag if I weren't the one who'd just personally witnessed it downloading, so maybe the inability to erase it directly on the machine that downloaded it would be a big deal. MacFUSE implements writable NTFS on OS X, but I don't know how well it works.

Apple's HFS+ isn't conveniently readable on Windows systems, so that's probably out, unless Red wants to license some of the existing software that does this and bundle it with the camera.

Red probably doesn't want to write new file system drivers for two or three operating systems, so something custom is probably out.

This basically leaves FAT32....

Volume size with FAT32 isn't actually a huge problem. It can go up to 8 TB with 32K clusters (which wouldn't waste a lot of space in this application.) The 32 GB limitation a lot of people are familiar with is actually just a limitation in the Windows disk formating utility. OS X and numerous third-party Windows utilities will create larger FAT32 volumes, which Windows can use with no problem.

The real killer for FAT32 is the 4 GB maximum file size. This would mean any shot over about 2.4 minutes (at the REDCODE 4K data rate) would have to be broken into multiple files, which would be a big hassle in post. REDCINE could have a feature where it recombined them automatically as appropriate while copying them to your computer, though, which would make it tolerable.

 

[Jarred Land]

Is the existence of this port on the Red confirmed?

And I couldn't believe eSATA failed to provide for power, considering it's a standard from 2004. What a ridiculous omission. People never learn.

yes... the sata port on the back of the camera goes to the RED DRIVE and records 4k Redcode.

No power on E-sata is indeed seriously retarded. It makes Firewire 800 look like gold.

even more retarded about Sata is that the power port uses so many damn pins.. 3x the amount it uses for data.

 

[Chris Kenny]

There will be lots of EC storage devices based on FLASH memory and many of those due out over the next several months will have transfer rates plenty good enough. SanDisk showed both CF and EC media at 32GB and 64GB at CES that had write speeds in excess of 30MB/s. Read speeds were nearly 50MB/s. Off-the-shelf FLASH products that can cope with RED's I/O requirements are literally a couple months away.

And there are now SATA flash drives with similar performance. But 30 MB/s is really cutting it too close for REDCODE RAW 4K (27.5 MB/s) + four channels of audio. The interfaces support more, but the products currently on the market don't. True, they probably will in six months.

 

[Jarred Land]

Fat32, as much as it sucks, is the best solution for the application... and is why other systems like Panasonic's P2 uses it.

Multiple clips are not necessarily a bad thing.. i would rather have files save and new files open every 2 minutes while i'm recording long form.. because if a file gets corrupt, its only one small piece.

 

[Jarred Land]

And there are now SATA flash drives with similar performance. But 30 MB/s is really cutting it too close for REDCODE RAW 4K (27.5 MB/s) + four channels of audio. The interfaces support more, but the products currently on the market don't. True, they probably will in six months.

most of the specs manufacture's release are bogus.. they are at best measuring small bursts of speed.

There were some nice drives announced at CES that look to be fast enough.. but none of those are shipping, and wont be available for some time.

 

[Stuart English]

Write speeds

Write speeds

Jarred beat me to this post - above all don't believe what the spec says - test it!

Our experience is that few of the data write speed claims stack up in practice, so you would need a very healthy margin between needed speed and claimed speed before you could trust that off-the-shelf (non certified) media would always be up for the task. And there is nothing more critical than reliably recording your images.....

 

[Stuart English]

Clips in Post

Clips in Post

Multiple files per clip could be a pain in post, but its really a question of how they are handled / presented. There is no doubt that some of the existing IT based camcorder systems have failed miserably in that regard.

 

[Stuart English]

Emergency media use

Emergency media use

Perhaps as we mature the camera system over the coming months we could enable a mode where we can measure the speed of a piece of off-the-shelf media that you may need to use in an emergency situation. We'd have to write then read back test signals to the media and based on those results either lock out certain recording resolutions / frame rates or work out how to lower the recorded image quality via a higher compression setting.

If it were possible, is it a desireable function to have?

 

[Jeff Kilgroe]

Fat32, as much as it sucks, is the best solution for the application... and is why other systems like Panasonic's P2 uses it.

OK... I figured as much.

Multiple clips are not necessarily a bad thing.. i would rather have files save and new files open every 2 minutes while i'm recording long form.. because if a file gets corrupt, its only one small piece.

True, but such breaks can be introduced without resorting to FAT32 file and volume size limits to impose them. Ironically, if a more robust file system were being used (like ZFS, HFX, XFS, EFS, UDF etc..) data corruption would be a lot less common. FAT32 isn't very robust at all and the only reason people still use it is because, well... I honestly don't know.

Seriously, here's a consideration. UDF. The UDF file system is supported by OSX, Windows, Linux, BSD, Sun and even legacy Irix. It's not limited to 4GB file sizes or 32GB per disk/partition. It's the standard being used for both HD-DVD and BluRay media. Also used by various magneto-optical formats, prototype holographic units, high-capacity cross-platform FLASH products like some of those demoed at CES, XDcam media and Iomega REV drives. UDF is an open standard. And if you download one of the various UDF format utilities out there, you can format an external hard drive as UDF and it plugs into a Mac, Windows or Linux system and works just fine.

Sorry to pick at this, but FAT32 is just so 1998. It works OK for something 32GB and smaller, but the thought of FAT32 and having 10 partitions on a RED DRIVE with all video files capped at 4GB no matter what isn't all that appealing.

 

[Finner]

Perhaps as we mature the camera system over the coming months we could enable a mode where we can measure the speed of a piece of off-the-shelf media that you may need to use in an emergency situation. We'd have to write then read back test signals to the media and based on those results either lock out certain recording resolutions / frame rates or work out how to lower the recorded image quality via a higher compression setting.

If it were possible, is it a desireable function to have?

This is massively desirable. I would want to use red flash memory and would pay extra for the peace of mind but it would be terrible not to have any back up options if a person ran into an emergency situation.

You just mentioning looking at the option makes me feel a lot more comfortable with this workflow.

 

[Jarred Land]

OK... I figured as much.

True, but such breaks can be introduced without resorting to FAT32 file and volume size limits to impose them. Ironically, if a more robust file system were being used (like ZFS, HFX, XFS, EFS, UDF etc..) data corruption would be a lot less common.

Sorry to pick at this, but FAT32 is just so 1998. It works OK for something 32GB and smaller, but the thought of FAT32 and having 10 partitions on a RED DRIVE with all video files capped at 4GB no matter what isn't all that appealing.

I agree with you 100%.. Fat32 was the cool new kid on the block in friggin 1995 for christ sakes. But unless some kinda christmas miracle happens, I think we will still see Fat32 floating around in many devices and drives for the next decade.

 

[Chris Kenny]

Sorry to pick at this, but FAT32 is just so 1998. It works OK for something 32GB and smaller, but the thought of FAT32 and having 10 partitions on a RED DRIVE with all video files capped at 4GB no matter what isn't all that appealing.

See my earlier post. FAT32 volumes go up to 8 TB. The 32 GB thing is just a limitation of the default Windows formatting utility. It's not a limitation of the file system itself or of its implementation in Windows, OS X, or Linux.

 

[Jarred Land]

You just mentioning looking at the option makes me feel a lot more comfortable with this workflow.

Welcome to RED

 

[Jeff Kilgroe]

I agree with you 100%.. Fat32 was the cool new kid on the block in friggin 1995 for christ sakes. But unless some kinda christmas miracle happens, I think we will still see Fat32 floating around in many devices and drives for the next decade.

It's sad, but I know you're right...

 

[Jeff Kilgroe]

See my earlier post. FAT32 volumes go up to 8 TB. The 32 GB thing is just a limitation of the default Windows formatting utility. It's not a limitation of the file system itself or of its implementation in Windows, OS X, or Linux.

Er... Microsoft chose to limit FAT32 to 32GB partitions for a reason. I've got the whole FAT32 spec in a binder on a shelf here. Yes, it can be extended to much larger volumes -- actually beyond 8TB if you really wanted to. That doesn't mean it should be done. Capacities aside, there are plenty of good reasons to NOT USE FAT32.

FWIW, Windows isn't the only operating system (or disk management software) out there that limits FAT32 volume sizes.

With FAT32, there is no internal garbage collection. No real error correction or data verification. No drive surface mapping functions to monitor for bad or currupt data areas, etc.. FAT32 is just a way to dump bits of data to a storage volume and use the allocation table (over-glorified table of contents) to find everything.

I don't even trust NTFS or HPFS for my SAN volumes in excess of 1TB over RAID-5, why would I want FAT32 in anything that size or larger?

 

[Chris Kenny]

Jarred beat me to this post - above all don't believe what the spec says - test it!

Our experience is that few of the data write speed claims stack up in practice, so you would need a very healthy margin between needed speed and claimed speed before you could trust that off-the-shelf (non certified) media would always be up for the task. And there is nothing more critical than reliably recording your images.....

Well, those flash hard drives drives I mentioned get pretty close to the 30 MB/s, and of course they're stolid state, so they're a lot more consistent than mechanical storage. Take a look here: http://www.tomshardware.com/2006/09...e_obsoletism/page4.html#data_transfer_diagram

But, like I said, this is pretty iffy for REDCODE RAW 4K + audio. If that line were close to 40 MB/s instead of 30 MB/s this format would be more plausible. Although, it's probably not exactly the form factor you want.