Flash Incompatibility

[Chris Kenny]

Er... Microsoft chose to limit FAT32 to 32GB partitions for a reason.

[...]

I don't even trust NTFS or HPFS for my SAN volumes in excess of 1TB over RAID-5, why would I want FAT32 in anything that size or larger?

Frankly, in this application, the answer is because it probably beats messing with 10 partitions on a 300 GB REDDRIVE. Unless REDCINE does, in fact, hide all of this nastiness from the user, in which case smaller partitions would be fine.

 

[Jeff Kilgroe]

Unless REDCINE does, in fact, hide all of this nastiness from the user, in which case smaller partitions would be fine.

Hopefully it does hide such things... I would expect it to actually because we have to transfer our footage from REDCODE RAW to REDCODE RGB or other editing format (unless I've misunderstood something along the way). And I doubt anyone is going to want the brain damage of managing their footage in clips no larger than 4GB.

 

[Chris Kenny]

Hopefully it does hide such things... I would expect it to actually because we have to transfer our footage from REDCODE RAW to REDCODE RGB or other editing format (unless I've misunderstood something along the way). And I doubt anyone is going to want the brain damage of managing their footage in clips no larger than 4GB.

Right, but it's conceivable it doesn't have a 'copy' mode, but only a 'transcode' mode. If you can just copy footage straight off a single partition in appropriately sized files, this is fine, because you can just use the file manager to copy your 'digital negative'. But if there are going to be multiple partitions and split files, you'll want to do copies though REDCINE to hide all that stuff.

Actually, you probably want a 'copy' mode even if you don't have to hide file system limitations, so you can automate the process of copying, verifying and possibly reformatting. Red is probably ahead of me on this stuff anyway, though.

Actually... here's a request for a less obvious feature. Give REDCINE a copy mode that lets it write footage to two targets at once. This should let you end an ingest with your footage already backed up, with no significant time penalty (if you're writing to at least two physical drives, of course). Yes, there's RAID mirroring, but that doesn't protect against all the failure modes that writing to two independent volumes does.

One reason why this is desirable is because it means even the cautious can erase their digital mags and return them to shooting immediately after ingest (otherwise they're going to wait for a backup of the ingested footage to finish before they wipe the digital mag). And, actually, if you could do things this way, you could confidently set REDCINE to automatically erase a mag after ingest/verification. This would mean you'd almost never have to manually format mags, which would be desirable because if erasing them is a manual process, someone, somewhere, is going to have a really expensive "oops" moment.

 

[Rob Lohman]

Frankly, in this application, the answer is because it probably beats messing with 10 partitions on a 300 GB REDDRIVE. Unless REDCINE does, in fact, hide all of this nastiness from the user, in which case smaller partitions would be fine.

As indicated before the 32 GB is an artificial limit, and yes, I have the FAT32 spec as well. It will be a single partition. Obviously file size limit is still there.

Writing and installing file system drivers is a whole other ballgame. UDF has similar problems since it is not supported on a hard drive out of the box on some Operating Systems. Yes, this has been tested and verified.

I don't get the earlier remark about not trusting NTFS on a large volume (I can't speak for HPFS). It's an extremely reliable journaling file system. I've used it for over 10 years and the server market trusts it for a good reason.

We all feel the same as you do, but it is what it is. We need a system where you can copy data off without the need to install file system drivers and whatnot. This has all been thought out, discussed and debated for months. It's not a simple decision.

Finally: we are looking at the complete workflow so it will be as smooth and easy as we can make it. There will be tools to assist. Everything about the product is modular so we can upgrade (on both ends) if something better comes along. For now there's just not much choice.

 

[Blair S. Paulsen]

Backups please

Backups please

I really appreciate Rob giving us the scoop on how much thought was put into the file system selection. It seems to me that the biggest headache is that coding RedCine will be tougher. RedTeam coders, I hope you get to see your families again someday...

One feature request that slipped into this thread that I want to echo is for running multiple backups of the DCN from RedCine. The RedTeam is probably all over this but the file system discussion got me thinking just how many issues could crop up in the very critical backing up task. Making back-ups simple and solid is a deal-maker/deal-breaker issue during the transition to the no tape, no film workflow - bungle this and we got BIG trouble.

 

[Rob Lohman]

I had to say goodbye to my family when I joined RED, but I got a new one in return!

REDCINE coding is not really complicated by this. It will load a sequence of camera files as one movie (or not).

 

[Jeff Kilgroe]

As indicated before the 32 GB is an artificial limit, and yes, I have the FAT32 spec as well. It will be a single partition. Obviously file size limit is still there.

OK. I turst you guys to do your thing... So I will also assume that REDCINE will include an interface to format drives and FLASH media?Windows requires command line intervention on the users' part to format larger than 32GB.

Writing and installing file system drivers is a whole other ballgame. UDF has similar problems since it is not supported on a hard drive out of the box on some Operating Systems. Yes, this has been tested and verified.

I personally haven't run into any systems that don't support UDF on hard drive out of the box for general use, but then again, I'm just using it with my own XP/Vista installs, OSX 10.4 (10.5 beta)... Some Linux distributions include UDF formatting abilities -- I do my UDF formats with SUSE Linux. But you're right, this is an issue.. You would have to provide your own format utility and other disk tools. UDF support in OSX and Windows is fairly minimal. In fact, Windows requires a registry mod to use UDF as you would a normal hard drive. Otherwise it tries to treat it like packet-write optical media.

Ideally, it's not a good situation just yet. I do think UDF will replace FAT32 in the coming years unless something miraculously better comes along that is cross-platform. Microsoft is fully committed to NTFS (which is fine I guess), but it's too bad they won't share full licensing with everyone else, citing security as the reason.

I don't get the earlier remark about not trusting NTFS on a large volume (I can't speak for HPFS). It's an extremely reliable journaling file system. I've used it for over 10 years and the server market trusts it for a good reason.

I don't trust NTFS compared to something more robust like ZFS. NTFS is a fine file system and has excellent security features and journaling abilities. But it also lacks a lot of the self-managing features that go well beyond journaling that are available in a system like ZFS. Having managed a couple SAN systems in a large datacenter a couple years back, I can say that the Sun servers running ZFS for the SAN were far less problematic and required a lot less user interaction than Windows server systems with NTFS. HFS/HPFS on OSX is pretty equivalent to NTFS... It lacks a lot of the secuirty features, but I think the journaling is more efficeint and it's a lot easier and friendly to implement on other platforms (never mind Microsoft doesn't release license to NTFS for any purpose other than read-only). But Apple is incorporating ZFS support on a basic level within OSX 10.5 and on a much higher level with upcoming OSX server versions and XSan software.

Everything about the product is modular so we can upgrade (on both ends) if something better comes along. For now there's just not much choice.

I know I'm beating a dead horse with this discussion... And I'm mostly just fantasizing about the way it should be rather than the way it is. I know you guys will make the workflow as efficeint and reliable as possible.

 

[GlennChan]

And I doubt anyone is going to want the brain damage of managing their footage in clips no larger than 4GB.
You know what? It's not that hard. Having to deal with the annoying PC/NTFS versus OS X/HFS issue... FAT32 works. The only thing that doesn't work is when you try to make files that go right up to the 4GB limit. Vegas will try to save marker and region information in the originating AVI file, but can't because there's no room left (this is my guess anyways; slightly lower than 4GB seems to fix things).

Other than that, dealing with a bunch of clips instead of one isn't hard at all.

 

[Poi Boy]

I could care less what kind,how conveniently available, what brand,what standard,...does it work ? that is all I care about. I think you all are forgeting what we are talking about here. 4K for $30K or so...all these details fall into the whatever catagoty.
Aloha
-A