Increased RedCode RAW data rates? Long term archiving?

[conrad gaunt]

Encoding/Decoding:

I was just wondering, would it be possible to upgrade the encoding/decoding data rates with a future software upgrade or is Red encoding/decoding hard-wired to a circuit, or limited by resolution and processor speeds etc. I don`t know if this is of interest or has been discussed before, but I could imagine I`d really enjoy being able to lower or raise data rate/quality settings as I need, especially if drive speeds/size keep increasing.

Future of `Film` archiving:

What is the future of digital film archiving? Traditional film ages over the decades, but digital lasts forever, right? Or does it? Well it probably does if you use a fail-safe raid system (so you can detect errors, as well as fix them) and then transfer it to a new raid every five years or so. Thats about twenty new raid systems every hundred years (if your lucky). Unlike film, that degrades slowly, digital copies can disappear instantly if your not carefull, or just unlucky. Then there is Power surges, Fire, Lightning, Electro Magnetic Pulses..
Files can also develop errors quietly too, unless you use a safe RAID, or keep two copies on different disks, it can be hard to fix and harder to spot errors, especially if a clip isn`t being played often (ie archived). TIP, If you do this two disk thing, i recommend you zip each copy up (yes it takes ages), not necessarily for compression (but that can be useful too), but error-detection - build with recovery records and verification data. By unzipping you can confirm its an error-free copy and replace a known damaged copy. No point replacing damaged with damaged. I use this to back up important WAV files at present. I don`t think a good solution to this archiving problem exists at present, short of engraving your work onto slabs of stone in pure binary using a very small hammer, or, and this will seem silly, transfering a copy onto film, which isn`t an exact copy anyway (or a binary copy on micro-feish perhaps?), long term survival of digital media might be tricky. I`m thinking more about historical footage than blockbuster movies. My hope is that future disk systems cost the same as they do now, but become thousands of times bigger, so I only have to replace one at any given time.
I won`t mention the problems of proprietory file formats and their future compatibility, or suitability for archiving, but it may be necessary to migrate digital media between different formats through the years, which will be interesting.

ps, I`m personally very excited at the thought of using DWT (wavelet) as apposed to DCT (blocky 8x8 mpeg style) encoding. I`ve been getting sick of red and blue being treated as second and third class colours too, viva 4:4:4

pps, If you want a lossless encoder capable of encoding YUV or RGB colour, search for HUFFYUV codec using google.

 

[GlennChan]

IMO, it might be preferable to keep the format and the data rates the same. It sucks to end up in a situation where you can't use a file because it's a variation on the format and that variation isn't supported. Sort of like the 35mbps mode on XDCAM. For Red, it might be that Redcode is so good that you aren't going to get much better quality.

The old Redcode stills on cinematography.net look pretty good... they aren't perfect (some minor image details are reduced) but it looks visually lossless and better quality than 4:2:2 uncompressed.

There are too many formats around IMO... it would be nice to keep that down unless there is a very compelling reason to use a new format/variation.

 

[Joe Carney]

I"ve wondering why MNG and FLAC haven't been looked at for long term archiving. Both are capable, have both simple and complex lossless compression options and are patent and royalty free. MNG can easily be extended to go beyond 16bit per channel color.
I'm thinking long term here, not for a single project. More for permanent archive, similar to film. Even if it needs to be stored on several high capacity Blu Ray data disks.

 

[GlennChan]

Historically...

Film has proven itself to be pretty long lasting. And even if you screw up its storage (e.g. shrinkage, scratches, etc.), the image is salvageable (through expensive image processing tricks?).

The binding material in tape will eventually degrade after 15 years.

With CDs, moisture will seep through the plastic bottom and then start corroding the aluminum layer (though I'm not sure about dye-based discs).

2- With data you need to check your data periodically and migrate your data onto new storage. And two two copies in separate locations. I believe that's the way to do it.

You also need to ensure that you can read the file system and understand the file format. (But I think there are enough threads around on archiving.)

 

[conrad gaunt]

I guess I started this thread out of my own fustration over the years with data migration. I am a programmer by nature, and drag a lot of data around. That looks set to increase how I`ve ordered a Red One. I think many indies won`t have thought seriously about long term storage issues yet, and frankly who wants to. I wonder how much digital data will survive for the documentary makers of the future, I`m sure a lot. Thats why I wondered about lower data rate option possibilities for the Red One, although I must also point out the options as already described are amazing. I gotta smoke less.

FLAC is good, but I find uncompressed WAV in a rar better compressed and can contain recovery data. FLAC is immediately playable which is good though. I wouldn`t ever trust any writable disk media. I wouldn`t trust a DVD-R (- OR + or whatever) to keep something safe for a week, and if it did I`d feel lucky. Take a WAV file for example, it has a 44 byte header. If it gets corrupt, Sound Edit etc might reject it while loading. If an error occurs anywhere else in the file, Sound Edit probably won`t mind, because the data doesn`t mis-inform the software (ie sample rate number or No. Channels data corrupt), its just PCM data, and the corruption might only last milliseconds, and not be audible (until more disk clusters etc get damaged). DVD-R errors don`t always stop the disk from playing, but they`re dying from the moment they`re born. I thought about archiving my family videos some time back and decided the best option was to get a dozen factory made DVDs made (pretty expensive), with laminated coatings. Factory made disks do last infinitely better, although like laserdiscs of old probably get disc rot eventually. Next best option in terms of reliability, it seems to me, is solid state memory sticks/cards. Good for audio I find, but small and expensive. If storing data disks in a domestic house, finding a temperature stable spot helps. DVD-R dye hates sun. Sun hates all data storage devices. The XD-CAM disc format is guaranteed for 50 years and made by TDK and a couple other companies. Its basically a BlueRay disc in a cartridge, and can apparently be used after a dunk in a canal. I won`t be testing that one out personally, even if its in a sun/water proof box, but I don`t know its a false claim. Hang on, they said 25 years on one of those DVD-Rs I bought for £3. Based on that, XD-CAM cartridges probably last two weeks. My sister and husband own an XD-CAM so I guess eventually I will find out. I find mini DV and DVCAM corrupt usually after about a year or so, or sooner sometimes (nasty green blocks aaargh). I`ve had no experience long term of large DVCAM tapes. It seems a little ironic to me that digital data can be copied losslessly back and forth for an infinite amount of time, but if left alone for only a few years can fade away. Keep backing up. backup before disk failure, and keep old disks after backup. Keep working data recovery software available for when the worst happens. It probably will eventually. Contact specialist data recovery people if the lost data is very important.

Multiple locations is a good point, even if in the same building. Downstairs is usually the best temperature environment for data storage. Also, don`t smoke in a room with a Halon 2000 fire prevention system, which can apparently suffocate fires very quickly. They had one where I worked, to protect the main frame computers and their DAT backups. If I hit a big red button it could lockdown the room too. Unfortunately that button was inside the room. Unfortunately humans need the same gases fires do, especially oxygen. I believe they`re illegal now in the UK. Never work for a company who puts data security above human life. Do trust your data with them though.

 

[J. Bernard Vallon]

If i remember right, the limiting factor in the system right now is the image processor. It can only output 30fps @ 4k redcode. The sensor itself is capable of outputing 120fps. With a processor upgrade (one day) we should be able to get 120 fps @4k redcode for 108 MB/s

 

[conrad gaunt]

If i remember right, the limiting factor in the system right now is the image processor. It can only output 30fps @ 4k redcode. The sensor itself is capable of outputing 120fps. With a processor upgrade (one day) we should be able to get 120 fps @4k redcode for 108 MB/s

I think thats right, and Redcode is temporal too, so in theory doubling framerates shouldn`t equal double data rates, but every frame will need processing fully.
...
(!CORRECTION!, it is SPATIAL compression only, NOT TEMPORAL as I wrote above. You probably knew that, and so did I ?! The rest is accurate now, again..sorry ronx),
...
Someone put a link up on the redforum to an experimental camera, very very high res, that in theory could shoot at 4000fps. People were sceptical about the technology. It got me thinking, what would happen to frame/compression ratios once ten frames in a row are visually nearly identical. Then I remembered the fastest camera in the world (well about five years ago) can shoot at exactly 50000 fps if memory serves me well (so more PAL than NTSC . If I can work out how I`ll post a filmstrip of a cartridge being fired from a shotgun and the cartridge disintergrating, complete with shot dispersing. Imagine a Bruce Lee kick thats so slow you could cook and eat a traditional english breakfast, without getting indigestion, untie his laces (as it creeps towards your face), nick his shoe, call him a wimp and then still have time to escape by missing a bus and catching the next one. A flea circus could be filmed for the first time using such technology. 120 fps sometime would be better than great though.

 

[I Bloom]

I think thats right, and Redcode is temporal too, so in theory doubling framerates shouldn`t equal double data rates, but every frame will need processing fully.

Hold the phone. Are you saying REDCODE uses interframe compression?

IBloom

 

[Rob Lohman]

Each frame is compressed on its own. There is no compression between frames.

 

[JohnF]

I agree with ronx points about archiving.

This is my big concern with RED workflows.
I'll give an example:
Some years ago now I worked on a complex FX shot that involved 26 different pieces of meterial being blended to form one smooth shot. It looked really good and I backed up the picture data and the project data onto hard disk.

The other week I needed to rebuild the shot and make some changes so I pulled out the hard disk and reloaded the footage only to find that the picture data had somehow become corrupted. This was not such a big deal as I was able to pull out the original tapes to obtain the footage and rebuild the profect.

What surprised me was that the hard disk corrupted before the tapes. (thoug it was only a mild hard disk corruption as other files from the same disk were fine)

This topic also reminds me of research carried out by the BBC(in the late 80's early 90's) into long term archiving of their video material by using film to record, frame by frame, the waveforms of the analogue video data and using some clever technique rebuild this data when required.

Film being very reliable over the long term is probably still the best solution.

What do people think of possibly mastering for long term archive their material
by printing their material to film? (providing the machines that offer this service drop in cost)

In the meantime I'm thinking along the lines of a combination of RAID storage alongside tape based data back-up.

JohnF

 

[conrad gaunt]

My appologies, I thought RedCode was temporal, it isn`t, which is good. I didn`t think it was originally, so I`m not sure where I got that one from. I do understand basic wavelet compression (simple haar functions), and should point out that DWT based squashing is seriously better for image size/quality than DCT (mpeg) temperal based destruction. Also, I should mention, I`m not worried about archiving Red footage especially, I was just rambling on about digital data storage in general, and the twenty years of data in general that I`ve been dragging around. Red footage will no doubt add. I think many people probably have DV tapes as masters and think they`re safe, In my opinion/experience they`re not. Mine are on h/d, which results in a nearly annual migration of at least some of my data. My old Lacie died two weeks ago so some of my stuff is on single external drives with no backup, until I find another TB of fault tolerance. All my Lacie drives are fine, but my laptop has decided to throw a write cache error about a minute after a file copy to the Lacie. I turned disk cache of, but still the error. I downloaded a firmware update from the Lacie website. It contained a file with a .txt extension, though its not a text file. Could be update. Contacted Lacie. They haven`t emailed. They won`t I bet. Before firewire arrived I captured at 8mb/s, about ten years ago now. Like wilderbeest, those files keep moving.

 

[I Bloom]

Each frame is compressed on its own. There is no compression between frames.

I had no doubt.

Ronx, I don't think REDCODE is a candidate for archival storage because it is a RAW Bayer format. Most likely finished films will needs be in RGB, which will have been interpolated from raw.

As for increasing or decreasing the data rate. It's hard to tell, it may be very easy for them it may be difficult. In some ways I think their compression is hardwired, though I think they use "soft" hard wiring (field programmable gate arrays). Whether or not they can adjust the parameters easily between shots is an unknown. I'd say its possible, POSSIBLE, that this adjustment could be difficult from a programming standpoint.

I'd say it's likely that they might eventually offer different levels, like, fine, coarse etc. As in DSLRs. But this is all speculation.

IBloom

 

[I Bloom]

The other week I needed to rebuild the shot and make some changes so I pulled out the hard disk and reloaded the footage only to find that the picture data had somehow become corrupted. This was not such a big deal as I was able to pull out the original tapes to obtain the footage and rebuild the profect.

Are you certain that the files were corrupted while the disk was in storage. Or is it possible that it happened at the end of your original work. It just seems to me like corruption is an unlikely result of long term storage, especially over such a short period.

IBloom

 

[JohnF]

Well that's a good question ibloom!

I'm still trying to find an answer as not only are the files not working the entire drive is now not showing up when plugged in. I'm trying to fix it but without knowing what has or is possibly causing the errors this might take some time...!
Plus that's 200GB of stored data knocked out (for the moment) which is a personal first for me as I've never lost one of my storage drives before.

As you rightly suggested the error on this drive was probably caused when the drive was last in operation rather than during the storage period(unless the HD got bashed). And this highlights the archiving issue as we're all going to have to verify our data, on a regular basis, to make sure none of our "digital negatives" are lost.

Anyone for suggesting any other method of archiving...?

JohnF

 

[mdo]

Are you certain that the files were corrupted while the disk was in storage. Or is it possible that it happened at the end of your original work. It just seems to me like corruption is an unlikely result of long term storage, especially over such a short period.

IBloom

Hard drives store the CRC of each sector immediately after the sector. If the hard drive had failed, there should have been an error returned by the OS when the CRC that was written to disk didn't match the calculated CRC. So, as pointed out, the corruption would have been due to something other than hard drive failure.

 

[conrad gaunt]

"

"

Are you certain that the files were corrupted while the disk was in storage. Or is it possible that it happened at the end of your original work. It just seems to me like corruption is an unlikely result of long term storage, especially over such a short period.

IBloom

Im not suggesting Redcode is a suitable archiving format, its proprietory so it definately isn`t. I do however keep original file captures for important stuff since difficult doesn`t always mean impossible, and formats can become opensource eventually (not that I`m suggesting that this is Reds plan).

On the subject of disk reliability, I have no doubt, after 20 years of hard disk use, that magnetic storage devices, just like regular magnets, lose, er , magnetism, and consequently data. Other problems too, including mechanical. The likelyhood of corruption during long term storage, I would say, ranges from somewhere near `likely` all the way to `inevitable`, thats why people migrate data to new drives periodically. This doen`t mean that well cared for newer disks can`t suddenly die. As everything shrinks to an atomic level (like processor circuits) their durability is somewhat untested. I have two year old disks that have failed. I always scandisk new disks. In the past many new disks did have bad sectors from day one, FAT wrote them off. This happens less I`ve noticed with modern disks, but that doesn`t mean they are necessarily equivilant in durability to older/bigger/slower disks. I`m sure I saw a news report about somebody who was using an early lightbulb for about seventy years, with its big fat filament still intact. A thinner filament makes for a higher performance (brighter) bulb, but a fat one don`t burn out so quick. Of course, bulbs normally blow while being turned on and off, and last longer if left on. I don`t think this is true of hard disks. I think they`re probably more like cars, best to turn them over occationally. Somebody mentioned the BBCs attempts to find a long term solution above which is interesting (signals stored on film). I know they have been developing their own video codecs for a while now too. In the eighties ITV (UK) had a novel idea, just throw away huge amounts of your archive material, including 'Dads Army` episodes, some of which were found later in a bin (but not all episodes), and everything recorded on SVHS (understandable lol!). Different medium, same problem, money.
Most raid systems allow hot-swappable disk setups, so you have only to swap the damaged disk (then the volume rebuilds itself), but, do I continue using my Lacie, with old disk types, or do I get a new SATA based one? (I know the answer to this - money!) Repair Lacie and use as a temp storage device, if nothing else. I have a parallel port dongle for running Newtek Lightwave, but no parallel ports anymore. Time moves forwards, don`t be over-confident all your data, hardware and technology will, even if its not being worn out through usage. They can still develop the technological version of `Bed sores`. For your benefit, I`ll disagree about just how quickly that can happen (even without UPS etc being involved), my experience is very different.

My archive approach =
1. Definately buy a raid (must have fault tolerant options),
2. When its full, buy another
(hopefully its twice or more times the size of old one because of technological improvements)
3. Copy old data onto this drive and use until its full, if you end up having three copies of data, relegate oldest drive to tempory storage status, then back to 2.

Also, I`ve used NTFS drive compression for years (on selected folders). This can increase space, and depending on data compressed, can also increase speed of reading from the disk (since the time to uncompress "? m/b" is lower than extra time needed to access "? x 1.5 m/b" of uncompressed data, because H/Ds are a mechanical bottleneck in general, whilst decompression happens in RAM) and possibly reduce mechanical disk wear(?). I wouldn`t recommend it for the sharp end of editing systems dealing with compressed video data and high CPU usage, but playback of uncompressed formats can benefit from sitting in a compressed folder.

PS, A RedCine suggestion.
I`d love Redcine to offer a 12bit packed or 16bit linear RGB raw export option (like photoshops raw, no header just an interlaced or planar RGB sequence) ,so I can archive footage and access it from software I write myself. If I can get my hands on 12/16bit linear RGB data from the camera, happy days. I`ll have no workflow issues, as I could mostly adapt my own tools written in C or Delphi for what I need. I`m used to slow rendering, even at 4k and above (timelapse, D-SLR) .If the Red team makes data access easy (exporting a raw 12/16bit rgb sequence is a trivial function to add to Redcine software. please do it Rob/Graeme etc if you haven`t already), I won`t have to worry then about dealing with system api`s/codecs etc to get at pixel data, and could provide some interesting post production tools and "intelligent" filters for free to anyone interested in effects. At the moment I`m working on a package I call "Endian Inks",and specifically, at present, "Hollywood night" effects (make day look like night), and I`m pretty pleased with the 8bits per channel results. 16 bit colour would be awesome. Go on Red, add that simple export option!

 

[conrad gaunt]

Hard drives store the CRC of each sector immediately after the sector. If the hard drive had failed, there should have been an error returned by the OS when the CRC that was written to disk didn't match the calculated CRC. So, as pointed out, the corruption would have been due to something other than hard drive failure.

Hard disk CRC and even simpler parity checks are great for identifying data corruption (as long as the CRC data doesn`t become corrupt independantly from the sector data) ,but CRC `checks` can`t repair it (usually..).. your right, the OS would have noticed an error (during the original write), but these errors don`t really concern me, because at least I know about them. Errors that occur whilst drives are not attached to a computer, or any OS, are more worrying to me. If you don`t access the file, or run a full disk check, these errors can go unnoticed for years. Not realising this is very risky, long term, if you hope to keep data reasonably safe over time, or assume it is. A disk error occurs when data becomes corrupt, however it occurs (disk/OS or memory based), and whenever (whilst being written or later whilst sitting in a desk), when it should be safe and accurately stored. I have lost data to disks dying, and the most annoying thing can be realising you will never know exactly what you`ve lost because, like most people probably, I did`t have an offline log of what is stored on the disk, through sheer file numbers.
All these things said, I do love the things H/Ds allow me to do, and I don`t want to upset the ones I have (they read all my data), incase they rise up against me. Obviously, one day, we will record directly into our brains using USB3 (wireless), and accessible by local councils. The former British prime minister was fitted with a prototype (wireless) HSUB connection so he could be controlled via the republicans (lol!). Did someone say Red One could be modified to work as an ion cannon too?

 

[Jeff Brue]

There are answers for long term data storage. It just involves actually caring fiscally about your data. My personal opinion is that LTO-3 is tops. Its robust, relativly cheap, and best yet they can't be slapped into a computer and reformmatted. That being said holographic storage is the next thing 1.6 TB at 120 MB/s is really applealing.

 

[Gavin Greenwalt]

Actually the RED datarate isn't even softcoded. I'm relatively certain it's a VBR codec. So each frame's compression is unique (within reason).

 

[Stokestack]

IMO, it might be preferable to keep the format and the data rates the same. It sucks to end up in a situation where you can't use a file because it's a variation on the format and that variation isn't supported. Sort of like the 35mbps mode on XDCAM. For Red, it might be that Redcode is so good that you aren't going to get much better quality.

It's not data-rate changes that screw things up; a properly designed system should handle that without problem. Look at DVDs; their data rate is all over the place, and even this old lame format handles it fine.

Problems occur when you start with a shoddy format that wasn't designed with the future (or perhaps even the present) in mind. So you create a hack to extend its functionality; well, no wonder it's half-assed and unreliable. Granted, this is a mistake that's made over and over again. DVD+-R DL anybody?

I don't think REDCODE is a candidate for archival storage because it is a RAW Bayer format. Most likely finished films will needs be in RGB, which will have been interpolated from raw.

Probably, but mainly because storing only the raw files would require you to also store the project files for the editing, color-correction, and compositing software, in addition to the software and Redcode itself. Then, if you needed a new render of the film someday for a new delivery medium, someone would have to know how to use that software well enough to render it.

But you'll certainly want to archive the original format also. Not only is it more space-efficient than an RGB format that would retain the full dynamic range, but it also makes future deBayering algorithm improvements possible.

But back to the original point: I too would really like to see a flexible data rate that maxes out the storage and processing capabilities at all times. I don't remember seeing any official response to questions about it, though.