Red and Linux

[MartyMcFly]

I imagine that when Linux has a mature NLE available, REDCine for Linux will happen naturally, especially so when you consider how similar OSX and Linux is from a developer point of view.

 

[oldphart]

Yeah, I actually run Ubuntu, and have been watching Studio since it was announced back at the first of the year. Unfortunately, their NLE of choice is Cinelerra. Unfortunate, because Cinelerra lacks some real finesse. It reports to do a lot of cool stuff, but when it comes down to actually using it, its not a pleasant experience (when it doesn't crash).

I have a friend who has done some QA on Cinelerra, and while he is rather upset about the current state he said there is hope for serious improvements in the future.

I am running a couple of Macs, a non-Mac BSD system, several Windows systems, many different Linux systems and even an OS/2 system. They all have their specific advantages and weak points.

One nasty catch with Linux is the poor Out Of Memory-handling. This will cause applications to crash when some process tries to use memory which is not available, and is one reason I would prefer to run something like the improved Cinelerra on BSD. On the other hand, adding plenty of RAM is generally a good option in any case.

Windows still does not scale well on multiprocessor systems and the memory management is not impressive (although it does not kill random applications in OOM situations like Linux).

Anyhow, I use Vegas and Avid Express Pro on Windows despite all my Linux systems - but I'm just an amateur in editing, so I do not master the tools well enough to have a qualified opinion about usability. What I really want, is an open system where I can add my own special processing, work in non-integer coordinate systems with non-integer frame numbers and tinker until my code does what I want.

 

[Ryan Damm]

I use Cinelerra on Ubuntu in a professional capacity. (I do not, however, generally sit with clients.)

What it lacks in finesse, or look, or even reliability, it makes up for in speed of use and compatibility with certain choice command-line video tools I've grown to rely on.

And for what it's worth, I would love an all-Linux solution. Currently, I may have to have my employer pick up a Mac for me and teach me to use it. Otherwise, I can stay in Linux and take advantage of Cinelerra's built-in render farming -- if I can, I'll be getting some virtualized cores on a screaming server to handle 4k.

Or it's final cut, I guess.

 

[Dan Hudgins]

Ubuntu Network Question?

Ubuntu Network Question?

I am planing on running a port of my Digital Cinema post production software to run under Ubuntu.

I plan on using Ubuntu for two main things:

1) to run frame cookers to do the color correction and filtering uncompressed.

2) store the raw, intermediate, and finished large size frames and maybe some of the "proxy" size frames being used with my Edit list command.

What software should I download (and the URL) to make a network of about 20 computers where two or three run Windows 98SE and the others run Ubuntu so that I can move frames quickly between the computers?

The Ubuntu computers will have maybe 1.5TB each, and the W98SE computers about 100 to 300MB each.

Can I use the LAN network connector on the mother board? And do I dasy chain the computers with two LAN ports, or how do I make a server that feeds 10 to 20 computers? What do all the wires plug into? How do I tell the computers to open/mount a drive or port on the other computers?

Sorry for the nube questions, but I have not set up a network of this kind before mixing OS types.

Will FTP applications work as a file manager?

It would be best if I could move files much faster than 100MB/sec (Mega Bytes, not Mega bits)...

Thanks for any advice about this since I need to spread the wrokload and storage over several computers to work all uncompressed...

I am sure others will be looking for places to keep frames as their HD fill up very fast... The speed of moving files around will be a big issue for me I am sure...


>>Note, Some tasks or programs used require Windows ME (tm) or Windows XP Home SP3 (tm) with the FreeDOS FAT32 (tm) disk mounted as a slave drive. Things may work better if all disks are formatted FAT32, which should work up to 2TB per disk.<<

UPDATE: DANCAD87.EXE (tm) now supports 16bit RGB 48bpp TIF files for import and export, so you can use REDCINE (tm) to de-Bayer the images and save the frames as 16bit TIF REDLog into the project folder my programs use. Then process RED ONE (tm) footage and do color correction, then save the edited frames back out with high quality as 16 bit TIF frame files.

 

[Jeff Kilgroe]

You have computers running Win98SE? Damn, that wasn't even a good thing back when Win98SE was a current OS, 9 years ago. I'm sorry my post wasn't more constructive... I'm still trying to understand the paradox and why you wouldn't just move to Linux on those systems.

 

[Jeff Kilgroe]

FTP can work. But SAMBA would be better as you can use it to map drive volumes and support Windows naming and Windows workgroups and not have to deal with FTP.

Yes, use the ethernet (LAN) connector on the motherboard. Provided your systems all have the proper drivers installed and everything is running fine. Also make sure the integrated network ports are 100Mbps full duplex capable. In this day and age, most network controllers are 1000Mbps (gigabit), but hey, you're talking about systems that run Win98SE. If you want to move files faster than 100MB/sec (MegaBytes, not megabits), then you will need "gigabit" ethernet adapters (1000Mbps) for each system that you want to have that bandwidth and a network switch to match.

You can't daisy chain them... Unless you have older coaxial-based network 10-BaseT interfaces and that isn't going to be what you want for any modern network that will be handling data of this type. Get a good network switch that has plenty of internal bandwidth to support all attached systems. You plug each system into its own port on the switch.

Why is it you want to work all uncompressed?

If you want to go with gigabit ethernet, prices start adding up. Not too expensive, but if your systems are older and you have to buy network cards, then it adds up. Also you have to be realistic about transfer rates of all your hardware too. Gigabit is still a max real-world rate of about 112MB/sec and to take advantage of that, your drives will have to be able to keep up... Your systems too... 133MB/s is the theoretical max of a bus-master PCI slot.

 

[Antoine Baumann]

why not run "ubuntu server" (it is debian based) on your server.

and yes SAMBA is your friend.

antoine.

 

[Dan Hudgins]

About network on mixed OS and Ubuntu...

About network on mixed OS and Ubuntu...

Thanks for the input on setting up a multi-OS network for Digital Cinema file processing.

The W98SE systems would be 2GHz+, the Ubuntu would be low end systems but maybe about 2GHz also.

The reason for uncompressed is going film scan to film recorder to keep what we can, speed is not a major issue, but being able to post in less that a year would be good. Some files may be able to be ZIPPED to reduce the disk space by maybe 30% that could reduce the HD space by 5-8TB maybe.

Development tools for conversion of my 300,000 lines of Pascal code are not good as far as I have been able to find, and if I have something that works with our cinema hardware, I need to balance software development with making films, making films is the goal. People always say why not update, but many of the things that the programs do probably would not work as they need to on any other format, at least not without a huge amount of conversion and compromise. It might take 5 years to rewrite everything, and by then some other new OS would make that inoperable. Like they say, if it works don't fix it?

I would like to port everything to a newer OS, but am not sure that the program would work as well, in some bench marks I have done DOS real mode seems to work faster than the same code compiled for Windows!

It can be very hard to find out details of how to get things to work like video modes and full screen under various GUIs, I was told by top end graphics programer that the video chip companies would not give me any support for GPU doing 2D graphics. My experence trying to get information from video card companies is that they switch you to someone who knows nothing, who says they will call back, and never does.

To be clearer I think I can run the non-graphics non-port parts of the Kinema code in Terminal mode under Ubuntu, which is all that is needed for the frame cooking, and then run the graphics and port parts required for the SMPTE time code and full screen uncompressed "proxy" viewing at 24fps and grading of the key frames under W98SE since that seems to be working well enough. When I output BMP frames and make an uncompressed AVI with VirtualDub (tm) MediaPlayer chokes and goes dead for the most part, so if MS cannot get uncompressed frames to play under Windows how would I, as the code I have now runs it works maybe better than MediaPlayer for showing the frames, or at least not much different???

A bit off subject, but I was wondering what resolution uncompressed AVI you can play at 24 fps on your computer? What res and what OS version? Can anyone play 1920x1080p uncompressed AVI on the first click of the play button? If so on what kind of system? On my AMD XP 2800 I might be able to get MediaPlayer to show 160x120 uncompressed

Anyway, thanks for the info...


>>Note, Some tasks or programs used require Windows ME (tm) or Windows XP Home SP3 (tm) with the FreeDOS FAT32 (tm) disk mounted as a slave drive. Things may work better if all disks are formatted FAT32, which should work up to 2TB per disk.<<

UPDATE: DANCAD87.EXE (tm) now supports 16bit RGB 48bpp TIF files for import and export, so you can use REDCINE (tm) to de-Bayer the images and save the frames as 16bit TIF REDLog into the project folder my programs use. Then process RED ONE (tm) footage and do color correction, then save the edited frames back out with high quality as 16 bit TIF frame files.

 

[lillbullen]

Doesn't autodesks Smoke and Fire count as NLE:s? They run under Redhat now. They are expensive as hell but still they are the best I've seen.

 

[Rob Lohman]

in some bench marks I have done DOS real mode seems to work faster than the same code compiled for Windows!

That's quite odd, unless you have other stuff going on in the background. Usually things like I/O (including disk & network) performance is much better in Windows, unless you write your own "OS" (which you can sort of do under DOS) with drivers, but even then I doubt it's worth the effort.

It can be very hard to find out details of how to get things to work like video modes and full screen under various GUIs

If cross platform probably OpenGL. If under Windows it's quite easy to use one of the native API's, I got a fullscreen playback system going under Windows in a couple of hours.

I was told by top end graphics programer that the video chip companies would not give me any support for GPU doing 2D graphics. My experence trying to get information from video card companies is that they switch you to someone who knows nothing, who says they will call back, and never does.

That's why you have drivers & standard API's

Can anyone play 1920x1080p uncompressed AVI on the first click of the play button? If so on what kind of system? On my AMD XP 2800 I might be able to get MediaPlayer to show 160x120 uncompressed

Uncompressed only cares about disk speed and memory & bus bandwidth. None of this will work good on older systems since they don't have the bandwidth to support it. That's why there's faster busses with fast RAID cards these days etc.

It sounds like you're doing a lot of custom work that might be available off the shelf (maybe combined with some custom glue).

Sure, stick with what works. But Win98 doesn't support NTFS for example, or larger drives, or probably new network stuff like (10) gig-E etc...

 

[Dylan Reeve]

I think the biggest driver for Linux support of RED would be Autodesk's products, but we'll have to wait and see if that happens. So far there doesn't seem to be enough demand from that angle to get a P2 driver for Linux, so who knows whether there will be demand for RED tools in Linux from Autodesk users...

As for opensource tools in Linux - the most promising has been Jahshaka, but it doesn't seem to have made much progress in the last year or so. Cinelerra works but is often described as 'clunky'. But commercial support for Linux as a workstation and rendering platform is strong and growing.

 

[Dan Hudgins]

About APIs

About APIs

That's quite odd, unless you have other stuff going on in the background. Usually things like I/O (including disk & network) performance is much better in Windows, unless you write your own "OS" (which you can sort of do under DOS) with drivers, but even then I doubt it's worth the effort.

Windows puts noise on the CPU all the time, there is no 100% suspend background, you can hear the stepper motors pausing when my CAM programs run under Windows. Under FreeDOS (tm) the motors run smoother since one of the main interrupt is just the time and day clock.

If cross platform probably OpenGL. If under Windows it's quite easy to use one of the native API's, I got a fullscreen playback system going under Windows in a couple of hours.

Could you please mention what resolution you were able to play back at 24 frames per second without glitches, i.e. 1920x1080p, 2k, 4k? And from what file type, i.e. JPG2000 or uncompressed?

Were the frames in Memory or read from disk?

How much memory was required to play 120 minutes of frames?

What kind of HD was used and how many GHz was the CPU?

In Pascal I do not know of a good development environment for OpenGL that runs under Ubuntu, FreePascal (tm) seems to have some issues and a lack of useful examples maybe. What compiler would you recommend for development of a complex editing/sound mixing system for Ubuntu with full screen play back uncompressed at high resolution at 24 frames per second?

That's why you have drivers & standard API's

Not all development tools can make useful access to them...

Uncompressed only cares about disk speed and memory & bus bandwidth. None of this will work good on older systems since they don't have the bandwidth to support it. That's why there's faster busses with fast RAID cards these days etc.

Yes, with a fast computer and RAID 0 my system should be usable as is?

It sounds like you're doing a lot of custom work that might be available off the shelf (maybe combined with some custom glue).

Maybe, but I have to interface to the hardware we have, and work uncompressed all the way through, and anyway it is done, I would like to make it more slick, but just need more places to stick frames, frames, and more frames, hence the need for a network...

What would it cost total to use off the shelf products to edit uncompressed 4.5k 120 minute feature including the software and hardware?

Sure, stick with what works. But Win98 doesn't support NTFS for example, or larger drives, or probably new network stuff like (10) gig-E etc...

I have overcome the 65534 file per folder limit in W98SE with my divided filename format, it also overcomes the 2GB DOS file limit, so I can run hours of frames non-stop with only 16MB of RAM in the computer. It also fixes the speed slow down problem with folders that have more than 1000 files in them.

W98SE does not need to support gig-E if I only hold the "proxy" frames on those systems, along with the keyframes at full res for the grading. If I hold the large size frames on the Ubuntu computers they should have large drive support and the latest network support, since the frames will be cooked on the Ubuntu computers the limits of W98SE are not a large handicap. If the raw scans are erased I would not need as much storage space, and if I zip files that are not in use I might be able to save more space. Maybe I can get away with 10TB or less if I backup frames not in use on DVD disks.

The bigger question is would anyone else who is not well off as I am want to make a feature film, and if they did would it ever get any kind of showing in theaters? The odds are against any filmmaker, and putting the money he has on the screen rather than into lab fees was my goal in sharing.

My system is "freeish", works as far as I know, does 4k or more uncompressed Digital Intermedate with sound mixing and film out, and can work on a typical PC of 2GHz or better if you shuttle the frames on and off using DVD disks. Yes it is slow in some tasks right now, but if you have more time than money you could save tens if not hundreds of thousands of dollars in the production of a feature film using it. If other tools are available that the filmmaker can afford, and taking that money away from the production values will improve the odds of the film getting distribution then they should of coarse go that route.

I appreciate the comments, but a pratical road map would entail specific resolution of many complex issues that on the surface may seem simpler that they are, although I would very much welcome insight...

I do look forward to having import of uncompressed RAW Bayer data at full resolution from the RED (tm) camera, hopefully by the time that is avalable I will have some kind of network to store and process the frames on.


>>Note, Some tasks or programs used require Windows ME (tm) or Windows XP Home SP3 (tm) with the FreeDOS FAT32 (tm) disk mounted as a slave drive. Things may work better if all disks are formatted FAT32, which should work up to 2TB per disk.<<

UPDATE: DANCAD87.EXE (tm) now supports 16bit RGB 48bpp TIF files for import and export, so you can use REDCINE (tm) to de-Bayer the images and save the frames as 16bit TIF REDLog into the project folder my programs use. Then process RED ONE (tm) footage and do color correction, then save the edited frames back out with high quality as 16 bit TIF frame files.

 

[Kevin Halverson]

Windows puts noise on the CPU all the time, there is no 100% suspend background, you can hear the stepper motors pausing when my CAM programs run under Windows. Under FreeDOS (tm) the motors run smoother since one of the main interrupt is just the time and day clock.

It really sounds like you should be using a RTOS for machine control applications such as that. Any consumer OS is so excessively bloated in comparison that they just can't compete on anything but massively higher performance hardware. If you really want performance, machine level on application specific hardware is the way to go. Its really nice to know every instruction and exactly how long any routine will take (and you can now down to the individual clock cycle).

Sorry, I am a hardware engineer so I always avoid OS things whenever possible. There fine for man-machine interfaces, but if you really want to get work done, there is no substitute for a purpose built piece of hardware.

 

[Jeff Kilgroe]

Dan...

You're not going to find much of anything that's going to work with PASCAL compilers, they're relics of history at this point. If you intend to keep supporting your software and/or developing new tools, you need to step up to a decent and supported C/C++ compiler or potentially Java, depending on your application and intended platforms.

I'm still trying to comprehend your workflow and just what you have to gain by using this whole approach. I realize you have your own software and it's dated, and there's no possible way you're making efficient use of current hardware with it. I'm trying to understand how off the shelf software, with possibly some custom additions couldn't be more productive when run on more current hardware.

Of all the systems you have listed, you don't have anything powerful enough to even think of digesting uncompressed RAW from a RED camera. In fact you don't even have any systems (that you have mentioned) that can even run REDCINE to deal with REDCODE.

You talk of working with uncompressed. Why? What is your goal with this footage and frame files?

You're asking about playback of uncompressed streams in real time. Here's something for you to chew on. 1920x1080p uncompressed "10 bit" HD is 7.4MB per frame. That's 178MB/s bandwidth required from your storage systems and any data path that information must cross in order to reach your display. Of the systems you have mentioned you own, none are capable of this without special hardware and futzing around with it to the point where it makes no sense vs. just buying a new system.

Don't let 178MB/s fool you, it's a bigger number than you think. That's nearly double the bandwidth capability of Firewire 800. It requires 2.5Gbps (300MB SATA-II) or 360MB/s capable SCSI or SAS or similar for a storage interface. Your existing system with a "RAID 0" is not going to do it, assuming you're speaking of that Athlon XP 2800 system. Perhaps if it's based on a more server/workstation oriented motherboard, you may have a PCI-X slot or oddball 64bit PCI slot into which you can install a capable SATA RAID controller. But there's plenty of reasons that you should really look at other options, because the disk subsystem is only one issue you will encounter. And remember that I said it won't load REDCINE.

Oh, and if you want to capture uncompressed RAW data from a RED camera, you will need a system and RAID capable of connecting to single-mode fiber 10Gbps Ethernet. And for comparison, the 178MB/s I mentioned above for 1080p 10bit HD uncompressed... That 178MB/s is only about 15% of the bandwidth you will need for uncompressed RAW from a a RED One! We're talking a RAID the size of an under-counter dishwasher (at least) and network interface cards that cost upwards of $7500. That RAID, or more likely that system with capable RAID storage, will probably run you somewhere in the $85K to$100K price range.

 

[Dan Hudgins]

About about,

About about,

Dan...
decent and supported C/C++ compiler or potentially Java,

What supported C/C++ compiler, does VisualStudio just output .net files now?

I do not understand why you think I need to convert the program to anything, as it does the job now, if slowly, and moving the frame cookers to Ubuntu should take care of most of that speed issue. The answer is that here are no tools that will work for me, so I just need to make due and get some work done, which anyone who wants to can also now do also.

I'm still trying to comprehend your workflow

Please read my other posts. The Kinema Edit list command was desigined to edit feature motion picture sound tracks with up to 512 tracks going to 16 output channels. Each of 10000 shots has 256 tracks, and there are 256 tracks that run the project length for after picture lock tracks running maybe 88 minutes to 180 minutes. I wrote this to eliminate the use of 35mm magfilm in the mixing of the soundtrack. The output goes directly to an optical film recorder to electro print the sound on the motion picture print stock.

In order to sync the sound there is a edit list to edit the shots, and a shot viewer, the Pick, command that lets you view the edited part of each shot with a sync sound mix for that shot. There is another viewer that lets you view edited sequences.

Since feature film sound tracks are mostly foley there are noise gates, pull up/pull down, and compressors that run in software to do the mix automaticly.

The movie cameras I can use use sync motors or pilot tone, so the starts are marked by slate clap. Each set of scanned frames, or digital frames, starts with a numbered set for each shot, with frame 0 being the slate clap frame. All the shots are copied into the file directory structure. Each image frame number mates with as many as 11 other "proxy" or color corected image frames derived from the original high resolution frame, and up to 256 audio frames that make up the sound for that image frame.

There is total positive sync between the audio and the image frames that is 100% sample locked sync over the full length of the projects 88 to 180 or so minutes. The live on monitor playback may dift a little because of HD/SMPTE issues, but the sync playback from the WAV file to the optical recorder should be in good phase lock on the 60Hz pilot tone driving the optical recorder sync motor.

Because I had the edit list, mix, and on screen "proxy" workprint I added some features to output frames in edit order to make a DVD by way of using VirtualDub (tm) to join the WAV mix with the BMP down res files. My grading can resize and color correct to make the 24 bpp BMP files to make the DVD from.

In addition since the film scans were required to make the digital workprint it was decided to scan the film at high resolution both in size and color depth and let the Edit list output frames or a frame list for the edited shots to another program that can operate a monitor at up to 2048x1536x32 to output to a film recorder built to output directly to color print stock. The sound would also be electro-printed.

The goal was to make a low cost Digital Intermediate system that alows the avoiding of having to cut the negative and the cost of a negative cutter, make a printing negative, or have lab printing. Since the system allows input of digital frames from any source files generated by the RED (tm) camera that can been stored on other systems could be copied into the Edit List by using a removable harddisk. Real time conection to the RED (tm) camera was not a requirement or goal, except perhaps through the use of a SDRAM memory module to record up to 30 seconds for later non real time download. Using a SDRAM module with 12 SDRAM cards running in parallel being feed from 64 ot 128 bit shift regesters would eliminate the need for any kind of RAID HD stack to aquire and edit 4.5K RAW BAYER footage from the red camera. The RED (tm) camera only needs a 15 pin D connector on it with the 12 data bits, pos/neg. and pixel clock. You do not need more than 30 seconds per shot to make a feature film, and you can have four or more SDRAM modules to shoot with in rotation as they are downloaded off the camera.

The workflow due to the size of the RAW Bayer files is easer than the about 4k RGB 48 bit files, so the RED (tm) 4.5K RAW Bayer files is less hard than working with the files I have anyway from our Oxberry film scanner.

It would not take much to get the RED (tm) camera outputing RAW BAYER data and editing it would not be any more of a problem than editing film scans which is done already.

...don't have anything powerful enough to even think of digesting uncompressed RAW from a RED camera. ...don't even have any systems ...that can even run REDCINE to deal with REDCODE.

I do not need REDCINE or REDCODE to work with, they do not output RAW UNCOMPRESSED BAYER data from the sensor, as others have told me here.

What the camera needs is 15 pin D connector that can be connected to an array of computer SDRAM arranged to divide the bandwidth by 64 or 128, this is low cost memory which is why it is used in computers. Since the modules are erased after a few shots and reused they do not need to be large, and with the cost of film would probably recover their cost the first week of use.

The SDRAM modules would be downloaded in non-real time and so could be downloaded to any AT compatable with a harddrive large enough to hold a few shots. Movie shots average 5 to 7 seconds, with data packing it is possable to store the frames from a shot on one or two DVD disks, so harddisk space can be recovered in the field by off loading the frames from the field computer to DVD disks, or USB external harddrives.

...uncompressed. Why? What is your goal ..?

The "footage" is to be output to print stock. Direct output print stock is sharper than any print can be on a production contact printer, or a step contact printer, or most optical printers.

You're asking about playback of uncompressed streams in real time. ...
Don't let 178MB/s fool you, ... And remember that I said it won't load REDCINE.

Sorry for not being clear, I was asking specificly what resolution, and file-data type the code you wrote was displaying full screen, and at what frame rate, and if it was buffered in RAM (and how much RAM) or coming from the HD?

...if you want to capture uncompressed RAW data from a RED camera, ... the $85K to$100K price range.

I do not need fiber optics, RAID harddisks, or $100K anything. There is no RAW port now as far as I have been told, and as I mentioned above a 15 pin connector and some SDRAM chips would do the job for a fraction of what you quote, and better since HD are subject to dropped frames, and other losses.

Anyway, all I as saying, is that someone could send me a DVD with the RAW Bayer data on it packed 2 pixels in three bytes, pixels end to end without metadata, and I could write a de-Bayer to 48 bit RGB that would let users process the RED (tm) camera sensor data without compression.

The signal bandwith of the 12 bit parallel data can go through short wires so optical fibers are not needed.

4096x2048x24=201Mpixels/second or 100.5MHz on the 12 data lines.

That is less than SVGA speed since the frame rate is lower. If the data rate is uneven, the frequency after going through the shift regester is still low enough to record in SDRAM or Flash.

If you shift regester and latch the 201M/64=3.2Mwrites/sec

SDRAM can take 3.2Mwrites/sec. Even compact flash cards can take more than that.

You could probably just parallel 12 compact flash cards and record,

at 201Mpixels/second, 16GB*8bits (x12) gives you 128000Mbitsperplane/201Mpixels/sec= 636 seconds = 10 minutes.

10 minutes per shot is more than enough, it is more than you get on a 100 foot load in a Eyemo, and that is plenty to do the job.

If you get about ten minutes per 16GBx12 = 192GB, on a 500GB USB drive you can store 26 minutes of RAW Bayer data at 4k.

4096x2048x1.5=12.6MB per frame, 12.6x24x60=18.2GB/minute

500GB/18.2=27minutes when packed 2 pixels in 3 bytes.

If you use 12x2GB flash cards you still get 1.2 minutes which is enough to shoot with.

Sorry I was not clearer, I was looking to derive high quality uncompressed 48 bit RGB frames from the 12 bit RAW sensor data.

The computers I was talking about are more than enough to store frames from a shot and work with those files. Almost everybody reading this has worked with image files that size on their computer. The files are not much larger than those made my the film scanner, infact they are about the same or smaller than the files made by a Canon XTi (tm) DSLR...

If standard Flash cards are used a USB Flash card reader could read each card in turn and a simple program could join the 12 files into a single Bayer data file for processing in another simple program to make the RGB files.

I think what you have described is making something much more complex and costly than it needs to be.

>>Note, Some tasks or programs used require Windows ME (tm) or Windows XP Home SP3 (tm) with the FreeDOS FAT32 (tm) disk mounted as a slave drive. Things may work better if all disks are formatted FAT32, which should work up to 2TB per disk.<<

UPDATE: DANCAD87.EXE (tm) now supports 16bit RGB 48bpp TIF files for import and export, so you can use REDCINE (tm) to de-Bayer the images and save the frames as 16bit TIF REDLog into the project folder my programs use. Then process RED ONE (tm) footage and do color correction, then save the edited frames back out with high quality as 16 bit TIF frame files.

 

[Jeff Kilgroe]

I think what you have described is making something much more complex and costly than it needs to be.

Funny thing is, I'm thinking the same thing about what you're saying. :biggrin: I guess we agree to disagree. I have a different outlook and that's get in, get it done and move on. If I or someone I'm paying has to shuffle large amounts of data like you're talking about around on DVDs or across slow, antiquated networks, etc.. I'd probably go completely insane. Or worse.

You mention a network to store files. Yep, many of us already are working with such a thing. It's called a SAN (Storage Area Network) and ubuntu is not the answer for anything on a serious level. There are dedicated SAN server modules out there for major operating systems that coincide with redundant and mission-critical storage hardware. More expensive than a bunch of 3 year old computers running ubuntu? Yes, for sure. But in my view, time is money and time is often my most valuable commodity.

As for your RAM buffer for capturing uncompressed. Sure, it's been discussed on these forums more than a few times. The idea has merit, but I think it targets a very limited audience. And so far I haven't seen a description of a RAM buffer that would make sense to build... If recording was restricted only to 4K 2:1 @ 24fps, that's roughly 290MB/s for uncompressed RAW assuming 12bpc sensor data. That is manageable. like you said, an array of a dozen CF cards could probably pull it off. Roughly 24MB/s per CF card for sustained rate. Provided a decent memory controller can be built to minimize the overhead. There's also a slim chance that the camera may be able to push that out over the SATA bus rather than requiring another proprietary and expensive module to mount to the side like the RAW port. But 4.5K x 2.5K @ 60fps. That's roughly 1GB/s and a whole other animal.

 

[Rob Lohman]

Could you please mention what resolution you were able to play back at 24 frames per second without glitches, i.e. 1920x1080p, 2k, 4k?

2K for display, 4K downsampled to 2K or cropped to 2K.

And from what file type, i.e. JPG2000 or uncompressed?

DPX (uncompressed)

Were the frames in Memory or read from disk?

Disk initially, some were cached in memory

How much memory was required to play 120 minutes of frames?

Didn't check memory requirements, don't really care about that these days. Memory's cheaper than the meals you eat (don't take this literally)

What kind of HD was used and how many GHz was the CPU?

A big fat RAID array. CPU was a modern one, probably dual-core

What compiler would you recommend for development of a complex editing/sound mixing system for Ubuntu with full screen play back uncompressed at high resolution at 24 frames per second?

Intel C++ compiler for Linux

Not all development tools can make useful access to them...

That's why software development tools advance as well...

Maybe, but I have to interface to the hardware we have, and work uncompressed all the way through, and anyway it is done, I would like to make it more slick, but just need more places to stick frames, frames, and more frames, hence the need for a network...

To a lot of post applications you can create custom device controls. Or what happens a lot is that you write tools to do playback / capture through something custom while working with standard tools in the middle.

so I can run hours of frames non-stop with only 16MB of RAM in the computer.

I come from a demo scene and assembly background so I understand what you're doing and trying to say. It's just not really relevant anymore these days (outside of don't break what works).

Your custom stuff for Win98 sounds pretty scary

Anyway, I long lost track of what this thread is all about. In the end you need to do what you want to do and works best for you. I'm not trying to change your mind on anything, just giving you some thoughts....

Best,

 

[Rob Lohman]

What supported C/C++ compiler, does VisualStudio just output .net files now?

Visual Studio (when doing C++ projects) still outputs native code in both x86 and x64. Obviously you can also do .NET with C++, but that's totally up to you.

Visual Studio 2005 did add manifests though which can be a pain in the ass... but VS 2003 is more than good enough for 99% of your things outside of .NET.

 

[chocblu]

Hey Dan,

I Find it kinda Mad that you know that much about programming and all but struggle to work out how to set up a network. Your obviously not scared to learn, so in regards to setting up a highspeed network server for as little money as possible, i would look into a few things. Most of these are made by Sun and all of them are now free and opensource. Always a good thing. I'll just concentrate on the software as at least some part of the hardware was discussed. Make it a switched network and gigabit. Gigabit cards are pretty cheap now (i think an intel one is about $60 AUS) and a switch (make sure its a switch) is only a couple of hundred.

1. ZFS file system. This is the big daddy of files systems out there (well non niche ones anyway) 128bit, copy on write, variable stripe for raid, does every type of raid in software that you can think of and is FAST. With enough drives running in parallel you could make the speeds that you need to get while having your data nice and safe. Also because its a file system/volume manager you can increase the size's of your hdd arrays without any kind of rebuilding. And rebuild times are much less than conventional hard drive raids.

2. Samba. Just use this to share all of your files out. Its defacto, it works, its cross platform, have yet to run into a system that wont use it. Oh and it really is easier to set up that windows file shareing, especially on win98se.

3.Opensolaris has a way where you can bind a group of network cards together, to not only get fault tolerance but also to gain speed. So therefore you could get 4 gigabit network cards, bind them together and get 4gigabit network bandwidth (obviously some overhead but probably worth it).

4. Seeing as you seem to want the ubuntu servers really just as glorified hdd, you may want to look into iSCSI. you might not either but basically what it can do is have a network share that looks to the client like a local disk. Im not sure whether win98se has an iSCSI client. I would guess not, but you dont seem to have any problem with coding, and its an open standard....

5. Dtrace. If your the kind of coder you sound like, your going to wet yourself over this. Again havent really done much of it myself but it sounds like an invaluable tool. Basically its a low level Kernel and userspace debuging tool and scripting language. Say you want to know how many systems calls a particular process makes, program a little script, and run it and dtrace will trace all execution and tell you how many system calls have been made. This is a very very very small example of the power that Dtrace can do. Leopard has this feature and FreeBSD ported it just a little while ago.

now most of the Solaris stuff (ZFS, network binding, and dtrace) is mature on openSolaris (OpenSolaris is the open source version) but is coming online in other linux distros and BSD versions. the main problem with opensolaris is the Hard ware support. its a bit thin on the ground, but better than the hardware compatability lists would lead you to believe.

i Hope this helps you out or at least gives you some fun stuff to look into.

On a personal note, Im absolutley blown away with what you are doing, In my mind it is absolutley mad to have attempted to do it, and amazing that you seemed to have pulled it off.

Highest regards

Mark

 

[Dan Hudgins]

CIN and DPX file batch conversion under various OS?

CIN and DPX file batch conversion under various OS?

Thanks for the great tips about the Network issues!

I tried making a Harddisk with four partitions of 29GB under W98SE as a second disk, copying some frames onto it, then re-booting with that data disk as the second Harddisk under Ubuntu. Ubuntu opened the disk after I entered my password again, I copied the frames to the Ubuntu disk, made changes with GIMP, then copied the frames back to the four partition W98SE formated disk, then re-booted with W98SE and opened the frames there and everything worked! Ubuntu seems to read/write to Windows (tm) disks without any issues I have found so far. This would let users of my system run Ubuntu with a "frame cooker" port of my stuff later for faster working on larger disks.

So just using a Harddrive with four 29GB partitions would let me swap 116GB of 4k uncompressed frames onto the Ubuntu computers for batch color correction and storage. I do plan to look into setting up a network, but it is good to know that just Harddisk to Harddisk copy will work between different OS.

If anyone can help I have another issue I need help with, although my Kinema Edit list and utility programs are self contained and able to the tasks required for scanner to film out after using some freeware to convert the Canon (tm) RAW files into BMP files, I am still looking for a freeware to convert CIN and DPX files into uncompressed tiff (for tiff to BMP) or directly to 24bpp BMP files.

I tried the version of GIMP that comes with Ubuntu and it would not open a CIN file from Kodak (tm) of the Digital LAD. Does anyone know of a freeware program that will open CIN and DPX files under Ubuntu and or W98SE and save them as uncompressed tiff or 24bpp BMP?

Batch conversion would also be a help, or some way to work with a BASH or BAT file... Reverse conversion from 24bpp BMP to CIN and DPX in a freeware program might also be a help for some tasks...

BTW, does anyone know how many TB of harddisks I can plug into each Ubuntu computer?


>>Note, Some tasks or programs used require Windows ME (tm) or Windows XP Home SP3 (tm) with the FreeDOS FAT32 (tm) disk mounted as a slave drive. Things may work better if all disks are formatted FAT32, which should work up to 2TB per disk.<<

UPDATE: DANCAD87.EXE (tm) now supports 16bit RGB 48bpp TIF files for import and export, so you can use REDCINE (tm) to de-Bayer the images and save the frames as 16bit TIF REDLog into the project folder my programs use. Then process RED ONE (tm) footage and do color correction, then save the edited frames back out with high quality as 16 bit TIF frame files.