Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Red Tools

M

Mark L. Pederson

REDuser Sponsor
Joined
Dec 28, 2006
Messages
7,145
Reaction score
1
Points
0
Age
59
Website
www.offhollywoodny.com
new thread from this one -

http://www.reduser.net/forum/showthread.php?t=6552

Giant Caveats:

#1) Only 98% sure I can pull this off. Been 98% sure before and been kicked in the balls hard by the 2%.

#2) Best case, end of Jan release unless resources fall from the sky.

#3) I do not know what the elves at RED are doing behind the scenes. So, if I start to see all these features in RA and REDCINE - then I'll stop trying to build it. So, for all I know - this is 75% done already ... so RED TEAM please stop me if you think I am wasting my time -

#4) I think I am nuts to do this - but I need these tools - please cheer me on if you want them too -

#5) I could fall flat on my face - ZERO PROMISES HERE


Okay - here we go:

REED TOOLS preliminary specs

FILE>Create New Database
*fields for project details, producer, director, DP, etc.

FILE>Open Existing Database

FILE>Import r3d meta-data
*select destination, imports all meta-data - including:

Filename,ReelID,Camera,Reel,Clip,Frame Width,Frame Height,FPS,Total
Frames,TOD TC Start,Edgecode Start,TOD TC End,Edgecode
End,Kelvin,Tint,Exposure,Saturation,Contrast,Brightness,GainRed,GainGreen,GainBlue + audio info TBD + thumbnail (which you can update/change by selecting different frame from file)

If and when it detects DUPLICATE FILE NAMES it checks to see if in
fact it is the same file (previously imported into database) - or in
fact - a duplicate file name which is a different file- it TAGS file (color coded in database GUI/View) and displays "Warning -Duplicate File Names Detected")

**database also "tracks" imports and exports in a LOG which keeps
sources volume and destinations with data/time-stamp - so you can
highlight any data in the database and view the "history" of how that
data got there - and what drives/volumes data was extraced from

***unlimited user definable fields in the data-base- so, for example,
you can add an "LTO3" field and enter the ID# of the LTO3 the media is
archived on, etc.

****the GUI/VIEW of the database also displays other handy derived
from all data in the database - such as TOTAL FRAMES and TOTAL MEDIA
RT (running time) for all media in database (project)

FILE>Close

FILE>Export
*(tab-delimited for use in other DB apps like FilemakerPro, etc.)

FILE>Print

TOOLS>EDL Editor
* imports an EDL and displays it in a nice column view and adds EDL FC
(frame cound in EDL based on in and out TC and timebase) and TL FC
(frame count based of clip in Timeline derived from in and out in timeline and timebase) this is what I use to detect and know how to deal with OFF SPEED clips if they are present in EDL

** The main feature here is SWAP EDGE CODE and TIME CODE of any selected REELS (or CLIPS) in the EDL in the event that you have a project with media edited with a MIX of edge code and time code in the edit. Now, you can EXPORT an REPAIRED EDL.

*** Some basic goodies here that you do in other apps - just nice to have in one place with "one click" - such as ADD HANDLES - which exports a new EDL with defined handles and COMPENSATES FOR SPEED EFFECTED CLIPS -
--

TOOLS>R3D EDITOR
TOOLS>R3D EDITOR>OPEN
you can RENAME, RE-STRIPE TC, SWAP EDGE AND TC set In and Out points - it creates a NEW .3rd in a SELECTED DESTINATION - your RED TOOLS DATA BASE is now updated to indicate (in the LOG) that this was created.

TOOLS> R3D MOVER
TOOLS> R3D MOVER>OPEN EDL
displays ONLY the R3d files used in the EDL in REEL # order.
TOOLS> R3D MOVER>COPY
select destination - moves ONLY the .3rd files used in the EDL to location of your choice
TOOLS> R3D MOVER>TRUNCATE COPY
select "handles" and destination - makes new .3rd files used in your EDL with handles, compensates for speed effected clips

**** last feature to enable as I work out a backdoor to a little issue here - but 100% can do this - but could get boned by Red changing stuff - I can get VERY FAR with just R3D MOVER>COPY (no truncate) - so, I'd get everything built before lighting up that bulb -


Feedback ...?
 
Go for it, Mark! Unless RED already has something similar in the works to build onto REDCINE.

As Gavin and I have been poking about in R3D files, I've been thinking of a few things too. Contrary to Deanan's warning, I've successfully trimmed R3D files and extracted portions. I'm sure I could join two R3D files with the same frame size, haven't done it yet, but may give it a try today....

Anyway, as I was thinking of writing a small utility to do all this, Gavin mentioned it as well, I quickly shifted to something else. How about a utility to recover and repair lost or damaged R3D files? Accidently format a CF card full of footage in the camera? ...Just run the recovery tool. Obviously, if you over-write one bunch of R3D files with other data, then recovery chances will be slim to none, but I've already seen two people posting about "oops, I erased my CF card, how can I get my clips back?".

I may take a closer look at the idea this weekend. The only problems I have with it, is R3D will continue to evolve and it may be a problem keeping up without a proper SDK from RED. The repair aspect may be a pipedream for now since I still can't identify much of the data within the file and frame headers... A REDCODE / RED SDK would sure be handy.
 
Jeff Kilgroe said:
The only problems I have with it, is R3D will continue to evolve and it may be a problem keeping up without a proper SDK from RED. The repair aspect may be a pipedream for now since I still can't identify much of the data within the file and frame headers... A REDCODE / RED SDK would sure be handy.
Click to expand...

Yes. But, at the very least, if I fail, maybe I can give Red a good idea or two.

I wouldn't mind a hand with a little Xcode GUI "sexy" factor if anyone is up for it - and maybe someone can design a cool icon - maybe a red wrench or something ...
 
Hello Mark,

That is an ambitious list of features for such a short development cycle. As I am interested in "cheering you on" and equally interested in getting a usable utility, let me list a few things that might be worth considering.

In reading the list of features, one of the first that I read that caused me to pause was the "thumbnail". Do you actually intend to actually extract a frame from the .r3d file? Extracting a frame from the .r3d is a considerable task and requires several steps including; decompression, demosaic, colorimetry, output encoding. This feature alone could easily make up the vast majority of this project and frankly, is so trivial in importance (in my opinion) that I would certainly recommend that you consider its omission as to reduce the scope of the task.

The tool labeled "TOOLS>R3D MOVER>TRUNCATE COPY" is to me, the most useful of the bunch. The inclusion of a variable for handles (in frames) is a great idea.

I am wondering if you intend to produce this suite of applets as a commercial product to resell or as simply because it is something that you (and others) may need.

I assume that for at least now, RED is not interested in supply an SDK (nor supporting those that are interested in developing applets) so this will be a traditional 'hunt and hack' approach. Are you looking for development partners on this project?

Again, a well considered set of tools and I am very interested.

Best,

Kevin
 
khmuse said:
In reading the list of features, one of the first that I read that caused me to pause was the "thumbnail". Do you actually intend to actually extract a frame from the .r3d file? Extracting a frame from the .r3d is a considerable task and requires several steps including; decompression, demosaic, colorimetry, output encoding. This feature alone could easily make up the vast majority of this project and frankly, is so trivial in importance (in my opinion) that I would certainly recommend that you consider its omission as to reduce the scope of the task.
Click to expand...
I actually think I have a cool simple (but very sneeky) way to do this. Could be wrong. Will know shortly. Yes, this feature can wait - but I am gonna set this up to SPEAK DIRECTLY FINAL CUT PRO SERVER.


khmuse said:
I am wondering if you intend to produce this suite of applets as a commercial product to resell or as simply because it is something that you (and others) may need.

I assume that for at least now, RED is not interested in supply an SDK (nor supporting those that are interested in developing applets) so this will be a traditional 'hunt and hack' approach. Are you looking for development partners on this project?
Click to expand...

My GUT is FREE for everybody if I can find a way to offset my hard cost on this. Which I think is about $10-$15K - my time for free.

I also want to make sure I have Red's blessing, otherwise it would stay in house.

I will for sure need a bit of help with some of the basic stuff to crank this out - I'd focus my secret code rockstar(s) on the hard stuff -
 
ibloom said:
Listening.

Thinking.

Remember the adage, a good programmer writes a great program from the ground up. A great programmer modifies an existing program.

IBloom
Click to expand...

And an a good gunslinger just pulls his gun out faster and shoots with a steady hand -

PS - your phone is gonna ring this weekend ... like it or not - you're on the team - if you're UP for it ....?
 
khmuse said:
In reading the list of features, one of the first that I read that caused me to pause was the "thumbnail". Do you actually intend to actually extract a frame from the .r3d file? Extracting a frame from the .r3d is a considerable task and requires several steps including; decompression, demosaic, colorimetry, output encoding. This feature alone could easily make up the vast majority of this project and frankly, is so trivial in importance (in my opinion) that I would certainly recommend that you consider its omission as to reduce the scope of the task.
Click to expand...

You might be right about your suggestion of ommision, but I think also that the correct path to the data Mark needs for thumbnails is through the quicktime codec.

Mark does this thing need to run on Windows or Mac?

IBloom
 
I would love to see those features in the ap.
Or in oder words, we need those features.

I think RED is probably already on this stuff but it will take some more weeks.

Good ideas Mark
Tim
 
If you're looking for feedback, at the top of my wish list is a command-line utility to
1) read in R3D file, export new R3D file with a specific single frame, or range of frames, and
2) read in R3D file and generate a text file in .CSV comma-separated-values format (or tabs or anything simple) with whatever metadata is available.

...and hopefully it would run on a PC, eg. not Mac-specific. If this needs FCP to work, I guess I'm looking for something else.
 
I would always say to myself, I would rather try and maybe fail then regret never trying at all. Good luck Mark...sounds complex but innovative.

Cheers
 
Sounds great. What database are you going to run it off of? MySQL, Excel, CSV?

Here is my 18 month old feature list request This is still what I think the RED pipeline should move towards for offline/online EDL support:
Red Workflow:
Images are captured on set in 4k RAW and recorded to a portable hard drive.
The data is then ingested to an onset laptop connected to external hard drives for archival and editorial.

Once back 'home' the footage is generically processed out to an SD proxy using REDCine. Bulk naming is a possibility using simple scripting such as: TestProject_%Camera_%Tape_%Date_%ClipNum_SD.avi

Obviously some of the variables would be from the RedCode metadata and others would be defined within the REDCine project settings.

Within each of these processed RGB files there would also be some absolute clip and/or frame id meta data.

Within REDCine the software would also be logging a Reference Table for the footage:
REDCode4k_ShotA == Test_Project_CameraA….SD.avi
etc.

I would then hand off the SD proxies to an editor, and they can do whatever it is they normally do in their post workflow to edit. They will do no grading on this footage.

The editor will deliver an EDL to the post house (or using later mentioned features submit an EDL for processing to the post house.)

REDCine loads the EDL. Sorts through its Reference Table and starts exporting clips in one of several configureable manners:

Let's say for example that you have shot two takes that you have now edited.

You have a 401 frame long sequence named "DV_A_FOOTAGE"
and a 200 frame long sequence named "DV_B_FOOTAGE"

You edit this in say AVID and export an EDL which consists of

AVID EDL:
CLIP_0001: DV_A_FOOTAGE from frames 100 - 150
CLIP_0002: DV_B_FOOTAGE from frames 0 - 10
CLIP_0003: DV_A_FOOTAGE from frames 250-400

REDCINE looks at this EDL, sees that DV_A_FOOTAGE is only needed between the frame range of 100-150 & frames 250-400. And DV_B_FOOTAGE is only needed from frames 0 -10 The user has configured REDCine to use of a 10 frame padding in addition to the necessary frames the EDL requires for existing transitions in case they want to adjust transitions in the online, and a 5 frame slate.

So REDCINE now exports a new 4k online cut of two files consisting of:

4K_A_FOOTAGE: [5 frame slate][DV_A_FOOTAGE: 90-160][5 frame slate][DV_A_FOOTAGE: 240-400]
4K_B_FOOTAGE: [5 frame slate][DV_B_FOOTAGE: 0-20]

REDCINE now performs a simple math operation on all of the EDL frames (assuming frames start at 0):

REDCINE EDL:
CLIP_001: 4K_A_FOOTAGE from frames 15-65
CLIP_002: 4K_B_FOOTAGE from frames 5-15
CLIP_001: 4K_A_FOOTAGE from frames 90-240

The user exports this new EDL and loads it into Fusion, Combustion, AE, Smoke, Avid.

A second option would be instead of grouping 4K_A_FOOTAGE, split it up into two files:
4K_A_1_FOOTAGE: [5 frame slate][DV_A_FOOTAGE: 90-160]
4K_A_2_FOOTAGE: [5 frame slate][DV_A_FOOTAGE: 240-400]
4K_B_FOOTAGE: [5 frame slate][DV_B_FOOTAGE: 0-20]

EDL then is updated accordingly.

Settings to the user would include:
[]Reduce Footage (Instead of exporting the full clip frame for frame from the original)
[]Pad cuts by ________ frames (Contextual if reducing)
[]Create new Clip for Each Cut (Each cut based on the EDL)
[]Generate Slate [design]

Slate would be a ridiculously simple text creation tool maybe even following HTML basic formatting and XML parsing:
<ORIGINAL FRAME RANGE><BR>
<LENGTH IN FRAMES><BR>
etc...

This would greatly facilitate working with VFX and Color Grading where you don't necessarily want to send someone more frames than they really need. You don't want to grade frames you won't see and quite honestly people just might not have enough hard drive space for *another* copy of a significant portion of their film in 4k.

This way, if you only use 2 frames of a 1000 frame clip, REDCINE only exports the two frames (or maybe a few more if you have a defined Front and End clip padding). This would also greatly accelerate the REDCINE Process since it won't have to Encode, White Balance and LUT 1000 frames.
These clips are then graded in whatever grading application the user wants such as Combustion, Fusion, Lustre, etc… and rerendered into a new folder.

The EDL is loaded into either an online 4k editor such as Smoke or else loaded into a compositing application for render such as Combustion (w/ Automatic Duck) or else Fusion which supports EDLs natively. And rendered out as a master 4k.
Click to expand...
 
jbeale said:
...and hopefully it would run on a PC, eg. not Mac-specific. If this needs FCP to work, I guess I'm looking for something else.
Click to expand...

Assuming the applet is straight C, it should be able to be compiled on any platform. I would recommend that it only opens the source .r3d as an input file and only outputs to a new file name (ie non destructive). A frame in and frame out point are all thats really needed. If its command line driven, then writing a script around it to automate the process is pretty simple. Assuming that the frame boundaries are locatable and that any necessary header information (frame count, offsets, etc) can be written out, I don't see this being a problem at all.

Let me know if anyone needs another hex hacker on this.

Kevin
 
khmuse made me think of one other thing to request.

Ideally I would like for this tool to be a bucket full of command line modules. That way if someone needs just the EDL parsing subroutine they could integrate it into say a Nuke TCL script. It would also allow modules to be swapped out and used interchangeably with RED's features as they develop and... IMPLEMENT A COMMAND LINE REDCINE HINT HINT... (nice and subtle huh ;)
 
i can understand from this thread that all what we need is like most
Pro editing workstations (softwares and hardwares) already have
correct me if i am wrong ?:umm:
 
You must log in or register to reply here.
Back
Top