Unique Frame Identifier

[I Bloom]

Don't confuse legacy tape formats and metadata with can be achieved in a file-based world. Timecode can only be unique when an additional piece of metadata is part of it such as shoot date. Add to that multiple cameras and that is even more metadata. Even unique frame ID's still need the "context" metadata for it to best fit into a workflow.

I cannot give more details as to the format since I am under NDA. But it does promise to deal with much of this, but still needs to part of a greater context.

Source metadata is one, then you get into compositional metadata which adds far more complexity right away in coming to an agreement as to how these should be presented. That's why after all these years, an EDL for better or for worse is still the fallback position when all else fails.

Michael

I think the basic issue with these formats is that software makers get lost in a world of compositing complexity. That will, I believe, never be standardized, but what is standard in my mind is simply that each step of the way you are pulling frames from different sources. I see those frames as nodes in a tree. That tree is in essence your workflow, and different programs can and will traverse it differently, but the structure itself is simple.

What happens as we traverse the tree is not so important, one program, and NLE might suggest a composite by putting several frames together. How they are composited is up to another program actually, one better suited to compositing as in the case of an NLE to Flame workflow.

I think in most cases passing the metadata and context data about how things are composited is optional. It's nice if you can create that interoperability, but it's not necessary you can leave it up to Flame to understand the particular NLE or not. The "backbone" is simply that various programs need to conform together into a single finished project, that all syncs up nicely to an editors intent. And that we can allow that intent to change without heaping on mounds of human effort to make it work. EDL's fail to do that for many reasons.

IBloom

 

[Jason Diamond]

The "backbone" is simply that various programs need to conform together into a single finished project, that all syncs up nicely to an editors intent. And that we can allow that intent to change without heaping on mounds of human effort to make it work.

This reminds me of a post stu maschewitz had on his blog back in sept 2G7 about standardized color metadata format

its a great article and echo's some of these ideas.

 

[Mark L. Pederson]

Not withstanding the merits of an industry wide standard, each frame CAN be uniquely identified already IF you take the following -

Camera PIN - unique to each RED-ONE camera

Clip Name - i.e Camera letter plus Reel/Roll Number plus Clip Number and Date

(yes I know that there is an argument that there COULD be identical clip names, but lets assume that issue is resolvable / resolved)

Timecode(s) - T.O.D at least usually does not repeat, if it does then combining with the run record timecode value resolves that

So a unique I.D for a video frame recorded at 13.01.15.22, 15 seconds and 12 frames as clip 03, on reel 12, camera A, April 28th is -

ABC_XYZ_123_A012_C003_042808_13_01_15_22_01_00_15_12

plus a per-frame checksum :wink:

 

[Philip Powell]

interesting idea

interesting idea

That would be nifty, and the I can see the benefits of it. But here's my question, wouldn't this only really work if any changes to the footage frames be non-destructive?

As any rendered files would create new frames, like for an effects shot, and lose their identity code, right? Or am I missing something ?

 

[Mark L. Pederson]

That would be nifty, and the I can see the benefits of it. But here's my question, wouldn't this only really work if any changes to the footage frames be non-destructive?

As any rendered files would create new frames, like for an effects shot, and lose their identity code, right? Or am I missing something ?

That is the whole point. YES, you want apps to pass that meta-data to renders and exports.

 

[Shawn Nelson]

Ian Bloom (we just call him Boba Fett)

Now don't say that. Boba Fett dies like a bitch, and when I met Ian he was one cool dude, unlikely to die like a bitch

On everything else, hell yes! Bring it on Ian!

 

[MichaelP]

AAF is like a tree with nodes... but can get complex very quickly depending on the type of metadata you need to extract at any point in time - not every process needs all metadata at all times. So how one steps through that process is equally as important. A Pull list is an example of probably the most simple extraction of metadata you can ask for. Sorting of data is the next step, adding handles is next, checking for dupes, etc. is next, etc.

It would be great if the RED files included both time of day and record run as an option instead of each clips starting at 00:00:00:00. Perhaps it does. It could open up some nice additinal workflows for layoffs. playback on set, etc.

Michael

 

[I Bloom]

when I met Ian he was one cool dude, unlikely to die like a bitch

Warms my heart, I think. Thanks for "going there" Shawn. I'm sure Mark was considering Boba Fett's entire story arc.

Ian

 

[Stuart English]

metadata

metadata

It would be great if the RED files included both time of day and record run as an option instead of each clips starting at 00:00:00:00. Perhaps it does. It could open up some nice additinal workflows for layoffs. playback on set, etc.

How so Michael, can you expand on that thought?

 

[Nick Shaw]

It would be great if the RED files included both time of day and record run as an option instead of each clips starting at 00:00:00:00. Perhaps it does. It could open up some nice additinal workflows for layoffs. playback on set, etc.

Michael

The proxy QTs do currently have two timecode tracks in them, which are ToD and Rec Run. Currently FCP only sees one of them, even though it allows the use of auxiliary timecode tracks internally. Hopefully that may change, and the secondary TC track from the QT will show up in FCP as an aux TC track. Stuart?

Can Meta Cheater do the same for Avid?

 

[Shawn Nelson]

Warms my heart, I think. Thanks for "going there" Shawn. I'm sure Mark was considering Boba Fett's entire story arc.

Ian

Lol, no problem. *I* would have compared you to Han Solo.

 

[RedFriend]

Lol, no problem. *I* would have compared you to Han Solo.

Clearly he's the Robert Deniro character in Brazil.

 

[Mark L. Pederson]

Suit up Scum!!

http://www.thedentedhelmet.com/

 

[I Bloom]

Suit up Scum!!

http://www.thedentedhelmet.com/

Pliny gave you that link, be honest.

IBloom

 

[I Bloom]

Not withstanding the merits of an industry wide standard, each frame CAN be uniquely identified already IF you take the following -

Camera PIN - unique to each RED-ONE camera

Clip Name - i.e Camera letter plus Reel/Roll Number plus Clip Number and Date

(yes I know that there is an argument that there COULD be identical clip names, but lets assume that issue is resolvable / resolved)

Timecode(s) - T.O.D at least usually does not repeat, if it does then combining with the run record timecode value resolves that

So a unique I.D for a video frame recorded at 13.01.15.22, 15 seconds and 12 frames as clip 03, on reel 12, camera A, April 28th is -

ABC_XYZ_123_A012_C003_042808_13_01_15_22_01_00_15_12

Just to get this conversation back on track:

Stuart, I think you are correct, that would be a unique string for every Red camera, possibly for every camera. But in a workflow where new frames are being created, such as modified copies of a frame from a Red camera (color corrected, scaled, composited etc) it is better to start from something totally unique and I'm speaking specifically of software standard uuids http://en.wikipedia.org/wiki/Uuid
I've never implemented a generator for these and I don't know if there is a library for them that could link into your firmware so forgive me if I don't know of the difficulty or cost in cycles. But I think this is truly important for integrating camera original images with an entire pipeline of image transformation.
And some kind of per frame checksum is of course important in addition.

IBloom

 

[Dylan Reeve]

If there where a UFID I see some benefit in making work in a sequential way, which would provide simpler methods of expressing a sequence of frames.

So there would be a unique clip ID, built however is suitable, that was then amended with a sequential serial for each frame.

That way, I can still track an individual frame as A34F45-D33E5A19-0A0FA22-000009

But I can also represent a series of those frames as A34F45-D33E5A19-0A0FA22-[000002-000234]

I can also then determine a relationship to unspecified frames that may exist before and after the frame I am looking at without needing a complex index table.

Further, by adding a revision code to the system different versions could be tracked. Perhaps An application or process code also.

A34F45-D33E5A19-0A0FA22-000009-003-AFX

Where the first part (A34F45-D33E5A19-0A0FA22) represents the larger object (clip/shot), the second part (000009) represents the frame's temporal position within that object, the third optional part (003) represents the frame's revision and the fourth optional part is a code to represent what the frame is or what made it perhaps.

That way from any given frame I still have a relationship to the original version, it's revision, the larger object from which it's derived, and some method of understanding what this revision is in relation to others.

 

[Mark L. Pederson]

Further, by adding a revision code to the system different versions could be tracked. Perhaps An application or process code also.

That way from any given frame I still have a relationship to the original version, it's revision, the larger object from which it's derived, and some method of understanding what this revision is in relation to others.

To me, the whole point IS that the ID is EXACTLY the same. No application or process code.

I would track versioning with other meta-data as a proper, comprehensive tracking requires a database - and you want people to be able to work on the media on any system without accessing a database.

 

[Dylan Reeve]

To me, the whole point IS that the ID is EXACTLY the same. No application or process code.

I would track versioning with other meta-data as a proper, comprehensive tracking requires a database - and you want people to be able to work on the media on any system without accessing a database.

The problem with that is you are generating ID collisions. If anything happens to your versioning database, or some shot or frame isn't included for some reason then you have a number of frames with identical IDs and no clear indication of what revision they are or what process they represent.

To be able to work the media on a system without accessing a database you have to be able to differentiate versions of the same frame, otherwise without the addition of a database or additional pieces of independent metadata you have absolutely no way of know which of they six frames with exactly the same unique frame ID is actually the 'current' one.

By having a root ID of a known format (or at least a known length) then you can store revision and other basic information with the larger ID and still have an absolute relationship with the original frame.

 

[I Bloom]

The problem with that is you are generating ID collisions. If anything happens to your versioning database, or some shot or frame isn't included for some reason then you have a number of frames with identical IDs and no clear indication of what revision they are or what process they represent.

To be able to work the media on a system without accessing a database you have to be able to differentiate versions of the same frame, otherwise without the addition of a database or additional pieces of independent metadata you have absolutely no way of know which of they six frames with exactly the same unique frame ID is actually the 'current' one.

By having a root ID of a known format (or at least a known length) then you can store revision and other basic information with the larger ID and still have an absolute relationship with the original frame.

A unique clip identifier is a good idea. But giving every frame the same UUID + Frame number is a big mistake and missing the point. UUID's are an essentially limitless addressing space. If you want to know what clip a frame is a member of, include the parent clip UUID in the metadata of each frame. Create a new UUID for any derived frame and point to the parent frame by it's UUID, in fact hold the entire history of derived frames in a tree of UUID's. This all amounts to just kilobytes per frame.

The frame number is metadata, along with edgecode and timecode. They are great things to have. So is the Reel number camera name, date and time. Mixing these things in with the identifier is overcomplicating things. The reason UUIDs are so important in programming is because they prevent collisions across an entire universe of objects. A UFID won't just never be confused with another frame from any Red camera, but it will never be confused with every clip reference ever generated by FCP, every one generated by AVID and every object ever generated in Second Life for that matter.

We need an backbone addressing space that we can rely on under all conditions. My hope is that it will be entirely seperate from other metadata.

IBloom

 

[Dj Joofa]

I would track versioning with other meta-data as a proper, comprehensive tracking requires a database - and you want people to be able to work on the media on any system without accessing a database.

It is a fact well-known among pattern recognition and database communities that any searches in a data space of dimensionality greater than about 6-10, quickly degenerates to a brute-force search (search over all elements in a database), regardless of what structure (say tree based structure, B-tree, R-tree, M-tree, etc.) you put in the database, and retrieval performance suffers severely.

Each element of metadata increases the dimensionality of the space. For most practical purpose the intrinsic dimensionality may be much smaller. This is a very tricky aspect in database design and may not be taken lightly.

Please Google "cruse of dimensionality".