Database handling in Resolve for large projects.... Speedup tips...

[Gunleik Groven]

Hi

Just started playing around with conform as prep for two series going online soon.

One is 6 x 1 hour delivered
One is 24 x 30 minutes

we're also starting to shoot a third on 2 Epics which is 7 x 45 minutes methinks.

So, why this introduction?
Because all these projects will have quite large filedirectories and cosist of mainly r3d's + exr's from VFX.
The total database for these projets consists of many TB data (around 60-120 projected for the last case)

So...
The way to go online from xml in resolve, seems to load the database first, then import the xml and that will conform from what is loaded in that directory.

All fun and orange juice.
Until it seems that loading a database consisting of 60% of the shots for the project, takes 25 minutes, and any calls to alter the database clogs up saving.

And today I just conformed a 5 minutes trailer...

And saves also takes minutes.

So...
If had a smaller database, I guess it would be more responsive.
Is there a way to ditch all files you don't use from a project database?

Also, opening and copying a project from one user to another took ages....

Any advice is well taken.

The good thing was that resolve automatically read the .rmd's so that I could do a pregrade in RCX that then would follow the conform into resolve.
Sweet, as the resolve r3d windows aren't as accessible and flexible.

It seems that now resolve reads the curve parameters from the rmd's, even though you cannot access those parameters from within resolve.


Cheers!

Gunleik

 

[Jake Bastian]

That's a tough one. Personally, I don't preload my media pool with anything because I like to keep it as clean as possible. The XML will automatically bring the clips you're using into the media pool for you which does take a while (too long IMO) for each XML but it keeps your project clean in the end. I haven't found any way to locate clips you AREN'T using in the media pool for quick elimination though so for right now I find it's best to 'front load' my import time via individual XMLs instead of spending a lot of time at the end of a project cleaning up.

Hopefully this is an option they add to 8.1, maybe even color coding the clips in my database so I know which are being used and which aren't...

 

[Gunleik Groven]

When I tried ro conform on import, it didn't relink to the r3d's. Just complained the the .mov files were not found that directory...

I set the switch to look for reel and clip in the setup dialogue.

The only way I made it conform, was by preloading the clips into the pool, as to the manuals directions.

Problem is that basic operations then are allmost unuseable.
Like save, autosave, adding files to the pool and reconforming to them.

Hm...

I cannot be the only one using Resolve on these kind of projects?

It really started me on why they have chosen their own database structure, when it works so slow...


Hm...

Cheers
This will get solved, I guess

Gunleik

 

[Rosario Balistreri]

Usually I work with FCP and the Log and Transfer is really helpful in these cases:

You edit with Proxies or what you want, then at closed editing you capture all the used footage with log and transfer without handles in REDRAW and only of used clips making a "consolidate".
Now you have 500GB of data instead of 60-120TB.
You can now send the XML to Resolve and start grading.
A lot less clips (only used files) with only the used frames!
I don't know if the DB will be smaller (in number of rows), but it manages shorter clips.

For the currents project I've not any tips...
I usually grade shorts with no more then 100 cuts.
For a webserie I create a unique database but with a "Project" for every episode. I think that is easier to manage and faster on saving.
When I need a specific grade for an old episode I load it, save the grade/still, reload the newer episode and continue grading.
That's fast because the duration is about 5-7 minutes and opening another project is like 5-10 seconds on an iMac.

I hope that with 7 projects (inside the same db) with a timeline of 45 minutes will be better then 1 project made by 7 timelines of 45 minutes each one

When I tried ro conform on import, it didn't relink to the r3d's. Just complained the the .mov files were not found that directory...

What kind of .MOV?

Proxy _M files for example?

I'm not sure but you could try to batch-replace with textedit all the file names in the XML to be matched with the R3Ds... And then import it in Resolve

 

[Gunleik Groven]

We offline in prores. Online from raw (maybe, I have to add right now) and exr's...

Cam proxies is not an option as we are well into epic land.

L&T is not a real option for these kind of projects...

The hacking of the xml will maybe work.
I'll look at that. If we break the episodes down to 15 minutes and hack the xml... Brave new world...

Thanks
Keep 'em comming...

 

[Jake Bastian]

Gunliek, I haven't tried it out yet, but I understand Monkey Extract will read your XML and then consolidate only the R3Ds you're using to another destination. If you're feeling really brave and have REDline you can take it one step farther with EDL2RD http://foolcolor.net/foolcut which will not only read the R3Ds you're using but trim them down to only what you're using and then copy that over to the new destination saving even more space.

Again, I haven't tried these in my workflow yet as I haven't had time to sit down and experiment, but it's the direction I'm headed with my workflow - plus this way you could add all your clips to the database beforehand as you've been doing and keep that saved time.

 

[Dustin Cross]

I too was wondering about taking your XML and copying just what footage you need to a new directory. Or even better converting to trimmed R3D files of just the portion used in the XML with a few frame for heads and tails.

Dusty

 

[Filip Vandoorne]

check perhaps this :
resolve preferences:
Basic
the first volume in de media Storage Volumes
will be use for a invisible folder where dpx files
for each thumbnail or gallerystill of resolve will be saved.
choose a fast-big disk, dpx files are big.
we also had once a problem with seemingly corrupt dpx
files in this folder which crash-locked the resolve.
it is also used for caches, which probably are also dpx,
so again choose fast and big…
-- in the config tab/ settings - Resolve Working folders
you could move these folders -- you see that the gallery stills
are in a invisible folder.

the second volumes in your Basic prefs
are then the volumes of your red raws,
but for the first its best a seperate and fast volume.
this was with version 7 of resolve, but i guess that the 8
works in a similar way.

 

[Ido Karilla]

Like Dustin suggested....

To be able to work light I usually go through Redcine x and trim the R3D's with handles and use the trimmed media to conform in Scratch.

In Scratch it works well but you have to make sure you pick the correct instance of the shot.

Hope it helps.

 

[Peter Chamberlain]

Hi, Resolve uses a Database for all projects as many facilities have multiple resolves and they all work on a shared database server with projects being used in multiple suites, switching daily sometimes. It also allows for multiple colorists at the same facility to share projects and copy stills between them. I doubt your database in the TB range as I guess you are thinking the source images are stored in the DB, they are not. You can always export a single project if you like with or without stills etc but its not so efficient. For your new series, given the size, start a new database and use multiple projects, one for each episode. Having 20+ projects in a DB is nothing so this is not a requirement but will help you in backup of the DB and archiving.

The next issue you have is loading clips into the Media Pool and conforming the r3d's. Note the Media Pool is not the database but the database will store the location of the clips once you save the project. Be sure to save when you setup you jobs configuration and turn on autosave.

You can load r3d's manually, via EDL, AAF and XML. I suggest only loading the clips you will use for the project. If you are using an XML, select the storage subfolder where the source clips are for THAT program. If you select the whole storage volume then every clip must be opened to see if it has the correct start and stop TC as well as referencing the file name and reel numbers. This will take some time. If you also load all the clips you have, Resolve will also have to deal with 100's of clips that it does not use, each using resources. BTW, the media pool usage column shows which clips are being used in the conform. Right click and select 'Update Info' to update this column.

If your XML is not loading the r3d's you may have separated the r3d and mov files from their original shared RED folders and or don't have the correct reel name descriptors. I would practice with a three clip XML till you get the parameters correct.
Peter

 

[jake blackstone]

Hi Peter.
Thank you for thew info. I also wonder what strategy would you employ, if your project is a long form and it uses multiple drives with multiple directories. Of coarse, the consolidation of files would be the preferred method, but sometimes that is not practical, as well as it would require a third party solution. (Any desire to include such function in Resolve?)
So, right now you'd select an XML or EDL and Resolve would ask for the directory, where you have those files residing. Once Resolve finds those files, it indicates, that there are missing files and that usually would be the end of it. Would it possible to do an additional conform with just the missing files, that are on another drive or another directory? Or would it be possible to see which files are missing and do a manual directory change just for those missing files?

 

[Gunleik Groven]

Hi, Resolve uses a Database for all projects as many facilities have multiple resolves and they all work on a shared database server with projects being used in multiple suites, switching daily sometimes. It also allows for multiple colorists at the same facility to share projects and copy stills between them. I doubt your database in the TB range as I guess you are thinking the source images are stored in the DB, they are not. You can always export a single project if you like with or without stills etc but its not so efficient. For your new series, given the size, start a new database and use multiple projects, one for each episode. Having 20+ projects in a DB is nothing so this is not a requirement but will help you in backup of the DB and archiving.

The next issue you have is loading clips into the Media Pool and conforming the r3d's. Note the Media Pool is not the database but the database will store the location of the clips once you save the project. Be sure to save when you setup you jobs configuration and turn on autosave.

You can load r3d's manually, via EDL, AAF and XML. I suggest only loading the clips you will use for the project. If you are using an XML, select the storage subfolder where the source clips are for THAT program. If you select the whole storage volume then every clip must be opened to see if it has the correct start and stop TC as well as referencing the file name and reel numbers. This will take some time. If you also load all the clips you have, Resolve will also have to deal with 100's of clips that it does not use, each using resources. BTW, the media pool usage column shows which clips are being used in the conform. Right click and select 'Update Info' to update this column.

If your XML is not loading the r3d's you may have separated the r3d and mov files from their original shared RED folders and or don't have the correct reel name descriptors. I would practice with a three clip XML till you get the parameters correct.
Peter

Thanks!

To start off:
We are such facility, currently running 5 "full" resolve suites + 2-3 smaller setups towards the same sans.
Many colorists share them and there are usually 2 shifts a day on each suite.

So far - after switching - we've basically ran smaller projects where everything is online through it. Now we are moving the drama-series to resolve as well.

Trimming and making copies of the r3d's is not an option.
Then we can as well do what we did, conform to uncompressed log, but we don't want that. Especially not with the Epic material currently running into our SANs...
On the R1 material, I see little practical difference working from 10-bit pregraded uncompressed log files and grading from RAW.
Epic... that's another story.

The issue is probably a user issue, but here's the structure of the files on the SAN:

Offline proxies/date/all offline files
r3D's/date/each magazine/files

We used to keep the folder for the magazines back in 2008/2009, but since then - all tools I have used until now... have been able to relink from a topdirectory, pointing to the one containing one type of files to another...

The conforms will need to look at files from the whole recording period, as the series are rarely shot chronological.

The size of these projects are typically 4 - 12 hours delivered.

And they all consist of a mix of R3D's and EXRs from VFX.
Thus I'll have to reconform from R3D's to EXRs for relevant shots.

I know a few color-tools that do this elegantly, but that is not the point

What seems to be the problem (and probaly is a user error...) is that when I load XML (from a prores proxy-edit) and try to load only the relevant files from the R3D directory, does not import the XML "empty", but just kills it and complains that the file "filename".mov is not in that directory. Which is true, and also not the file I am looking for...

I have checked the "reel and clip" option in the setup window.

I can see that if I alter the xml, and batch change *.mov to *.r3d or just try without the suffix, it might load correctly.

What is a bit irritating, is that it actually doesn't load at all. And using the suffix at all for the conform, will make issues when I re-conform certain clips to EXRs.

But these are all very newb questions, and probably related to user error...

I am totally open to that!

Cheers

Gunleik

 

[Rohit Gupta]

Hi Peter.
Thank you for thew info. I also wonder what strategy would you employ, if your project is a long form and it uses multiple drives with multiple directories. Of coarse, the consolidation of files would be the preferred method, but sometimes that is not practical, as well as it would require a third party solution. (Any desire to include such function in Resolve?)
So, right now you'd select an XML or EDL and Resolve would ask for the directory, where you have those files residing. Once Resolve finds those files, it indicates, that there are missing files and that usually would be the end of it. Would it possible to do an additional conform with just the missing files, that are on another drive or another directory? Or would it be possible to see which files are missing and do a manual directory change just for those missing files?

Hi Jake,

In the timeline, if you right-click, there is a function to "Automatically Import Media". This is what you are looking for I think. Let us know if that works for you.

Regards,
Rohit

 

[Jake Bastian]

To understand you - you're performing the edit using offline files (LTs) and then using something like clipfinder to link back to the proxies or are you editing from the proxies? You're working out of FCP 7 correct?

 

[Gunleik Groven]

To understand you - you're performing the edit using offline files (LTs) and then using something like clipfinder to link back to the proxies or are you editing from the proxies? You're working out of FCP 7 correct?

Is that for me?

We batch process and offline colorcorrect the files in RCX on-set
Transcode to prores
(L&T is not for these workflows...)

Edit

Conform for VFX

Export EDL or XML
XML would be preferable

import XML into resolve
From here, the issues rise.
What I would like to do is to point at "any" directory with a set of files with the same reel and clipname and conform to those
Without preloading "all" to pool, as the manual suggests, and which works fine
the issue is that the project takes ages to save and perform handling tasks with, if the pool is "too big"
I would have no issues with that approach, if simple stuff like saving and changing files in the pool was not clogging the work...

 

[Florian Stadler]

I have found that UNCHECKING that option works when working with the XML whereas it needed to be checked in earlier versions when working with EDL.

All my imports with XML have worked flawlessly since I turned the assist with reel option off...

 

[jake blackstone]

Hi Jake,

In the timeline, if you right-click, there is a function to "Automatically Import Media". This is what you are looking for I think. Let us know if that works for you.

Regards,
Rohit

Hi Rohit.
Thanks for the tip. I'll give it a shot tomorrow.
I normally work with commercials and other short forms projects, so usually I have no need for this kind of long form conform, but I was always curious about it. Could you clarify what exactly "Automatically Import Media" function does? Does it automatically scans all attached drives and it's directories and imports missing files? I suspect, that it is not the case...

 

[Jake Bastian]

Is that for me?

We batch process and offline colorcorrect the files in RCX on-set
Transcode to prores
(L&T is not for these workflows...)

Edit

Conform for VFX

Export EDL or XML
XML would be preferable

import XML into resolve
From here, the issues rise.
What I would like to do is to point at "any" directory with a set of files with the same reel and clipname and conform to those
Without preloading "all" to pool, as the manual suggests, and which works fine
the issue is that the project takes ages to save and perform handling tasks with, if the pool is "too big"
I would have no issues with that approach, if simple stuff like saving and changing files in the pool was not clogging the work...

Yeah it was for you, sorry How much control do you have over camera crew, specifically, what reel number drives are coming in with?

I'm just thinking that the reason your directory structure goes: r3ds>date>magazine#>files is because you're having drives come in with repeated reel numbers. I'm not sure how Davinci handles its scanning of directories for file locations but maybe simplifying that structure as much as possible would improve the search speed during the conform.

As far as saving taking a long time - it seems like the size of your media pool can't be helped due to your non-linear shooting schedule so there isn't a change I can suggest there. Are having to work in a situation where multiple people are working on the same project file at once so Resolve is having to reconcile differences between workstations? Because if what Peter is saying is true, rewriting what is basically a large excel file should happen pretty fast unless it's trying to change files on multiple computers. This leads to my next question:

If your workflow using RMDs is what's slowing the program down? I usually discard RMD info going into Resolve because my projects allow for it so I haven't seen firsthand what it does to Resolve's performance. Have you tried not using RMDs or are you in a situation where you have to use the Redcine X firstlight to save time?

I may be assuming you're using RMDs incorrectly though... It'd be nice if someone from Davinci would chime in on this as even though I'm not working on projects of the size of what you're working on now I have noticed that it can take a very long time for Resolve to "find" my files as well when it really shouldn't. Saving time has never really been an issue for me though.

 

[Jake Bastian]

Hi Rohit.
Thanks for the tip. I'll give it a shot tomorrow.
I normally work with commercials and other short forms projects, so usually I have no need for this kind of long form conform, but I was always curious about it. Could you clarify what exactly "Automatically Import Media" function does? Does it automatically scans all attached drives and it's directories and imports missing files? I suspect, that it is not the case...

It only searches where you tell it to. Kind of like in the initial XML import.

 

[Gunleik Groven]

@ Jake Bastian

Nope.
No double reelnames as long as I have been in charge of workflows...
But the shooting is indeed very nonlinear, and this organisation has worked very well... up to now.

Yes.
I need rmd's to get the files where I want them before grade.

No real grading work, basically redlogfilm/redcolor and small tweeks to wb/tint/curves to make the scenes go from the same startingpoint.
But the meta-tools in resolve aren't really set up for anything but batch, which is not what we want.

Thus: Conform to RCX
Firstlight to rmd
Load in resolve

The alternative would've been our old route going through uncompressed log 10-bit.
But we are gearing towards raw online...

I am not indifferent to changing the projectstructures, if that helps - on new projects.

Doing that on current ones, would be a mess...