Backup and Tape device answers

[warwickt]

Do you guys always choose to verify your tapes? If so, has the verification ever picked up an error?

HI Roy, well for me I use BRU-PE. BRU-PE does its own verification once the essence/objects have been archived. It does this by default. I recall there are two methods you can use. THe volume(s) is/are rewound to the beginning of the set and then verified using the BRU-PE method with its "checksum" I assume or you can specify a DISK object to tape object data comparison (I don't think you need to do the latter).

I have only found one dud object in my time and BRU-PEs picked that up on a verify of a few hours of some DVCPRO-HF P2 stuff... in fact I think it was just some glitch. I reran the archive and it was find. I recalled the offending object just to make sure..

hth

w

 

[Hans von Sonntag]

Do you guys always choose to verify your tapes? If so, has the verification ever picked up an error?

Yes and yes.

The errors were minor and irrelevant to the backup because the files were corrupt on the source disc already.

But I always verify. BruePE does the verifying without the source disc. I find that convenient and makes me sleep better.

Hans

 

[Roy Pearson]

Yes and yes.

The errors were minor and irrelevant to the backup because the files were corrupt on the source disc already.

But I always verify. BruePE does the verifying without the source disc. I find that convenient and makes me sleep better.

Hans

Good to know. Even if it's just to find corruption on the source disk, that is still something I want to know about.

 

[Jörgen Persson]

Hi, Tim. I just got the equipment last night and BRU PE was a breeze to install and use. But I have a question for you regarding backup of Red dailies.

Is there a way to restore a full backup plus all incremental backups as one restore operation in BRU PE?

I have a workflow where I offload Red files to several 750 GB partitions.

I start with a full backup for the first batch of dailies (50-100 GB) and then incremental backups (also 50-100 GB) every day until the partition is full.
This means each full partition is archived to a single LTO-4 tape.

The problem is that BRU creates a new archive "item" for each incremental backup.
If I ever need to restore the entire partition then I have to:

• Restore the Full backup
• Wait until it finishes
• Restore the first Incremental backup
• Wait until it finishes
• Restore the second Incremental backup
• Wait until it finishes
• And so on...

If there is a feature where everything is restored at once, then please let me know. (Something similar to the "Latest" folder in OSX Time Machine...)

Otherwise consider this a feature request :smile:

 

[Jeff Kilgroe]

The pending 2.1.0 update will have the import function reenabled as well as a few other updates. I'll let everyone know as soon as we have new news.

Hey, Tim, any news on the update? I'm actually in a situation where I really could use the ability to read or rebuild catalog info from a tape. I have a tape I'd like to restore and it's one I made with an older beta version of PE. I know what is on the tape, but my BRU PE doesn't have a catalog for it and at this point is not capable of going to look for it on its own.

I haven't fooled around with tape tools much, is there a way to regenerate the catalog info with what I already have? This isn't mission critical, but the tape has an obscure little thing I did for a client that I thought would never be needed again (they actually told me to delete it). If I could restore it sometime this coming week that would save me a couple days of work. :sifone:

EDIT> Solved. Command line is my friend... I guess I should get some sleep while I restore this tape.

 

[a kines]

Tim,
I was getting quotes for transcoding and storage and one place that had an LTO3 system said they weren't upgrading to LTO4 because, and I am paraphrasing here, "it compresses the data and you don't want that".
I've read a lot of the information available online about the technology and while I haven't used it directly, I was under the impression that the data compression on LTO4 does not alter any crucial data, at least not in any way that would affect R3D files.
Could you clear that up for me?

 

[Jeff Kilgroe]

LTO-3 and LTO-4 both offer compression and do it the exact same way. It is something that can be turned on or off. No big deal, so I don't know what the guys were talking about. I do know quite a few people who have been using LTO-3 and they are skipping LTO-4 simply because LTO-5 is due out later this year.

Back on the compression, even if left on, it doesn't have any adverse effects. It simply can't compress data that is already heavily compressed, but there is no data loss or anything to worry about. Think of it as compression similar to ZIP, LZW, etc.. If you try to comrpess an R3D or other file that is already compressed, you don't get any smaller file. LTO-4 media will give 800GB of native (uncompressed) capacity per tape. LTO-3 is 400GB.

 

[Tim Jones]

Compression and tape - fact and fallacy

Compression and tape - fact and fallacy

As Jeff has mentioned, tape drives have been compressing data in a lossless format since the original DDS-1 1.2GB DAT days (circa 1991). The compression used is a fully adaptive lossless method that provides 100% recoverability of the data processed. Lossless means NO altering of the data content written on the tape.

However, regardless of how lossless the algorithm is, it's the "adaptive" part that matters to what we do. All modern tape drives offer a large input buffer - as much as 128MB. This allows the drive's processor to examine the incoming data stream and decide if a section of data can even be compressed. With what we are all placing onto tape, the result is always no, so the compression algorithm absolutely does nothing to our data. So why don't we just disable it? Because other parts of the data stream involved with the actual archive format - whether tar, cpio, mtf, or bru - CAN be compressed.

This is why we are all only seeing a total per-tape capacity of 800GB on LTO-4 or 400GB on LTO-3 (1.5TB on LTO-5).

If that transcoding company is not moving to newer tape technology, it's their loss. However, don't let the fallacy of their claim of compression harming your data prevent you from making the right move.

I've been in the compression game since those original DAT drives (I was part of the design and delivery team at Archive), and I've never witnessed a single instance where you didn't get back exactly what you put in (assuming the software wasn't wonky).

Tim

 

[Tim Jones]

Restores can be easier...

Restores can be easier...

Is there a way to restore a full backup plus all incremental backups as one restore operation in BRU PE?

[snip]

The problem is that BRU creates a new archive "item" for each incremental backup.
If I ever need to restore the entire partition then I have to:

• Restore the Full backup
• Wait until it finishes
• Restore the first Incremental backup
• Wait until it finishes
• Restore the second Incremental backup
• Wait until it finishes
• And so on...

If there is a feature where everything is restored at once, then please let me know. (Something similar to the "Latest" folder in OSX Time Machine...)

You can simply select all of the archives from the Full through to the last Incremental (normal Finder-style multi-selection), set the restore to Overwrite existing and start the restore. It will automatically start with the Full and then restore each incremental in turn (prompting for any needed media), overwriting each older version with the version that supersedes it in the incrementals. All you need to do is change tapes.

The thing that's important about the way that we do it is that 90% of all restores are "oops" restores. This means that you're only usually going to restore a few files or a folder from a specific archive to correct a "bad edit" or a mistaken deletion. That's why we specifically segment things this way. That way, if what you're looking for is on Thursday's tape, you don't need the tapes from last week to get your file back.

For the remaining 10%, the method above gives you exactly what you're looking for.

 

[Jörgen Persson]

You can simply select all of the archives from the Full through to the last Incremental (normal Finder-style multi-selection), set the restore to Overwrite existing and start the restore.

I tried that and it didn't work. In restore mode I can only select one archive at a time. It doesn't matter if I lasso, shift click or command click - just one of the rows are selected in the Archives pane and then a dialog appears with "Retrieving Archive Contents List" before displaying the file path in Archive Content.

 

[Tim Jones]

Now THAT is definitly a bug. You should be able to select the first archive, let it parse, and then SHIFT-Click or CMD-Click to select the others. I'll see if that can be looked at in time for 2.1.0.

Tim

 

[Jörgen Persson]

Now THAT is definitly a bug. You should be able to select the first archive, let it parse, and then SHIFT-Click or CMD-Click to select the others. I'll see if that can be looked at in time for 2.1.0.

Tim

Thanks. I'll try it on a different machine tonight and see if it's an OS version thing...

 

[Tim Jones]

Thanks. I'll try it on a different machine tonight and see if it's an OS version thing...

No Need - I've already duplicated and found why it's happening. Sitting down with the engineers in a few to see if we can add it to 2.1.0.

Tim

 

[Maciej Roszkowski]

Hi Tim

I have a question for you in regards to bru.
In the past I had to restore Retrospect archives from tape with out the definition files present on the system. This has been a simple click operation.

I'm now using Bru Le on a single tape system for my red footage. I can't seem to find a way in the restore window to add or rebuild archives directly from a tape. It's not a problem I have ran into yet, but I want to be prepared for the possibility that I will need to do this.

 

[Jeff Kilgroe]

I'm now using Bru Le on a single tape system for my red footage. I can't seem to find a way in the restore window to add or rebuild archives directly from a tape. It's not a problem I have ran into yet, but I want to be prepared for the possibility that I will need to do this.

I can't comment about BRU LE, I thought it could. But it seems this functionality is coming in the next update of BRU PE.

Just last week I went to restore some files from an old tape, for which I didn't have the definition file. It was pretty easy from the command line to just extract the entire contents of the tape.

"man bru" has all your answers. But if you create a new folder where you want to dump the contents, then from that folder location enter this:

"sudo bru -xvvvvvf ntape0 -b 1024k -PA -a"

Just be forewarned that you will need the block size, which is 1024KB in the example above, and that's the setting I always use when writing my tapes. If you don't know the block size, you can always scan the tape in the BRU GUI or from the command line and it will detect the archive and report block size to you. BRU default block size is 128KB.

 

[Rick Allen]

Works for me...

Just for kicks, here's the link again:

Hard Disk Warning by Larry Jordan

HTH,
Tim

So I have used a Hard drive based system for archiving photographic shoots that I digital tech on for a number of years and while I've recently added Lto-4 to my KIT I haven't had a chance to transfer all 190+TB of data over to tape and so I'm pretty anxious to keep this data alive. So After reading the Larry Jordan articles I created theRefresh. Feel free to check it out. www.rapdigital.net/theRefresh.html
you need to provide an admin password to run the shell script and it takes ages. I'll redo the app when I have time to include a progress bar.

 

[Maciej Roszkowski]

I can't comment about BRU LE, I thought it could. But it seems this functionality is coming in the next update of BRU PE.

Just last week I went to restore some files from an old tape, for which I didn't have the definition file. It was pretty easy from the command line to just extract the entire contents of the tape.

"man bru" has all your answers. But if you create a new folder where you want to dump the contents, then from that folder location enter this:

"sudo bru -xvvvvvf ntape0 -b 1024k -PA -a"

Just be forewarned that you will need the block size, which is 1024KB in the example above, and that's the setting I always use when writing my tapes. If you don't know the block size, you can always scan the tape in the BRU GUI or from the command line and it will detect the archive and report block size to you. BRU default block size is 128KB.

Thnaks Jeff. Helpful info. Ironicly when I was looking for the scan tape option I found a GUI way of doing this. In the verify tab there is an import archive option. So this should be in PE already as well.

Makes me feel a lot better about my backups

 

[Maciej Roszkowski]

PE Questions

PE Questions

So one snag I am running into with BRU LE is that it seems to like archives that are limited to 250GB. It's listed in the FAQ but not in the feature sheet.

Using LTO 3 tapes I was able to get about 350 GB out of retrospect per tape, but much less using BRU (it varies but ranges from 250 to 300). The FAQ lists the server version as an alternative for larger archives.

My question is whether the PE version will do what want. Seems like it should, but there is no FAQ for it yet. If it will solve my problem what kind of upgrade path can I get for the product. Price is listed as 'CALL'.

 

[timo]

Hi Tim,

Do you have an idea of when 2.1 will be released? I'm keen to use the tape doubling feature for a single tape drive, and am a little paralyzed as to whether to start archiving now or to wait a week or two...

cheers
Tim

 

[Jeff Kilgroe]

@ Maciej,

Yeah, LE is very limited with archive size and requires too much manual intervention and manually splitting up your archives to fill your tapes, especially on LTO.

With BRU PE and BRU Server, there are no such limitations. I use BRU PE and routinely create backups to LTO-4 that are 2 to 4 TB in size and span 3 to 5, sometimes even 6 tapes. No problems.