What is the most standard based workflow for LTO-4?

[Vlad N.]

Hi all,

It seems those days this forum has become the state of the art ressource for all LTO-4 related stuff, so I'd like to ask you for some help with the purchase decision.

I've left tape based solutions 10 years ago after a well known in the 90s "horror story": tape drive fails, the new one isn't supported by the current version of the backup software, new version asks for new MS Windows version, other software is compatible with the drive and the current MS Windows version, but can't read the tapes made with our current software...

For 10 years I was using disk based backups, but now we need to find out a solution to archive our video assets for long term shelf storage, and, of course, I think a lot about LTO-4 drives.

However, I work in the company created after a merger of some smaller structures, so we keep the remaining of different IT decisions made in the previous periods. We have Mac OS X, Linux and MS Windows machines in different locations, so I try to figure out the most compatible solution.

We plan to purchase three LTO-4 drives and to install them on various locations. Everyone will next archive all video an audio assets, keep one tape and send a copy to central office for off-site storage.

Of course, everyone also wants to use the drives for regular backup.

I can cope with hardware part, but I'm lost when I try to find a cross-platform standard software choice.

Basically, all Linux users say we will use mt and tar, because they are standard for LTO. The day we'll need to use the LTO-4 standard built-in encryption, we're sure Linux will support it. For regular backups we'll use Amanda or Bacula, both free and based on the top of the a.m. standard tools.

Windows users want a GUI and while tar command exists on Mac OS X, I can't figure out how to talk to LTO-4 drive using only built-in or FOSS tools.

As I've said, I don't want any vendor lock-in or proprietary archive format. So, I'm trying to figure out the best workflow based on standard file formats.

1) We can buy three Cache-A LTO-4A drives. 8000 * 3 = 24 000 bucks, out of question.

2) We can buy 3 external drives and BRU licences for all three platforms. It's 4000 (drive, cable and controller card, software license) * 3 = 12 000 bucks. Half the price, but still very expensive.

3) I can get internal SAS drives for only 2000 dollars each. Maybe I should build three "poor man's Cache-A systems" using some kind of Linux boxes? I install a 1 Tb HD in every box as a cache, next I send all files to this shared HD from Windows, Mac or other Linux computers and finally they are written to the tape? That way we can use the LTO-4 drive at its full speed.

If this is the best way to go, please give me some advice regarding the hardware. Are there some small factor cases with at least two free places for LTO-4 and BR drives? Are there some MBs with a SAS connector to avoid a purchase of a dedicated card?

Of course, being a Mac guy :drool: I would like to have a Mac GUI and easy to use software, but I don't want to repeat my previous mistakes and use non standard, expensive and soon-to-be-unreadable-by-everyone tools.

Thanks in advance for your help.

 

[Naiche]

I have the same issue...

I have the same issue...

Trying to find a "poor man's Cache-A solution" as Vlad stated above.

Proprietary and archive should not be used in the same sentence. uke:

Long term demands a format that can be accessible and supported in the long term. Wow, did I just say that?

If anybody finds a way, there are a lot of people who would like to know.

 

[Tom Goldberg]

Trying to find a "poor man's Cache-A solution" as Vlad stated above.

Proprietary and archive should not be used in the same sentence. uke:

Long term demands a format that can be accessible and supported in the long term. Wow, did I just say that?

:iagree:

Couldn't agree with you more about archives needing to be accessible and non-proprietary... that's what we are all about

I'm sorry you feel you can't afford our solution - I'd argue it is less costly than you think - take Vlad's $4k and add $2.5k for the MacPro you'd put it in and you are getting to what we charge. And, our solution is no assembly required, with remote support and a warranty. And, you get interchangeable tapes. And a searchable database of all your content.

Compare that to buying a VTR...

 

[Naiche]

If/when the price comes down, you may have a customer. Of course, you could develop a stripped down version of the product which doesn't have all the extra inputs, which has the slower HP 1760? drive inside, etc. Perhaps that could shave a few K off of the list price. Then it's more accessible.

All I need is an LTO drive (w/ hard drive cache perhaps), an ethernet port (or 2) and an embedded linux GUI like you have. Simple needs. Strip it down and see more customers flock your way.

 

[Rick Allen]

Makes you wonder how hard it be to get a barebones system put in an internal lto tape and a license of bru.

 

[Tim Jones]

A few thoughts - warning: long winded response follows

A few thoughts - warning: long winded response follows

A few notes -

Why is there no FOSS tape software on a Mac? It's mainly because you can't simply "access" a tape drive on OS X as you can on Linux/Unix systems. You can't access remote tape drives on Linux or Windows systems from your Mac without commercial software. You will never be able to use the existing FOSS/included solutions on OS X unless you write your own FOSS option to do so.

There's a reason for this. To implement that tape I/O layer, you need to understand tape drive operations and error handling, the operating systems' kernel transport layer internals for I/O (SCSI, SAS, USB, et al), and how to move the data in a format that is recoverable between the two storage device mechanisms. Managing a tape drive is not the same thing as copying a file from one disk to another disk. Apple doesn't even provide a native tape layer as found on Linux, Solaris, etc.

One thought when cost is an issue, ask yourself: "Do I really need LTO-4, or would LTO-3 work? Or even something smaller like a DAT 320?" These can be purchased for much less than a full LTO-4 solution. I know we'd all love to have a private G5, but sometimes flying coach will actually meet our needs just fine.

Or, as Rick asks, if you're positive that you can purchase raw hardware for less than various vendors' bundles, you could then simply purchase software that fits your need and budget. If you're looking at BRU, it will work with any tape device that you can attach to your Mac or Linux (or AIX, FreeBSD, Solaris, IRIX, HP-UX) system - SCSI, SAS, F-C, USB, iSCSI, or Firewire. You could build up a Linux system, write the necessary scripts, mount you Mac's storage onto the Linux box over your network and definitely perform the backups and restores in that manner. The question is - what is your time worth? Since you'll need the tape drive, cable, media, and the HBA as a minimum, is our $499 price (less if you buy it in our bundles) for BRU PE less than you'll pay yourself for the time that it takes you to design, assemble and develop a homemade solution?

As for the comment about Archive and Proprietary not belonging in the same sentence - let me ask you a question -

When your FOSS archive operation fails and you can't access any of your data, who do you call? Who is responsible? Is the author of GNUtar going to help you recover the tapes' content?​

At TOLIS Group, we pride ourselves on the simple customer mantra "BRU ... Because it's the RESTORE that matters."

Since 1985, we've had 0 (that's zero) instances reported where a BRU archive was 100% non-recoverable when physical damage to the storage media was not involved. No one can reconstruct a hard drive or tape that's been run over and crushed by the UPS truck, but we have stepped up in instances where users have lost access to their tape technology by having the users ship their storage device (disk, RAID arrays, et al) to us with their tapes, allowing our team to perform the restore. In one instance, I flew with a tape drive that was no longer made to a customer's site in Florida and performed the restore on site. In a recent instance, I even went so far as to help a client recover a batch of TAR-written LTO-3 tapes that were provided to them by the team that performed the color corrections on the original footage. Yes, they did cover the various expenses, but the shipping or flight and my hotel were a lot less costly than the loss of data would have been.

As I demonstrated in another thread, even if you can't access your BRU tapes via the GUIs, there's always the command line capabilities. Our support team is always willing to walk you through the commands needed if you're not command line savvy. And, you don't even need to restore to the same operating system. If you lose your Mac and all you have is a Linux PC, you can completely restore your OS X BRU backups onto the Linux system.

It's all a balance of what you can do and what you must depend on others for. Similar to the way hiring a RED consultant / cinematographer for a shoot is cheaper than buying a RED camera and learning to use it yourself.

Tim

 

[Tim Jones]

Not all software causes this headache

Not all software causes this headache

Of course, being a Mac guy :drool: I would like to have a Mac GUI and easy to use software, but I don't want to repeat my previous mistakes and use non standard, expensive and soon-to-be-unreadable-by-everyone tools.

Unlike other tape apps, BRU has remained backward compatible to the version that we shipped in 1988 (6.11). In fact, we have a QIC150 tape from 1989, a DAT DDS-1 from 1992, and a DLT8000 from 1998 written using the then current version of BRU that we use in our final Q/A testing before a new version is released. If the new version can't restore all of those tapes, we've broken something and it must be fixed before we ship.

You can also use tools like dd on Linux or our Tape Tools on OS X to copy the archives from one device type (AIT-2) to a completely different device type (LTO-4 or even a disk file) and BRU will continue to restore the data. This means that we're even helping you prevent restore problems caused by product extinction or technology lock-in.

Additionally, BRU versions are also compatible across OS versions. This means that a tape written using BRU in 1997 on Linux can be restored to an OS X system running Snow Leopard in 2010. Or even restore data written on your OS X Snow Leopard system onto a Solaris-managed disk farm.

While I can't do more than commiserate with you over your past failures, you wouldn't have had them if you had been using BRU.

 

[Tim Jones]

SAS HBA versus Embedded

SAS HBA versus Embedded

Are there some MBs with a SAS connector to avoid a purchase of a dedicated card?

Not at this time. SAS is still enough of a new technology that the MB manufacturers haven't embraced it. Remember, SCSI had been around for almost 15 years before we started seeing the first embedded SCSI support on a Motherboard. Also, the server boards that include embedded SCSI are usually over $200 more than a simple desktop version of that same MB. Finally, be very careful since most SAS vendors are only focused on disk I/O and it's often impossible to get tape drives to function properly (ARECA and Adaptec come quickly to mind).

Tim

 

[Naiche]

Alright Tim, we get that you are selling BRU.

But I actual much prefer the file-based archiving workflow that Cache-A employs over the session-based flow BRU uses.

As to your question....

When your FOSS archive operation fails and you can't access any of your data, who do you call? Who is responsible? Is the author of GNUtar going to help you recover the tapes' content?​

I'm sure that if FOSS-based LTO systems become prevalent there will be a number of companies to call to restore your data. By extension would you argue never to use FOSS in any mission critical situations? Many large websites would beg to differ. And since it's FOSS, anyone can enter the arena without needing certification from a company, like Tolis.

Also, redundant tapes isn't a bad idea in archiving. And also accepting that no solution is perfect or foolproof. I don't trust people who tell me that their solution is.

But you make a good point about cost, I hadn't really thought to consider LTO3, as I tend to get the latest version available in products. But now I will!

In fact, if Tom/Cache-A is still listening, LTO-3 would be another way to get the cost way down for some smaller operations, or operations who need multiple devices.

 

[Tim Jones]

Didn't mean that to sound so markety...

Didn't mean that to sound so markety...

Alright Tim, we get that you are selling BRU.

That was not the primary goal of my reply. It was to answer back to you that not all tape solutions and non-FOSS solutions are evil or will leave you where your experience left you in the '90s.

But I actual much prefer the file-based archiving workflow that Cache-A employs over the session-based flow BRU uses.

Then you will need the non-FOSS solution that Cache-A provides for any archive access since there's is the only "tape as disk" solution currently out there that I have any knowledge of.

As to your question....

When your FOSS archive operation fails and you can't access any of your data, who do you call? Who is responsible? Is the author of GNUtar going to help you recover the tapes' content?​

I'm sure that if FOSS-based LTO systems become prevalent there will be a number of companies to call to restore your data. By extension would you argue never to use FOSS in any mission critical situations? Many large websites would beg to differ. And since it's FOSS, anyone can enter the arena without needing certification from a company, like Tolis.

To your certification comment - we don't do that. If you want to support BRU customers as a consultant, that's up to you. You don't need to contact us to get blessed or anything. If you do support BRU as a consultant, we're happy to send customers your way if you let us know that you're doing it and what your specific focus is (general / film and audio / small business / K-12 edu / manufacturing / scientific, etc.).

As to mission critical FOSS, in every occasion where an organization is using FOSS for business-dependent operations, I am very sure that you will either find that they have in-house programming skills that can fix things that go sideways or are paying an external organization to provide the support required. They're not simply posting a question on a public mailing list or forum to get a resolution. My argument is that there's always a price to pay - even for FOSS.

As for FOSS-based tape systems, you can have a solution with FOSS right now. Build a Linux system, Add a SAS HBA, add an LTO-4 drive, export your Linux filesystems to the net using NFS or CIFS/Samba, get your Python books out and write a wrapper around GNUtar. Job done. The only expenses are the hard costs - which are static regardless of the software side of things - and the value of your time.

With a proper understanding of tape control and the mt tool, you could even create your own "tape as disk" type solution. All you would need to add on the Linux side are the utilities for controlling connections, multiple tape access request management, the iSCSI target layer, the web interface, and the offline store management for tapes that are not currently "mounted".

Hmmm, maybe that's why those products command the prices they do :smile: .

Also, redundant tapes isn't a bad idea in archiving.

Which BRU PE provides...

And also accepting that no solution is perfect or foolproof. I don't trust people who tell me that their solution is.

I didn't intend to claim that BRU is foolproof or perfect - there are always creative ways to defeat any supposedly foolproof solution. We just try to offer as many paths to the recovery of your data as we possibly can. I apologize if my excitement about BRU's history of success sounded pompous. It's just that I really believe in what we do.

But you make a good point about cost, I hadn't really thought to consider LTO3, as I tend to get the latest version available in products. But now I will!

We see that in a lot of customers, so you're definitely not alone :smile:

Tim

 

[Naiche]

Tim,

It's good to work with something you're passionate about, hope I wasn't too harsh.

Agreed - TINSTAAFL

Perhaps I am misunderstanding Cache-A's marketing, but I thought they did write to tape in the standard tar format. Sure their GUI might be their own, which is fine by me, but my data is being stored in a standard wrapper.

I should say I support paid software development, it's unfortunate that more people don't. Note that I asked Cache-A to reduce the cost of hardware to bring the price down, not the software part. What I don't like is getting locked in to any particular solution. Thus my love/hate relationship with Apple, Windows, etc...

I like to have options and control, which is why I am a big fan of standards and flexibility or the ability to customize something to fit your needs.

It seems like all this stuff is ramping up for warp speed change(coming from an outsider), so it'll be interesting to see where we are 1 year from now.

 

[Tim Jones]

Heinlein fans unite!

Heinlein fans unite!

Tim,

It's good to work with something you're passionate about, hope I wasn't too harsh.

Agreed - TINSTAAFL

The Moon is a Harsh Mistress!

Perhaps I am misunderstanding Cache-A's marketing, but I thought they did write to tape in the standard tar format. Sure their GUI might be their own, which is fine by me, but my data is being stored in a standard wrapper.

You are understanding it to a point. Unfortunately, there's no answer outside of using the Cache-A on the Mac since there's no natively available TAR tool that will read the tapes created. That's my only point. Even if you could read and restore them on a Linux or Solaris system, that doesn't assist in a Mac-only shop in which many of you are working. Using the tar format is only open and free if you can actually use it.

I should say I support paid software development, it's unfortunate that more people don't. Note that I asked Cache-A to reduce the cost of hardware to bring the price down, not the software part. What I don't like is getting locked in to any particular solution. Thus my love/hate relationship with Apple, Windows, etc...

Understood and agreed. It's our product's maturity that has allowed us to provide a solution for from 1/3 to 1/20 the cost of other commercial Mac backup tools that support tape hardware.

I like to have options and control, which is why I am a big fan of standards and flexibility or the ability to customize something to fit your needs.

With BRU you get that in spades. We're the same tool and format across 21 different platforms and can provide you with simple guidance to create your own solution based upon our I/O engine. You can be as simple or as complex as you choose. Heck, you could even create a tool that competes directly with what we offer.

It seems like all this stuff is ramping up for warp speed change(coming from an outsider), so it'll be interesting to see where we are 1 year from now.

And we're working very hard to show how you can have a successful archival workflow with a proprietary (whatever that means today) tape tool.

Tim

 

[Naiche]

The Moon is a Harsh Mistress!

Haha! You know my high school teacher turned me onto that phrase, but I never read that book. Some smart lines though...

I'm not convinced about BRU. I'm sure it works great, but I take issue with the proprietary archive format it uses to wrap the data. We may have to agree to disagree on this one.

Another point to make is that Cache-A's network based approach does make sense for cross compatibility, since Windows/OSX/Linux can all communicate with a linux "server" via a web interface. Not apples to apples comparison, but I do like the approach. It's just out of my range at the moment.

 

[Tom Goldberg]

In fact, if Tom/Cache-A is still listening, LTO-3 would be another way to get the cost way down for some smaller operations, or operations who need multiple devices.

Interesting idea Naiche - we'll keep that in mind... but right now we are working hard to get to LTO-5 which will be, unfortunately, even more expensive.

That's what our customers are mostly asking for but we do hear those requests for lower cost solutions and will keep LTO-3 in mind as a possibility.

 

[Naiche]

Tom,

FWIW, and I know this is anecdotal, but when I went to the open house you guys held back in October? in Burbank, CA, the number one theme I heard from people there after they asked initial tech questions is that the upfront cost was too much even if they "made there money back in the long term."

Glad ot hear you'll keep in it mind.

Best,
Naiche

 

[Tom Goldberg]

Looking for the door to summer

Looking for the door to summer

You are understanding it to a point. Unfortunately, there's no answer outside of using the Cache-A on the Mac since there's no natively available TAR tool that will read the tapes created. That's my only point. Even if you could read and restore them on a Linux or Solaris system, that doesn't assist in a Mac-only shop in which many of you are working. Using the tar format is only open and free if you can actually use it.

Not to be argumentative, but any Linux / Solaris / Unix system with an LTO-4 drive can indeed restore our tapes with a simple tar command. We have customers submitting our tapes to larger shops with that capability regularly.

I'm sure you know what you are talking about when you indicate that "Apple doesn't even provide a native tape layer as found on Linux, Solaris, etc." But there are other ways of skinning the cat.

I am under the impression that solutions like Amanda under windows would let you run tar with an LTO drive - can you advise if that would work? Can you advise if running Linux in a virtual machine on a Mac would supply the necessary driver to use tar in that environment?

 

[Jamie Hardt]

Just throwing this out there... If you're on an Intel Macintosh, you could install Ubuntu or Debian Linux with Boot Camp, and then use that to run your LTO drive, since Linux comes with a proper mt(1).

I'm currently frustrated with my sound effects archives, and I'm the sort of person that encodes his audio to FLAC, and then downloads the FLAC source code and archives it along with the files, and then TARs the files, and then encrypts and collects the MD5s of the tarfiles before uploading to S3, so I understand why you'd be reticent to go with a proprietary solution.