Anyone using LTFS with LTO-5?

[Kenn Michael]

I'm currently using BRU PE with my LTO-5 drive for archiving, but is anyone using LTFS? Is there a GUI available for it on OSX or is it strictly via Terminal? Anybody think it'll be a standard for archival?

 

[joe hedge]

I assume you were at the Ultrium seminar at DV-Expo today... Bruce certainly made it sound like all we need with the LTO-5 format on Mac is the LTFS open source software, which purportedly makes the LTO-5 behave just like any other external drive as far as dragging-and-dropping and folder-naming conventions are concerned...but you're right, it would be good to hear from anyone who's actually done it in the Red/OSX environment.

More info here:

http://www.ultrium.com/About/faq.html

 

[Kenn Michael]

Wasn't at the expo yesterday. Crazy timing :smile:

 

[joe hedge]

Yeah, very useful and informative seminar from IBM Bruce today at DV-Expo in Pasadena. Admittedly he only showed LTFS working in Windows, but he said it all works the same in OSX, Linux and Windows. There are still vendor-dependent driver issues to be worked out regarding the tape decks and HBA's themselves, but LTO-5 and LTFS definitely looked like the way to go.

 

[Jeff Kilgroe]

LTFS looks like it's coming along.

Being open source, I would like to see other software vendors like Tolis Group (BRU) incorporate support for LTFS formatted tapes, or at least the ability to read them, into their software. I think LTFS is using a variation on TAR, as Cache-A is doing with their LTO-4 and LTO-5 units. Ultimately, the tape backup/archival industry needs to standardize on a format that can be interchangeable. It all seems to be moving that way, but it's a slow process thus far.

 

[joe hedge]

Thank you Jeff, was hoping you'd see this...according to the Q&A at the seminar today, LTFS looks to eliminate the need for 3rd party software, as it allows the LTO-5 deck to behave just like any other external drive at the finder level, "a 1.5TB thumb drive" was the description Bruce used.

 

[Jeff Kilgroe]

I think at this point, LTFS warrants a look, but I wouldn't standardize any of my serious archival process on it. It needs to become more widely accepted first. I have some questions about the data format and robustness of their implementation. I'm sure most of that has been addressed, I just haven't taken the time to look for myself.

But LTFS looks really great. It's said in the spec that it is intended for LTO-5 and requires it. However, I'm seeing people adapting it to LTO-4, which somewhat concerns me as LTO-4 lacks the partitioning ability, which LTFS uses to segment off a portion of the tape for the table of contents, file descriptions, etc.. So we're getting bastardized versions of LTFS out there; only on Linux as far as I know, but so far it looks like other software to replicate the partitioning via the tape driver on Linux. Once again, not sure, just have run across it.

Having IBM as its primary developer and IP holder, LTFS is in good hands though. I don't know if it's TAR based for the actual data partition or not. Haven't looked that far into it. If LTFS is indeed robust and offering this level of convenience, then I would think it's exactly what the backup and archival industry has been needing for several years now. I'm hoping it's good and if so, adoption will be swift. I personally don't own an LTO-5, but have been starting to look at them. I will probably purchase an LTO-5 early next year, but I'm currently running two LTO-4 units, both HP 1760 desktop LTO-4's. I intend to replace one of those with the new LTO-5.

 

[Mike Cavanagh]

LTO-5 synchronization from disk

LTO-5 synchronization from disk

I'm currently using BRU PE with my LTO-5 drive for archiving, but is anyone using LTFS? Is there a GUI available for it on OSX or is it strictly via Terminal? Anybody think it'll be a standard for archival?

Hi Ken- Went to a SMPTE Hollywood chapter presentation last Tuesday (11/16/2010) where Dr. David Pease, Senior Technical Staff Member
Manager, Storage-Specific Solutions and Services, IBM Almaden Research Center, Ed Childers, Senior Technical Staff Member and Manager, Tape Development IBM and Tridib Chakravarty – CEO, StorageDNA outlined LTO-5, LTFS and supporting technologies. With LTFS being open source, the industry will be seeing a lot of solutions coming to market around the overall technology. Tape for archive is still a valid, cost effective solution. The cost of a tape for LTO-5 of 1.5TB is sub $100. The problem has been the wrappers like TAR, etc. LTFS gives a robust directory for searching. With LTFS being open source, the industry will be seeing a lot of solutions coming to market around the overall technology.

StorageDNA has one of the first sync check, snapshot based systems for archive. What they we're able to do was leverage existing technology (http://storagedna.com/solutions-san.html) to sync to the LTFS LTO-5 drives. I've installed the system at Key Code Media in Burbank. The following link outlines some additional IBM-LTO-5/LTFS for media (http://keycodemedia.wordpress.com/2010/11/16/ibm-tape-and-media-production…there-is-a-connection/).

Ping me if you'd like more info.
Mike Cavanagh
818-303-3900
mcavanagh@keycodemedia.com

 

[Tim Jones]

Danger Will Robinson... LTFS curves ahead

Danger Will Robinson... LTFS curves ahead

Folks,

I know that some will read what I'm about to write as sour grapes, but TOLIS Group is actively developing for LTFS support in future versions of our BRU products. In the mean time, I'd like to share the following thoughts about the tech.

Please be cautious with your enthusiasm about LTFS. My team has been following the progress since first announcement and it still has a LONG way to go to becoming a real technology. Currently, you can only use it on a very specific Red Hat Linux platform. Support for OS X is "indeterminate" and Windows support is still listed as mid to late 2011.

I realize that the LTO group and IBM in particular are pushing this in an attempt to reinvigorate tape technology in a world where major marketing is telling you all that disk is the wave of the future, but they've missed the simple fact that LTO-5's combination of speed, capacity, reliability, "green-ness", and cost per gigabyte have already done this.

However, even if it does come to be, there are a number of factors that you need to keep in mind based on the current design parameters:

1: Access is not as fast as a true backup application for either writing or reading as the filesystem layer provides quite a hit (in Linux tests, throughput rates of under 40MB/sec are currently consistent). Users are witnessing 140MB/sec using BRU and 120MB/sec using tar with LTO-5 on OS X systems right now.

2a: It's not "update friendly" because when you delete a file, the space on the tape is not returned. This means that if you are editing a shoot on an LTFS tape, multiple edits of the same clips can fill an otherwise empty tape.

2b: When you do perform an update the process of marking the old copy removed and updating the directory pointers for the new copy can take twice as long as the original write.

2c: To recover the lost data space on a tape volume, you must reformat the tape - losing all of the current data.

3: When a LTFS tape fills, you can't cross over to another tape, you simply run out of space. With a true backup app, your data can spill over to many contiguous tapes for theoretically unlimited capacity (16 Exabytes with BRU).

4: Unlike a true backup and archival app, there's no current mechanism for tracking what's on a tape once it's removed from the drive. This means that you'll only know about the data that's on the currently loaded tape.

There are other, more nuts-and-bolts issues that developers must be aware of, but those are the biggies for end users.

As to Jeff's comment about TOLIS Group supporting it, we are already implementing framework support based on the specifications, but with no true working models, all we can do at this point is design and stub. If it does become a standard, BRU products will help alleviate issues 1 and 4. However, it will be up to the driver and filesystem implementation experts to improve on 2b and 3.

Tim

 

[Tim Jones]

I think LTFS is using a variation on TAR, as Cache-A is doing with their LTO-4 and LTO-5 units.

Slightly incorrect there - LTFS stands for Linear Tape File System and it will require specific device drivers and filesystem level support at the OS level, not by a software vendor. The "idea' is that you simply load the tape and your computer knows what to do with the contents or space as if it was a hybrid disk drive.

Tim

 

[Tim Jones]

One more thought ...

One more thought ...

While it's a nitpick - your LTFS tapes will only be compatible on platforms where the LTFS drivers and filesystem support exists (currently only Linux, Windows and maybe OS X). Unlike tar or BRU where coverage is already ubiquitous, LTFS adoption will be highly dependent on the OS vendors for proper support.

Tim

 

[Tim Jones]

Equal - well, sort of...

Equal - well, sort of...

Yeah, very useful and informative seminar from IBM Bruce today at DV-Expo in Pasadena. Admittedly he only showed LTFS working in Windows, but he said it all works the same in OSX, Linux and Windows. There are still vendor-dependent driver issues to be worked out regarding the tape decks and HBA's themselves, but LTO-5 and LTFS definitely looked like the way to go.

Unfortunately, it doesn't currently work the same in all three OSes and you can't even get a Windows driver outside of IBM's test world for now.

Remember, take marketing input with a teaspoon full of precaution :scared:.

Tim

 

[David T]

Some straight talk on LTFS

Some straight talk on LTFS

In response toTim Jones’ post we would like to provide the following:

As of November 2010… Supported OS’s include:
· Windows 7 (32 bit) became available in late Nov. (Windows 64-bit planned for the 1st half of 2011)
· MAC: OSX 10.5.6 support was available with the original announcement (HP lists OSX 10.6 as well)
· Linux: Red Hat Enterprise Linux 5.4 & 5.5
[FONT=&quot]o [/FONT](HP lists SUSE Linux Enterprise Server 11 as well)

( http://www-03.ibm.com/systems/storage/tape/ltfs/ or
http://h71028.www7.hp.com/enterprise/us/en/solutions/storage-linear-tape-file-system.html )

For IBM, Downloads and Fixes are at:
http://www-933.ibm.com/support/fixcentral/
Once there, use the following navigation tips:

For Product Group select Storage Systems
For Product Family select Tape Systems
For Product Type select Tape device drivers and
software
For Product select Long Term File System
(LTFS)

For HP - http://h20000.www2.hp.com/bizsuppor...peId=12169&prodSeriesId=4249221&lang=en&cc=us

Regarding the ‘raw’ strengths of LTO-5: All that Tim mentions is true (re: speed, capacity, reliability, "green-ness", and cost per gigabyte) about LTO-5. What LTFS does in addition is enable those benefits for a larger and more diverse set of users. Once the LTFS driver is installed, ANY application can realize, through judicious access, the compelling benefits of LTO-5.

As an open format, storing valuable data on LTFS is superior to proprietary backup or archive programs which require client licenses and a compatible version for every user. TAR is open but inferior to LTFS due to its high geek factor and susceptibility to loss of data from corruption of the tar headers.

Users are very familiar with USB storage devices and LTFS enables LTO-5 tape to follow the same paradigm: you plug it in and it looks like a disk drive.

Here are two examples:
Imagine having source material that exists only on data tape. While editing a scene that the director requires immediately you realize you need that source material. If the source material is stored on an LTFS formatted tape you would simply need to drag-n-drop the file into the clip bin of your editing system and complete the scene for the director. If that source material had been stored in a TAR format or a proprietary tape backup format there would be multiple steps to retrieve the file including potentially waiting for the transfer of the entire contents of the tape to disk before having direct access to the specific file needed.

When using LTFS you get a self-describing tape volume (cartridge). So when a LTFS tape is written and then delivered to another facility, the next user simply has to mount the tape, not completely unload it, to see all the files and their related directory data on that tape - no proprietary system required for this task, just an LTFS (open source) enabled workstation of server.


In response to the numbered points:
1.) Lab tests on Linux have demonstrated the ability to read and write from/to a clean LTFS formatted tape at the hardware rated speed of 140 MB/s. Tests using standard OS copy utilities have produced rates in the 70-90 MB/s range. LTFS will perform better with larger files than with files under 10 MBs. One of the value-adds application vendors can provide is designing their applications to exploit LFTS’s strengths and to avoid, where possible, behaviors that cause performance issues. Several vendors have already done this.


2a.) There are multiple concepts intertwined in this comment. While LTFS makes tape look like disk, it still must act like tape. No data tape technology (LTFS or not) allows for updating a file in place without either writing the update as an appended file at the end of the data currently on the tape OR updating the file and also rewriting all the data on the tape that follows (which means you had to read it all first). In addition few, if any, data tape formats support the traditional video tape edit concept of insert editing.


2b.) Updated data must be written to the end of the tape so delays due to seek time are expected. Directory operations use data cached in memory and take minimal processing time. Note also that a read of a file with non-contiguous blocks due to updates can take longer as the tape seeks from the original data to the updated data and back. Update supports some key use cases where the overhead of a minor update is much more efficient than rewriting an entire file, but in general should be used sparingly.

2c.) This is an issue common to all data tape applications and has its own dedicated term: reclamation. Sparse valid data is copied off to a new tape and the old tape can then be reclaimed in its entirety. Data subject to extensive changes is probably better suited to disk storage.

3.) LTFS is not a backup or archive application. It is a file system to be used by applications A backup or archive application would manage LTFS tape overflow analogous to how it’s done for LTO-5. . . except the tape would be more useful afterwards. There is no limit to the number of tapes that could be used, just as there is no limit to how many disk drives you could plug in to a server and then remove. Any limits would be in the tracking application, not in LTFS itself which operates on a per tape basis

4.) Like 3 above, LTFS is not a backup, archive or MAM (Media Asset Management) application. It is a file system to be used by those and other applications. LTFS tapes can be used in 2 modes, standalone as a self-describing entity analogous to a USB drive, and/or centrally managed by an application which tracks each tape’s contents. Most backup and archival applications operate ONLY in the centrally managed mode. Their tapes are useless, or difficult to use, standalone. There are already dual mode applications currently in the marketplace utilizing LTFS that provide tape spanning and keep track of all the content that has moved through them, whether the particular tape is mounted or on the shelf. DNA Evolution from StorageDNA is one example.

For more information on LTO-5 and LTFS, this month’s SMPTE PDA Now webcast, 1/13/11 at 10:00 AM PDT, will cover these topics with the opportunity to ask questions.

David and George – Media Technology Market Partners (MTMP)

 

[Kenn Michael]

I downloaded the LTFS driver for my HP drive and tried the workflow on OSX 10.6.4 a few weeks ago.

I was able to format a tape and mount it on the desktop which was very cool. I could drag and drop files to the tape and restore those same files. Doing one or two at a time worked fine - but when I tried copying a huge project over with thousands of files, I got errors at random points during the copy/write operation. Only had to try this three times with different sets of files to ultimately put it down and wait for some more robust implementations for OSX.

I see the immense potential, but at the moment, I don't have confidence in doing large backups with LTFS on OSX with my HP drive. Is there even a verify function built-in to LTFS?

 

[Tim Jones]

LTFS 1.1.0 Still not quite there

LTFS 1.1.0 Still not quite there

Warning - long-winded response follows. If you're really not following the LTFS discussion, I recommend you grab the next new thread :smile5: .

I've read David's response and I do appreciate his posting the links to the downloads from the various vendors. However, his arguments actually help to make my points. While the various marketing arms want users to believe that LTFS is disk-like, the technical truths point to something entirely different. I did chuckle outright at the statement (emphasis mine):

ANY application can realize, through judicious access, the compelling benefits of LTO-5

While LTFS (Linear Tape File System according to LTrg and HP, Long Term File System according to IBM) has promise, the current implementation is failing in a number of situations. Let me state again that we think LTFS will have a place in a high-end desktop environment once it has matured, but after the report I've received this morning, that maturity still seems a bit away.

Here's an Executive Summary of the technical data below -

After testing the latest 1.1.0 versions of the LTFS code on the last three generations of Apple Mac Pro systems running 10.5.8 and 10.6.5, we can only come to the conclusion that LTFS is not anywhere near ready for real world implementation under Apple's OS X. There is no non-BETA 64 bit support (a missing part of MacFuSE, not the LTFS binaries) and copying more than a file or two in the Finder on a system where it does work results in write failures that can leave a tape in a corrupted state after ejecting and remounting. In real world tests, it's still not as fast as using the ordinary archival tools like GNUtar and BRU. And a properly managed catalog system makes locating and restoring files on tape in these archive formats just as fast as LTFS when it does work. Please note that I'm not talking about Windows or Linux - even though there are still issues on those platforms as well. I've forwarded detailed logs to engineering contacts at HP and IBM on this in hopes of seeing this status improve.

Again, this is not competitive sour grapes because we will be supporting it in the BRU family of products once it reaches a mature and stable state, but rather a simple bit of cautious guidance from someone who believes that your data is the most important aspect of your computer systems.

Technical Details follow - please wear appropriate eye protection :coolgleamA:.

In one evening in our lab, my test team have reported the following:

• Linux drivers only work properly with the limited kernels provided by the Red Hat team in the current RHEL. This means that there is no opportunity for non-RHEL users to use the drivers under Linux without understanding fully how to update the source code and kernel for their platform.

• Even with the new 1.1.0 code under Linux, our sustained write performance still only hit 78MB/sec using 550GB of actual video and audio data (versus manufactured data) that provides 142MB/sec using BRU and 128MB/sec using GNUtar. We tried LTFS block sizes ranging from 64K to the default 512K.

• The OS X implementation requires that the tape be formatted for use. This process took a surprisingly quick 60 seconds. The time for partition changes must have been dramatically improved by HP in the latest firmware as our original tests required format times of over 5 minutes on the original Linux test platform during the partition change phase. We're verifying this with our HP engineering contacts.

• The OS X drivers required a lot of additional work since we (and we suspect most end users) don't have the MacFuSE 3rd party filesystem framework installed by default.

• As on Linux, the tape must be manually mounted, so the claims of "just like a USB stick" are not quite true. This can probably be resolved with a launchd entry, but we don't know what to look for explicitly to recognize media insertion and enable automounting.

• Upon mounting, we received the following errors under 10.6.5 on a MacPro5,1 system:

/Library/Filesystems/fusefs.fs/Support/fusefs.kext failed to load - (libkern/kext) link error; check the system/kernel logs for errors or try kextutil(8).
the MacFUSE file system is not available (71)
LTFS20091E Unable to get directory properties for '/Users/Lab1/Library/Logs/LTFS' - No such file or directory
LTFS20076I Triggering drive diagnostic dump
LTFS20090W Unable to create dumpfile '/Users/Lab1/Library/Logs/LTFS/ltfs_20110104_091251_HU1042D5W9.ltd' - No such file or directory
LTFS20079W Unable to save drive dump to file
​

The result was an unusable tape filesystem.

This also identified that the current installer does not create the LTFS folder in the user's Logs folder, causing a further error.

We have verified that our MacFuSE version is 2.0.3,2 as specified in the INSTALLING.txt file and the OS X Installer and that there are no non-BETA updates available.

To take it a step further, we installed MacFuSE 2.1.5 onto a non-production system and received the same errors. It turns out that MacFuSE is NOT K64 compatible, so if you are running a 64 bit OS X kernel on a newer Mac Pro (3,1, 4,1, 5,1), the fusefs.kext will fail to load and you're stuck. Checking the MacFuSE forums and WIKI, it is clear that we won't have 64 bit support from Amit for some time. Since OS X ships on modern Mac Pro systems as 64 bit by default, this means that LTFS is not available on new systems. We are looking at reloading one of our older MacPro1,1 systems with the 32 bit kernel to see if we can get anywhere with this.

--------End of Lab Email--------​

After some further research this morning, we located a test build of MacFuSE 2.1.7 which is 64 bit. While the kernel extension loaded, and we can mount the LTFS tape on a MacPro5,1 64 bit system, attempts to copy data in the Finder resulted in write failures, however, copying a single file appears to result in the file being copied, however, accessing the file in the LTFS folder resulted in the original file being accessed. The actual message is:

The Finder can’t complete the operation because some data in “01 Don't Cry.mp3” can’t be read or written.
(Error code -36)​

Trying the same operations in the Terminal resulted in a reported successful copy.

We then ejected the tape from the Desktop and then physically ejected the tape. Upon reinsertion, the tape now showed the failed folders that were previously copied with the failure and the single files. Accessing the single files accessed the tape, so simple single file copies - ignoring the Finder error - work. Large drag and drop copies in the Finder appear to properly create any folder structure at the top level, but the contents of the folders fail to copy. This makes the LTFS practically useless in the day to day Mac environment.

 

[Tony Kliphuis]

LTFS does work.
On a new xeon (true 64 bit) MAC Pro OS X 10.6.
I did have difficulty finding a MacFUSE package for 64 bit kernel. I did find this:
http://static.caurea.org/MacFUSE/MacFUSE-2.1.7.dmg (it works)
I then installed the HP binaries.

With both IBM and HP providing some level of support I am optimistic that LTFS does have a future.

Performance is good.
110 MB/sec from internal HFS drive to tape.
70 MB/sec from eSATA NTFS drive to tape.
Restore time from tape to drives are about the same.

We are testing LTFS to archive large video files and typically fill a 1.5 TB tape with 30 to 60 files.

I like that LTFS does not let me span a file across multiple tapes. We, and our customers, don't like the first part of a file being on tape 40 and the remainder being on tape 36.

LTFS is not ready to be a complete solution. We will continue to use Bru and Retrospect for years to come.

As I continue to test things I will provide updates if I have anything intelligent to say.

Peace,

Tony

 

[Curran Giddens]

http://cache-a.com/canews/Cache-A_HP_LTFS_012511FNL.pdf

Interesting...

 

[Curran Giddens]

http://melrosevideoprofessionals.wo...5-product-line-including-flagship-pro-cache5/

 

[Tim Jones]

And so does BRU PE ...

And so does BRU PE ...

I guess we could follow that up with:

TOLIS Group, Inc. announces support for LTFS in the 2.3.0 release of BRU PE. BRU PE adds a full blown LTFS Manager tool which allows users to format, mount, unmount, and import LTFS volumes into BRU PE so that the LTFS volume contents may be accessed even when the media is offline or offsite. BRU PE's LTFS support is fully compatible with the LTFS volumes created buy other vendors' products on all three supported platforms.

LTFS support is only available on Intel-based Mac Pro and Xserve systems running Snow Leopard 10.6.5 or later.

LTFS support is a no-cost update for all BRU PE users.

TOLIS Group will be showing it off at our booth at NAB (SL11909) and shipping BRU PE 2.3.0 either the week of or the week after NAB.
​

However, everyone still needs to be aware that LTFS is not disk.

Also, the following statements are true for ALL LTFS implementations on the Mac platform:

• You are still accessing serial media.
• LTFS volumes cannot readily support multiple simultaneous read/write operations. While you can do it, it is incredibly slow.
• You should not use it as live storage (i.e.: don't edit files on the LTFS volume directly in Final Cut, Media Composer, etc.).
• Access in Finder is VERY slow because Finder tries to read the file metadata every time you expand a folder - this translates to a lot of seeking.
• Access from the command line is not as limited since command line tools like cp, mv, mkdir, bru, etc. don't access the metadata in that manner.

Hope to see you all at NAB! We're in booth SL11909.

 

[Tim Jones]

Yes, now it does.

Yes, now it does.

LTFS does work.

And we agree. See the announcement above.

[snip] -----8<------

I like that LTFS does not let me span a file across multiple tapes. We, and our customers, don't like the first part of a file being on tape 40 and the remainder being on tape 36.

If you're using BRU, you will never run into such a situation. Having to deal with splitting up your data sets before time is a waste of both your time and tape space. Let BRU do what it's been designed to do. All BRU archives are sequentially written, so if a file starts on tape 40 and needs to cross over, the remainder will be written to tape 41. We know other apps can do what you mention, but we've never understood why they would either want to or need to. Additionally, because BRU properly uses a tape drive's "Recover Buffered Data" mechanism when we see the early warning flag for end of tape, we properly close the current volume, recover the last buffer of data and rewrite it to the beginning of the next volume. A slight bit redundant, but we live by the rule "better safe than sorry."

LTFS is not ready to be a complete solution. We will continue to use Bru and Retrospect for years to come.

Hopefully, with the new feature set in BRU PE, LTFS will become a useful addition to our customers' archival needs.