LTO-3 still the way to go?

[Alex Carr]

I just purchased LTO 4, which is backwards compatible, Etc...

My point is, a Client will never understand how this works, or even need to worry about how it works. When you hand over DigiBeta tapes to a client what do they do? Put it on the shelf, and when they need something they send it to a Post House to ingest.

Just tell a client that it takes a Technician to retrieve their data, instead of educating them about something they really dont want to understand. It's a tape it holds information for longer periods than HDD's... anything more is waaaay too much.

Besides I just picked up a used Lto 4 for $1300 with a SAS Card & 4 800GB Tapes on Ebay. Why bother with Old Tape Decks and non-standard ones at that.

 

[albert rudnicki]

I've actually got the HPT 2322 also in my system and it doesn't seem to conflict with anything. I will grab the drive tomorrow and try and do a backup of the RED footage from my last shoot. Will let you know what happens.

Matthew

Thanks Matthew, much appreciate.
Cheers

 

[albert rudnicki]

I just purchased LTO 4, which is backwards compatible, Etc...

My point is, a Client will never understand how this works, or even need to worry about how it works. When you hand over DigiBeta tapes to a client what do they do? Put it on the shelf, and when they need something they send it to a Post House to ingest.

Just tell a client that it takes a Technician to retrieve their data, instead of educating them about something they really dont want to understand. It's a tape it holds information for longer periods than HDD's... anything more is waaaay too much.

Besides I just picked up a used Lto 4 for $1300 with a SAS Card & 4 800GB Tapes on Ebay. Why bother with Old Tape Decks and non-standard ones at that.

You got it for a great price and I am glad it works fine for you.
Personally I would never get a second hand Lto drive or any media drive as a matter of fact; at this stage you really want a maximum insurance that the unit is reliable.

 

[warwickt]

I just purchased LTO 4, which is backwards compatible, Etc...

My point is, a Client will never understand how this works, or even need to worry about how it works. When you hand over DigiBeta tapes to a client what do they do? Put it on the shelf, and when they need something they send it to a Post House to ingest.

Just tell a client that it takes a Technician to retrieve their data, instead of educating them about something they really dont want to understand. It's a tape it holds information for longer periods than HDD's... anything more is waaaay too much.

Besides I just picked up a used Lto 4 for $1300 with a SAS Card & 4 800GB Tapes on Ebay. Why bother with Old Tape Decks and non-standard ones at that.

Agree, that's certainly an excellent price, especially with 4 x Ultrium4 tape cartridges.

What software are you using with it? or are you just using native cli tape commands.

 

[Allan Stallard]

After extensive reading here and elsewhere we decided on LTO-3A. The decision was not based on the cost of the actual unit but based on the fact that LTO 3 or 4 (NON "A" versions) requires the same software to extract the data as it was written. LTO-3A lets us access the data from any FTP application. I was also told numerous times that LTO 3 & 4 could be considered "backup" solutions and that the "A" series could be considered "archiving" solution for whatever that is worth.

 

[Michael Lindsay]

We've gone LTO4 with retrospect for MAC. Seems to work fine 3gb per min easy.

However it does bother me ALLOT that there is no compatibility mode for back ups (sorry archives) that would allow other software to read it. (the previous posters 3a point is very interesting).

Is there anything on the horizon? Or is there the possibility for some media archive only software becoming an attractive development opportunity (with all thousands of media users leaving video tape for data tape)?

regards

Michael

 

[albert rudnicki]

That is interesting point. So in case of LTO 3 or 4 when they say backwards compatible it actually means nothing. Your tape format may not be readable in couple of years?

 

[Mitch Deoudes]

That is interesting point. So in case of LTO 3 or 4 when they say backwards compatible it actually means nothing. Your tape format may not be readable in couple of years?

My understanding - and someone should correct me if I'm wrong - is that the hardware is actually backward-compatible, i.e. an LTO-4 drive will read an LTO-3 tape. However, the various proprietary archiving utilities each write their own format archives. So, similar to the way that you could save the same text in a Word document and an OpenOffice document, and be locked into the software, while the floppy you saved it to will read in any drive. (Does anyone remember floppies?)

However, an LTO-3A tape can only be read by an LTO-3A drive. So while you're technically not locked into the software, you are locked into the hardware... which to me seems just as bad.

All of this is a thinly veiled attempt to get someone to answer my previous question about "tar" on LTO. Tar is an open format, and is as close as one can get to ubiquitous - in fact, I think even some proprietary backup solutions can be forced to write tar. It's like saving your text file as plain text. So tar on LTO-3 would provide no software or hardware lock-in. (Or close to it.)

 

[Allan Stallard]

All of this is a thinly veiled attempt to get someone to answer my previous question about "tar" on LTO. Tar is an open format, and is as close as one can get to ubiquitous - in fact, I think even some proprietary backup solutions can be forced to write tar. It's like saving your text file as plain text. So tar on LTO-3 would provide no software or hardware lock-in. (Or close to it.)

Even though tar is open and would be great to use, just how do you "talk" to a LTO-3 or 4 deck without the backup software? Tar formats to me are just zip files, I see no way for the app itself to have handles to read/write to a deck.
I agree it would be nice to have a utility that was "open" to be able to write files to LTO non A series. If someone is holding please share!

Being locked into the hardware is no better than being locked into software I agree. But I think it also depends what you are using it for. We use it for internal archiving of our SAN. As for being locked in to hardware we have never given it a second thought that a lot of our old archives are on Digi-Beta. I guess you have to decide how long you need your footage/data. In the commercial world, not sure about others, but we do not intend to keep footage indefinitely. In the feature/doc world I can see wanting to keep that preserved forever, in which case you are probably going to have to move your data around at some point anyways to a new format.

As for backwards compatibility I have read and have been told by Quantum that it will always be 2 generation of backwards compatibility of read only. (to be able to transfer data)

 

[Jeff Brue]

if you're intereted in tar based formats I'm looking at Xendata. Essentially it presents a tar based LTO strategy as a file system drive... Really think it would be great as the tapes written arne't proprietary and you can access your files like a hard drive. It's windows only but thats what Vmware is for.

In the end though if going for a tape back up strategy I highly encourage a open source approach. I can get anything back from tar based LTO's even if the tape takes a hit it can be repaired because you can just fast forward past the bad tar block. Not something you can do with quantum or bru.

 

[Allan Stallard]

Do you mean the software Quantum provides (Symantec) or the drives themselves as I was under the impression that most LTO drives were Quantum...

 

[warwickt]

HI Meotech,

1. the Ultrium tape format (the single spool job) is a standard in the I.T. world. It is supported by the LTO (the Large tape organisation) who were (and are) HP, IBM, Seagate(Certance), and others now like Quantum etc.

2. It is writable with tar ans in any other .appp/util that can be recognised by the OS + logic that supports (in this case) basic SCSI command tape set.

3. one MIGHT consider data formats form SUN/STK (9840, 9940, T10K, 9490), IBM's 3592, 3490, Quantums 'DLT' + SDLT, SONY's AIT and SAIT , Mammoth, DDS, Travan etc etc also 'proprietary' however I believe this is nonsense because of the sheer popularity and penetration of these devices particularly over the last 15 years+ (for some) . Simply there is NO one particular tape format that fits into and and can be read or written into al manufacturers. the LTO consortiums ULTRIUM series comes the closest and is usually READ backward compatible.

3. MOst tape drives using SCSI, SAS or FC interface can be used with OSX and LINUX AS LONG AS THE relative HBA supports it! (you must check for this). THere is some good software out there for ARCHIVING (and backup if you must) that supports standalone tape drives and also media exchangers (such as tape robots). This software ranges for the single user stuff from say TOLISGROUP BRU (see this forum) to the high end SGL FLASHNET, FPDI DIVARCHIVE (linux.. maybe solaris + windoze) and XENDATA to name a few. If you want to always have a .tar format, then be aware that the high end stuff uses its own formats. In the case of FLASHNET it wraps all objects into MXF then archives them. Tolsgroups BRU PE/LE uses tar.. so you're safe there to open the tape volume using tar. Naturally you will have to negotiate the metadata with a crystal ball and a local mindreader at your side.. (as an aside, I was a retrospect user ages ago, aside form its relative lack of features these days, it does not use an open format.)

5. if the OS can see the tape device at initialisation time, then it is highly probably that the OS's basic command set (e.g tapectl in UNIX) to locate (seek), rewind, make file marks, 0, then most .apps running on that platform that utilise tape will work. Again I'd highly recommend you work with the tape drive vendor and the HBA vendor. Right now a REAL SAFE bet is HP + ATTO on OSX.

5. basing the assumption that you have used an .app or util that uses an "open" write format [tar], you can read/"extract" a 'file' by negotiating TM's (tape marks) , file markers and EOF's etc.. even BLOCK NUMBERS (locate block aka 'seek') with OSX shell commands for example.

However it is much more productive to use an .APP that has all this metadata stored away with a super easy to use UI that has active interfaces into your workflow production tools if possible.

Recently at IBC I saw CATDV (a MAM for OSX) with assimilation of FCP IMPORT and they claim assimilation with TOLISGROUPS BRU LE(PE)... so this is very cool.

HTH
w

 

[warwickt]

Do you mean the software Quantum provides (Symantec) or the drives themselves as I was under the impression that most LTO drives were Quantum...

the Quantum Software I received with my ULTRIUM4 SAS desktop tape drives was really super useless. It is as you point out a Symantec (Veritas) BACKUP EXEC for windoze 2003 server.

USeless to be because:
• it is not for OSX
• is a BACKUP regime not an archiver , which is what I think most of us need for our workflow.

w

 

[warwickt]

In the feature/doc world I can see wanting to keep that preserved forever, in which case you are probably going to have to move your data around at some point anyways to a new format. For myself I can READ ULTRIUm3/4 and write ULtrium4. Within 5 years+ I will need to move that content from those data Ultrium4 data tapes to something else cost effective that meets my workflow.

This is an interesting thread! however there maybe some confusion about purposes for using data tape for an ARCHIVE MEDIA format and a then as a DISTRIBUTION MEDIA format.

Distribution of CMF on a DATA TAPE
Many on this thread are correct in saying that for distribution, that many recent/contemporary removable media storage carries (Utrium2 vs DLT vs Ultrium3 vs BD) may be troublesome for the customer/client. In short if the recipient of the distribution format does not have the physical media carrier, then they are hosed and so are we as well. I don't have an easy answer for this but there are obvious workarounds.

CMF (contemporary media formats) Archive using DATA TAPE

OK this is easy to answer. As the technology changes you MUST change over time to use it. Same as most of have moved from one acquisition format to another. Sounds a little presumptuous, however this is how it is in the I.T. world since at least 1960.

The archived material MUST be transferred (copied) to the new media type (carrier) usually under the control of the archive/backuo .app. THis is simply done. For larger houses with a tape library this is a brainless activity as the software such as FLASHNET™, DIVArchive™, XENdata™, Masstor etc will do it for you. For us in small shops, we can take care of it ourselves but need to be very tidy with the archival metadata. Here's where some software comes in. So this is the way to go. (see this thread for many examples of archival software).

Thus this needs to be weighed up for financial benefits. MOving to a higher capacity media (Ultrium4 @ 800GB native) from say Ultrium2 (200GB native). Should you be using DLT7000/DLT8000 today that's over 11-12 years old tehnology nd not really supported by uantum and worse parts forthe tape drives are hard to get, the media take up great space compared to ULTRIUM 4 (meaning the content capacities are higher for the new formats).. and so on.

Simply moving older stuff from CD's-->DVD5's-->BD's is the practice these days and from BD---> SVOD (spacial volumetric optical disks)--> beyond? who knows.

As for backwards compatibility I have read and have been told by Quantum that it will always be 2 generation of backwards compatibility of read only. (to be able to transfer data)

YEs certain and possibly further as the market demands.

 

[Mitch Deoudes]

5. basing the assumption that you have used an .app or util that uses an "open" write format [tar], you can read/"extract" a 'file' by negotiating TM's (tape marks) , file markers and EOF's etc.. even BLOCK NUMBERS (locate block aka 'seek') with OSX shell commands for example.

However it is much more productive to use an .APP that has all this metadata stored away with a super easy to use UI that has active interfaces into your workflow production tools if possible.

Warwickt - thanks for the info. With respect to your point above... Am I correct in saying that navigating tape marks is only necessary when there's more than one archive on a tape? That is, you first position the head at the beginning of the archive, and then tar handles the rest? (So "tar -x" to extract a file or directory, etc.)

Also, in terms of metadata being stored by proprietary software - what sort of data are we talking about, exactly? Aside from the usual unix info (date/permissions/owner/etc.), and maybe the date & other stuff you can write on the tape with a pen, I'm unsure what sort of data would be necessary... At least in the context of backing up R3D's & tiff's en masse, then nuking them off your drives. Possibly more complex in a general IT environment, where you may be backing up a rapidly changing file/directory structure.

 

[warwickt]

Am I correct in saying that navigating tape marks is only necessary when there's more than one archive on a tape? That is, you first position the head at the beginning of the archive, and then tar handles the rest? (So "tar -x" to extract a file or directory, etc.)

sort of. You need to know the file sequence number. ,, however this is why a commercial .app is far better because it maintains this type of metadata...

.. "files" are just structures that are maintained between KNOWN data structures such as File Marks (FM)| tape MArks (TM's) ) and other things on the tape media. THese are NON-DATA individual items that are overheads on the medium used to help segregate the "data" and are only accessible using tape controller commands for positioning etc.

The file system is usually the component that organises these transparent non-data tape objects, else you can do this yourself using many of the tape control commands that invoke the SCSI command tape set or write something that does it yourself, (taking great care to handle the error situations). i.e a driver.

FM/TM's in many systems deliniate files on a tape volume.. I guess you could say in most cases a single tape mark (TM) = file or data separator. Hence references to FILE MARKERS

i.e (crude example [TM=FM)

file001[ASCII/EBCDIC_volume_label_data] + TM+TM + file002 + TM+TM +file003 + TM+TM +.... filennn+TM+TM .......... >EOT

Thus a SCSI SPACE (FSF) command will use a non-READ mode to locate the NEXT TM+TM in the tape or until EOT/EOM (physical end of tape/media) is reached. (then the OS file system jumps recovery in and takes over). (all varies from tape drive model BTW). Simply the tape controller genertes an interrupt and a particular STATUS when a FM/TM is found, you need to then look at it to see what to do next. (the tape transport is stopped at this time).

It is simple to write tape/file marks after any block of data on a tape (see scsi write filemark command). THis is DISREGARDED by the TAPE controller for a READ DATA COMMAND but drive a STATUS usually when they are found as the next non data block. THese are useful for navigating, but these days unless you're a real pro at writing this stuff, its best to stay simple.

These similar controller positioning objects appear for disk controllers as well (not tape makes of course!). There's loads of info in this type of stuff.

the construct of a file is anything that is not one of these tape controller objects. Ths ANY BLOCK of data is just that.. data.

Files exist as formal contructs on a tape volume. There's ANSI and other standards out there that are decades old to describe this. There are other 'FILE" contructs on a standard tape volume like 'VOLUME SERIAL" labels, , maybe first file on the tape.. etc .. too long to go into here.

Summary-
• an 'ARCHIVE" is one or more tape FILES (data blocks seperated by at least on TM/FM).
• the file can be a COMPLEX OBJECT such one of the R3D V00x_C00x components such as the .R3D or one of the _F.move or _H.move objects.
• if you tar'ed these objects individually (3 separate operations) you would this have THREE (3) file on the tape volume.
• if your approach was to make a TAR 'archive' you would have one file ( whose size would be the sum of the 3 components above).

Further, for smarter tape controllers (drives) and software....
As TM's are just objects/data structures recognised by the tape drive controller. FILEMARKERS in many cases are a series of tape/file marks (TM's/FM's). As above, two TMs in succession with no 'other' structures are seen by many tape drive contollers as an END-OF-DATA ( i.e. eof=TM+TM. )

Simply a FSF | SPACE ( forward space file) scsi command negotiates the tape drive to find the next "FILE MARKER" in any tape. This is recently done at seeking/locate speed of (8m/sec for example).

THus second lowest level of tape nagivation on a tape drive is the space/FSF (fwd space file) or similar in other worlds. As above you can NAVIGATE by FILES.

Thus this is known from the LOAD point of the tape (BOT) . usually the VERY FIRST BLOCK on the tape when it is MOUNTED on the tape drive. (not the same for Ageus or STK9840 for example (these have TWO SPOOLS in the cartridge like a standard video tape). THese mount in the CENTRE of these tape media.

(A lower level of navigation exists using LOCATE BLOCK (aka locate locate block or seek), using high seed tape transport commands to movenavigate the tape volume directly to a specific block on the tape volume at say 8 or 10M/sec).

Thus, if you had a schema to MAINTAIN some of the tape block information, then you could naturally use 'warp mode' in the tape drive controller to quickly locate/position anywhere within an OBJECT on a tape regardless of what ever file it as in. Note today using special s/w you can transform a TCR to a data tape block id! (see later)

Oh!: and ULTRIUM LTO does not support tape partitioning

Let me demonstrate. (sorry forthe long reply).. his helps add value to the idea of a tape archive.

Lets say you had some miracle software like SGL's flashnet™ or FPDI's Divarchive™ and lets further assue they understood R3D codec (just indulge me ).

• YOu have 1 hour of R3D at 288Mbs ..(that's about 129-130GB give or take and just for example)
• you have it archived on your favourite brand of Ultrium4 LTO tape drive and you know it is the 14th file on the tape volume and
• you know also that the space(size) on the tape occupied by the previous 13 files is over 50% of the LTO4 tape volume and the file sits around 400M into the tape volume[SOT+400m {SOT=start_of_tape}]. (the avg length of an LTO tape media is around 800M when I looked last).{you non metric guys can work out your inches and feets, yards, chains and furlongs and rods]:sarcasm:
• and the tape LOCATE speed is a fast 10M/sec and
• your tape drive reads at 2m/sec (irrelevant here) and
• your tape drive reads data at 100MB/sec uncompressed in a perfect world
• you need to pull certain shots from the RED footage from this object so you will need to read it in using say a simpel utility like tar.
• YOu would probably do the following and READ the whole object of 130GB to you suite (mac/pc/linux) from the tape drive and perform you work.
• the time to do this is simple to calculate as:
a) find and mount the tape (zero seconds because you already have it in the drive at the LOAD point)
(b) LOAD (5secs)
(c) SPACE [Count,14] (goto the 14th TM/FM on the volume). secs = (800/2)/10 = 400/10 = 40seconds
(d) READ DATA until EOF (TM+TM) = (130*1000)/100 = 1300 seconds

total time to read 130GB .R3D = 0+5+40+1300 = 1345 secs

HOwever if you KNEW where the shots you wanted to recall from this archive were sets of IN:OUT TCR then you may also use a FAST SEARCH/SEEK/LOCATE block command to a region near the start of the shots on that tape using the tape drives FAST LOCATE command at usually 8-10M/sec!

this implies some TRANSFORM of the TCR's to the physical tape block location using the information around the codec. THis is not simple to do.

This process if call PARTIAL OBJECT RECALL and is used to extract portion of large bit rate material from log from contant such as SPORTS of material set aside for archive and post. Note that useful for long GOP MPEG2 8Mbs stuff...

An example using the for the 1hr of .R3d would mean that we could SIGNIFICANTLY reduce the transfer time of the actual pieces we need from the 130GB .R3D object by mearly reading in those portions we wanted. The reduced time is the difference between the transferring the WHOLE 130GB clip instead of portions of if (read at 2M/sec). The essence of the time save is from SKIPPING to the parts we need at 8-10M/sec.

The software that does this using codecs is SGL FLASHNET, DIVAchive and type of 'plug in to some brand-x HSM's) from Silex.

Also, in terms of metadata being stored by proprietary software - what sort of data are we talking about, exactly? Aside from the usual unix info (date/permissions/owner/etc.), and maybe the date & other stuff you can write on the tape with a pen, I'm unsure what sort of data would be necessary... At least in the context of backing up R3D's & tiff's en masse, then nuking them off your drives. Possibly more complex in a general IT environment, where you may be backing up a rapidly changing file/directory structure

The metadata to which I refer is that stuff useful for the workflow:
• content information
• shot list
• manifest
• codec
• times
• usage rights
• author
• block id of the start of the file
• block id of the end of the file (last block) (associated with that volume serial )
• volume sequence number,
• previous tape volume serial number (multi volumed file)
• loads more stuff.
• you get the idea.

There's loads of it as you might expect you would use in any content management workflow. So you eed to have some method of associtaing any content metadata with the archive.

This is very simple to do if you either tape care else invest in a COTS product to do this (i.e. something low end like CATDV™ will do the job). The COTS .app maintains its own DB with all this info.

Regadless of the format missing any metadata makes it dreadfully hard to reinstate any stuff form the basic data tape.

This is the same bad circumstance with any schema including disk storage systems that lost thier cataogue info or volume tapbel of contents/file table... and sectors have to be scafenged to try and get the data back.

Or more crudely a video or film that has absolutely no information with it.

hth
w

 

[warwickt]

Lto4 Ultrium HP and Quantum and LSI and ATTO SAS HBAs on MACOSX

Lto4 Ultrium HP and Quantum and LSI and ATTO SAS HBAs on MACOSX

HI GUys I have some wonderful success on arcihve and recall to LTO4 Ultrium data tape drives on MAC OSX (MAC PRO) with my HP & Quantum Ultrium LTO4 tape drives and ATTOTECH EXPRESSSAS (R380) and LSI LOGIC LSI SAS3442E-R PCI-E SAS HBAs using Tolisgroup BRU software

To avoid duplication - please refer to my recent detailed post at

http://www.reduser.net/forum/showthread.php?t=21284&page=6
w