How long does it take to empty a 64gb red ssd card?

[Brian Iannone]

I just did two tests on a CompactFlash card. The card is a SanDisk Extreme Pro 16GB rated at 90 MB/s. The reader I used is a Lexar USB 3.0 CF/SD Card Reader. The card reader is connected directly to a USB 3.0 port on an ASUS Maximus IV Extreme-Z logic board.

For the first test, I used ATTO Disk Benchmark to run a speed test on the card. Here are the results:

It shows maximum write speeds of 58 MB/s and reads of 101 MB/s.

The second test I did was a true real-world scenario "test." A full card (which was actually only 14.8GB due to the formatted capacity) of photos being transferred to the Desktop. It took 2 minutes, 47 seconds. That's a rate of 88 MB/s. Here's a screenshot and video from the data transfer:

So, those CF cards are pretty fast.

 

[Jeff Kilgroe]

I see. When you tested the card and got read speeds of 53 MB/s sustained and 95 MB/s burst, what speed test utility did you use? Have you ever tried transferring data from a full card over to your computer and clicking on the 'More info' dropdown check in the Windows file transfer dialog box? It shows a speed estimate along with remaining time, number of files left, etc. What sort of speeds does that typically show you? Also, what interface are you using between the CF card reader and your computer?

The reason I ask about CompactFlash card speeds is that I just (quite randomly, actually) performed a speed test on one of those "90 MB/s" SanDisk Extreme Pro cards about a month ago on a Lexar USB 3.0 CF card reader. The results were quite interesting. I'll run another test tonight and post some screenshots.

I use various benchmarking tools, I typically run a full compliment of tests when I'm trying to put speed numbers to something like this. Also test on 2 or 3 systems. The ATTO and AJA tests, HD Tach, etc.. Of course the regular speed info right there on the "more info" tab is always handy for straight copies. All my CF testing I did in the past was connected via eSATA, using a few different readers. I would also use FW800 as an additional reference point.

Your results with the SanDisk card are interesting. I never had one run that smooth, but it's also been well over a year since I did my testing and it's possible that newer productions of these cards have improved firmware and tuning. The ones I bought for testing were hot off the original production run of cards when first released. Your reported write speed isn't a whole lot faster than what I was experiencing, but if it's sustaining that, it must be quite a bit smoother. The read results you're getting are definitely a surprise. All my testing I did in the past was for a client who was trying to qualify certain flash media for a specific purpose. We came up with a few cards that fit their need and they actually standardized on those Transcend 32GB cards I mentioned. Since those have been discontinued, they have since moved to the Transcend 400X 64GB cards, which provide similar, but faster, performance and have been running very smooth. Or about 50MB/s write, 70MB/s read via SATA. The cards are claimed to be 60MB/s write, 90MB/s read, and they can burst over those speeds, but can't sustain them.

I'm definitely curious what sustained rates you're getting with the Sandisk card across the whole volume. Especially in 32bit UDMA mode with 128K~512K block sizes. Also wondering if the USB3 host on your system provides any additional buffering (most do).

 

[joshua csehak]

It took me 20 minutes tonight to empty my 64GB SSD over eSATA. It was totally full from shooting for less than an hour at 4k; RC 6.1.

Thanks Jon! That seems a bit slow to me. Are you copying to a laptop HD?

 

[joshua csehak]

I'm still not sure what configuration you were talking about that was giving you 27 minutes to copy 64GB over FW800, there is definitely a bottleneck or some sort of speed obstruction there. Or are you talking about the SanDisk card being read?

I'm talking about both the SanDisk and the OWC external drive, both connected via FW800. They both take 27 minutes to copy 64GB of data.

But it's time to be even more scientific here. Eliminate all variables. This time, I've got a LaCie Quadra Barracuda LP drive, which has both eSata and FW800 connectors. My results to copy 64GB of footage from one of my internal WD 2TB Blacks to it:

via eSata: 15 minutes, 40 seconds
via FW800: 36 minutes, 50 seconds

This leads to one of two conclusions:

1) The FW bus or controller on my Mac Pro (and/or drive(s)) is slow
2) FW is simply much slower than sata

But wait. WTF?! That's a full 10 minutes longer than the OWC. Let's check it again:

Same footage, to the OWC, via FW800: 26 minutes, 15 seconds

Yup, same time as before. If I check them in AJA test, they both clock in at 38MB/s. I can only arrive at two conclusions here:

1) FW800 may in theory be as fast as sata, but in practice, it all comes down to the quality of the controller on your HD and computer.
2) Don't trust test utilities. Manually copying the files over is the only way to truly test a disk's speed.

What do you guys think?

As you fill up hard drives, they don't run slower because they are full. What happens is you tend to delete files and then add new ones regularly over time as the drives fill up. This leads to fragmentation of your data and your read/write heads have to jump around and seek to lots of different locations to read or write data.

Agreed about fragmentation, but I think how full your drive is matters too. Check this out: http://macperformanceguide.com/Storage-WhyYouNeedMoreThanYouNeed.html

As Brian asked above, how come you don't have your 2TB HDD's in a RAID-0 stripe configuration? .

Data security, as you mentioned. I have backups, but they're not automated, so I wanted to be on the safe(r) side. Also, it seems like the performance gains wouldn't be that much. Do you guys see a big difference in day-to-day work when working with footage on your RAIDed drives?

I'm still a little fuzzy on some of your numbers, like the 27 minutes to copy the 64GB, as that is quite slow -- half as fast as it should be over a FW800 connection. One of those WD HDD's can saturate a FW800 connection, even full and fragmented as your's seem to be.

Can you guys post times you get over FW800, and what computers and devices you're using? I'm beginning to think now, that not all FW800 is built the same.

 

[Ian B. Johnson]

I finally got around to testing copy speeds using my Mac Pro. I have a Sonnet 2-port eSATA card, with the RED Station in one and a TowerSTOR RAID 0 (with 2x1TB drives) in the other. Copying a disk image from the RED SSD to the RAID, I got a sustained transfer of 95-105MB/s. Strange thing was after getting almost exactly halfway through it dropped to 75MB/s and stayed there for the remainder. Wonder what that's about...?

 

[Cail Young]

Apologies.

Earlier data was a single 64GB card image being moved from one TB drive to another.

R3D Data Manager doing two copies with checksums of a completely full 64GB SSD from eSATA to two Thunderbolt Lacie drives - 36 minutes.

 

[Jeff Kilgroe]

I'm talking about both the SanDisk and the OWC external drive, both connected via FW800. They both take 27 minutes to copy 64GB of data.

But it's time to be even more scientific here. Eliminate all variables. This time, I've got a LaCie Quadra Barracuda LP drive, which has both eSata and FW800 connectors. My results to copy 64GB of footage from one of my internal WD 2TB Blacks to it:

What specific model of LaCie? Is it 2.5" or 3.5"? Single drive, I presume? Either way, the FW800 number seems screwed up.

1) The FW bus or controller on my Mac Pro (and/or drive(s)) is slow
2) FW is simply much slower than sata

More than likely the FW800 interface on the drive is slow. Which is common. Not all interfaces are created equal. As for speed, keep in mind that typical eSATA is 3Gbps, which is 3.75X faster than FW800 -- 3Gps is 3000Mbps, vs 800Mbps of FW800.

1) FW800 may in theory be as fast as sata, but in practice, it all comes down to the quality of the controller on your HD and computer.
2) Don't trust test utilities. Manually copying the files over is the only way to truly test a disk's speed.

What do you guys think?

FW800 is nowhere near as fast as SATA, not in theory, not in reality. But the numbers you showed above don't make sense. The ~70MB/s you were showing via eSATA is still well within the abilities of FW800, so it should not be that much slower on that drive. So this points to a problem or a poorly-designed interface somewhere. The Mac Pro FW800 ports run fast with good hardware, so I would look first at the drive and/or cables.

Agreed about fragmentation, but I think how full your drive is matters too. Check this out: http://macperformanceguide.com/Storage-WhyYouNeedMoreThanYouNeed.html

They are right when looking at it within the limited scope that they are. But most hard drives use multiple platters and proper RAID controllers offset data locations to account for the differences in recovery speed and seek timings to give even performance across an entire RAID volume. A simplified way of relating that would be if you had two drives in a RAID-0 stripe, one drive would start writing its data at the outside of the platter working in, the other would do the opposite, starting from the inside working out. Many hard drives do similar things internally, offsetting data locations between platters to minimize this effect.

The other thing to consider is that a drive doesn't just become slower because there's data on it. just some data written to certain locations on a platter will take longer to retrieve than data written to other locations. Most current performance hard drives on the market that interleave data between platters, most of the latest drives with 2 or more platters do, show very even data read and write speeds across the entire data volume. Filling them up does not slow them down. However, filling up drives can have an impact on various OS filesystems -- namely those that catalog or journal themselves like OSX/HPFS or NTFS. More files means bigger file catalogs to manage when the OS needs to cache and reference what it has access too for search and other purposes.

Data security, as you mentioned. I have backups, but they're not automated, so I wanted to be on the safe(r) side. Also, it seems like the performance gains wouldn't be that much. Do you guys see a big difference in day-to-day work when working with footage on your RAIDed drives?

Data security is a valid concern. Even with good backups, you do minimize risk, or even just the down time to restore data, if you're not depending on one drive holding up the data on others as well, as it does with many RAID configurations. On the other hand, the speed increase from building a good RAID can be substantial. Generally speaking, for most any medium to large size files, you cut your transfer times in half by striping two drives, or cut it down to 33% by striping 3, and so on. A RAID-0 consisting of 4 fast 1TB or 2TB 7200rpm drives is fast enough to read and write 2K DPX files in real-time with room to spare or fast enough to not be a bottleneck when offloading RED SSD's via eSATA.

Can you guys post times you get over FW800, and what computers and devices you're using? I'm beginning to think now, that not all FW800 is built the same.

I'm doing offloads on a shoot today. DIT cart has an '09 Mac Pro and I grabbed some times for you. Using a 128GB RED SSD that has 82.2GB of data on it. Mac Pro has 4 x 2TB Hitachi UltraStars in a RAID-0 as well as G-Tech G-Speed ES connected to eSATA via an ATTO H680. Dual RED Station SSD readers on the ATTO card as well.

82.2GB offload to internal RAID - 5 minutes, 50 seconds - approximately 240MB/s. (via iPhone stop watch)
82.2GB offload to G-Tech RAID - 6 minutes 22 seconds - approximately 220MB/s.

Tried a couple via FW800 for comparison...
82.2GB offload to internal RAID - 18 minutes, 58 seconds - approximately 74MB/s.

82.2GB read from internal RAID out to Seagate 1.5TB 7200rpm 32MB cache HDD in Rosewill eSATA+FW800 enclosure, connected to front FW800 port - 20 minutes, 34 seconds - approximately 69MB/s.

These were all straight drag and drop times in finder.

 

[Brian Iannone]

82.2GB offload to internal RAID - 5 minutes, 50 seconds - approximately 240MB/s. (via iPhone stop watch)
82.2GB offload to G-Tech RAID - 6 minutes 22 seconds - approximately 220MB/s.

Tried a couple via FW800 for comparison...
82.2GB offload to internal RAID - 18 minutes, 58 seconds - approximately 74MB/s.

82.2GB read from internal RAID out to Seagate 1.5TB 7200rpm 32MB cache HDD in Rosewill eSATA+FW800 enclosure, connected to front FW800 port - 20 minutes, 34 seconds - approximately 69MB/s

How did you calculate the data rate from the amount of time it took for the transfer? Just curious. (I know, I've been very curious lately...)

 

[Jeff Kilgroe]

How did you calculate the data rate from the amount of time it took for the transfer? Just curious. (I know, I've been very curious lately...)

Er...

82.2GB in 5 minutes, 50 seconds would be 240MB/s based on:

82.2GB * 1024 = 84172.8MB / 350seconds = 240.49MB/s.



Replace 1024 with 1000, if you're one of those freaky people who like to use newfangled non-binary "megabytes" and "gigabytes". But make sure you use the proper value to coincide with what your system is set to report.

 

[Brian Iannone]

Er...

82.2GB in 5 minutes, 50 seconds would be 240MB/s based on:

82.2GB * 1024 = 84172.8MB / 350seconds = 240.49MB/s.



Replace 1024 with 1000, if you're one of those freaky people who like to use newfangled non-binary "megabytes" and "gigabytes". But make sure you use the proper value to coincide with what your system is set to report.

No, no, no... I wasn't asking because I don't believe you. I could tell from your "Er..." that you thought my question was strange. I really honestly, truthfully was just wondering. I've been recently using Wolfram|Alpha frequently to calculate data rates (example from your numbers) and just wanted to ask what numbers specially you use when calculating. I see you use 1024. I do too... But some other people don't. So when talking to someone about data rates, I like to know what numbers they base their values on.

Sorry if it sounded like I was questioning your numbers. I wasn't! =)

 

[Jeff Kilgroe]

Haha. Well, at least you weren't asking because you didn't know how.

The Wolfram example is just using the non-binary forms of megabytes and gigabytes by default. Which for some reason is the current hipster thing to do if you're not an IT guy or programmer, but want to bend over and give in to nearly 30 years of bass-ackward marketing from storage media companies. Of course, they let you use "mebibytes" and "gibibytes" if you want for the binary notation, then their numbers match mine. ...Two terms which didn't even exist until decades after the binary "kilobyte", "megabyte", etc.. were conceived and well-established. The movement seems to have really caught on over the past 2 or 3 years since the academic (or should I say overly-pedantic) community at large seems to have taken a liking to it. Because, after all, the prefix "mega" means a million, "giga" a billion, etc.. So why not simplify the represented quantities, right? Apple, Microsoft and Google have all incorporated it (and most Linux flavors as well) as an option, due to popular demand.

There's currently a similar movement to re-establish the liquid "quart" measurement to be equal to 1 liter instead of the 0.946 liters that it actually is, making a US Gallon the equivalent to 4 metric liters. On some levels it makes sense, especially at the consumer level and would have almost no real-world impact on household measurements in terms of food preparation or other daily tasks. On the other hand, it would completely fuck up the big picture on a global scale for industrial standards. Ironically, I'm surprised the oil and gas industry isn't pushing this, or maybe they are on a very low-key level, at the consumer level. Than again, on the production level it would be a mess. I don't recall what the proposed names are for what the current quart, gallon, cup, drum, tablespoon, etc.. would become, but just think about that one....

The biggest problem with the whole binary vs. base-10 representation or "mebibyte" vs. "megabyte" issue is that most computer "users" don't have the slightest inkling of a clue as to when to use either or why. A primary example of why the whole movement is just plain stupid. Or a better way to put it is, "if it ain't broke, don't fix it.".

As far as I'm concerned, my "megabytes" should be equal to 2^20 (1048576) bytes. Our data is binary information, living in a binary ecosystem. So let's just count in binary and not try to obfuscate the fuck out of it.

OK, political rant over...


EDIT> In case anyone didn't put 2 and 2 together on this one, my systems are all reporting in binary megabytes or gigabytes. Now known as "mebibytes" and "gibibytes". If you switch over to the base-10 new style representation, and test, you will (or should) get bigger numbers.

As non-binary "megabytes", my above numbers look like this:

82.2GiB = 88261.58"megabytes", so divide that by the 350 seconds to get 252.17MB/s.

 

[Brian Iannone]

Jeff, your post was a fascinating read. I am quite surprised however that I received a five paragraph reply from a one sentence question. =)

To be honest, I tend to give long answers to short questions also.

 

[Jeff Kilgroe]

Jeff, your post was a fascinating read. I am quite surprised however that I received a five paragraph reply from a one sentence question. =)

To be honest, I tend to give long answers to short questions also.

I felt like ranting. haha...

 

[Brian Iannone]

I felt like ranting. haha...

I could tell when you started writing about quarts and liters.

LOL

 

[Chris McKechnie]

What about...

What about...

What about using the http://www.bhphotovideo.com/c/search?Ntt=Sonnett+Echo&N=0&InitialSearch=yes

 

[Brian Iannone]

What about using the http://www.bhphotovideo.com/c/search?Ntt=Sonnett+Echo&N=0&InitialSearch=yes

Using it do do what exactly?

 

[Jeff Kilgroe]

Lots of people keep asking me about the Sonnet ExpressCard adapter. I swallowed the bullet and ordered one. But there seems to be some common misconception that it will make ExpressCard adapters (and anything they do) go faster because it connects via Thunderbolt. Not true, people. All this does is add an ExpressCard-34 slot to a computer if you have a Thunderbolt port to connect it to. Reason I'm buying one is so I can also use EC34 adapters with my 15" MBP in addition to the 17", or add a second EC34 slot to my 17".

ExpressCard-34 in v1.x mode, which is what this adapter supports, is roughly 250MB/s (a bit less, actually). So it's great for a single eSATA connection. Dual connections are fine too if your devices won't mind sharing that bandwidth. The lack of a pass-thru kinda pisses me off though... So if you connect to a MBP with only one Thunderbolt port, you can't connect any other Thunderbolt devices or an external display when this adapter is connected. If not for the fact it's the only one available right now (and in the near future), I would label it as a FAIL. Why? Because they didn't include a pass-thru port. It does not support ExpressCard 2.0 and they went through all the trouble to make a bulky adapter and only included one slot. So I'm kinda scratching my head, trying to understand WTF they were thinking when they designed this thing.

 

[joe hedge]

So I'm kinda scratching my head, trying to understand WTF they were thinking when they designed this thing.

Perhaps one day Sony will update the SxS card drivers to T-Bolt, and we'll have $150 T-Bolt SxS card readers

 

[Jeff Kilgroe]

Perhaps one day Sony will update the SxS card drivers to T-Bolt, and we'll have $150 T-Bolt SxS card readers

Won't really work that way. SxS are ExpressCard and limited by the ExpressCard interface. Which happens to be PCIe, same as Thunderbolt, but we're capped at the speed of the EC interface. Essentially, it's PCIe X1 (single lane). And SxS is yet another key point on which to question Sonnet's thinking here... The Sonnet reader should work as a standard ExpressCard interface, meaning they will support SxS media. So once again, WTF is Sonnet thinking??? Why not make a dual-slot model with pass-thru and it would be one badass SxS reader. Support EC2.0 and it could be a dual SxS Pro reader that could unload two SxS Pro cards simultaneously with bandwidth to spare. ...Nice going Sonnet, you missed the boat by at least 10 miles.

Kinda makes me think about canceling my order, but no one else has announced a similar (let alone, better) product.

 

[Les Dittert]

Note sure if this was mentioned, but we need what still guys call a 'digital wallet' for the Red ssd's. A small paper back book sized thing that you either plug the ssd's into directly ( if it's just reconnectored sata w power ) or plug the esata station into. It then vacuums out the ssd contents to a 512Gig cheap ssd, at full speed, say 200 meg/sec. With scarlet that's 4X record speed. Throw the wallet in your backpack and shoot some more. No laptop.
I already talked to one vendor rep, he seemed very interested.
What do you guys think?