Storage and Archiving Digital Workflow

Techno-babble disclaimer: This article contains quite a bit of jargon about RAID enclosures. For those not familiar with the various forms of RAID and how a RAID system works, I highly recommend reading the excellent Wikipedia page that discusses RAID in detail. 

Earlier this year, I taught a one day storage and archiving seminar with my good friend Tony Bonanno for the Santa Fe Workshops. I have also gotten a few calls from several pro photographers asking for recommendations for backing up large image collections and for backing up video content. This article appeared in my Winter 2015 Newsletter but I thought I would republish it here on the blog so it is more visible for those looking for a solution to back up their images. For those looking for a simple back up strategy, the diagram below will look ridiculously complex because I have a huge image collection to back up. I have over 72 TB of hard drives. For those with only 4 or 5 TB of images or less, I would suggest getting three hard drives that can accommodate all your images and using SuperDuper or Carbon Copy Cloner to back up your images to all three drives. The critical part of that back up strategy would be to make sure one of those hard drives is offsite. At this point, I am updating my digital workflow e-book and I will have a much more in-depth chapter in that e-book that covers a wide range of topics concerning backing up and archiving your images, including specific recommendations.

Caption: Backing up huge quantities of images can be daunting. For most amateur photographers, the fewer number of images and hard drives can greatly simplify this process, but for the pro the terabytes add up quickly. Above you can see my entire storage and archive workflow. I use the G-Tech G-Dock and various portable G-Drive ev HDs to backup my images in the field and this system allows for a very quick and simple ingest workflow once I am back in the office. My office storage consists of three OWC ThunderBay 4 enclosures: one with a RAID 0 configuration and two with RAID 5 configurations.  All of my images are also archived onto individual 3 TB hard drives and stored in a safety deposit box at my bank. 

Last fall, I upgraded my entire computer setup and with that upgrade I was also forced to replace the hard drives and RAID enclosures that I use to store and backup my images and motion content. I realize a review of an external hard drive enclosure is not as exciting as a review of a new lens or camera, but for the pro photographer, having a fast, robust and well-thought out backup and storage system is paramount. Hence, when I upgraded my computer system, I set out to find the fastest and most reliable storage option available that could deal with huge image files and the 6K video footage produced by the RED Epic, which I use for motion projects..

My old system was an Apple Mac Pro tower with multiple internal hard drives and a series of external RAID 5 enclosures, which were all connected to my tower via Firewire 800. My old Mac Pro didn’t have the option to connect Thunderbolt devices. With the new computer, I chose to update all of my external RAID enclosures and all of my external hard drives with Thunderbolt or Thunderbolt 2 devices. My new system is a top-end, maxed out 15-inch Retina Apple Macbook Pro laptop, which is wicked fast. I have the Macbook Pro mounted in a Henge Dock Vertical Docking Station, which helps to keep all the connecting cables coming out of it organized and out of sight. At some point, I may bring in an Apple Mac Pro for the office if I need more computing power, but for now the Macbook Pro is working quite well for my needs.

After a ton of research, including setting up a spread sheet to calculate, quantify and compare the Input/Output (I/O) speeds of various RAID enclosures vs. the price/terabyte, I purchased three of the Other World Computing (OWC) ThunderBay 4 RAID enclosures. In my research, I found quite a few respectable and very capable RAID enclosures that would work for my needs. CalDigit, G-Technology, and Areca all make fantastic products that will work for storing and backing up images in a robust manner, but it was the OWC ThunderBay 4 that was the most economical, and even more important, allowed for a much more flexible storage solution. The OWC ThunderBay 4 connects with a Thunderbolt 2 connection, which means it is a bit faster than the normal Thunderbolt options and also allows for a 4K monitor to be added to the daisy chain of hard drive enclosures. Going with a Thunderbolt 2 connection also means that the I/O speeds of the second, third, fourth and fifth hard drives connected in a daisy chain setup won’t suffer too much in terms of transfer speeds like they would using the older version of Thunderbolt. Thunderbolt allows up to six devices to be daisy-chained to one port.

The ThunderBay 4 can hold up to four 6 TB hard drives whereas some other Thunderbolt 2 RAID enclosures would only work with 5 TB hard drives. On top of these features, the ThunderBay 4 allows the hard drives to be set up as a JBOD (Just A Bunch of Drives) system or in any type of RAID configuration you could ever need. In fact, you can set up two of the drives as a RAID 1 or JBOD and the other two as a RAID 0 all within the same enclosure. Practically any configuration you can dream up is achievable with the OWC ThunderBay 4. This is possible because the ThunderBay 4 uses SoftRAID software to build and maintain the RAID configuration. In the past, software RAID systems have been less than reliable and a fair bit slower than hardware RAID solutions. SoftRAID has changed all of that so that the ThunderBay 4 is as fast, if not faster, than many hardware RAID enclosures (depending on the drives you put in the enclosure) and is just as reliable, if not more so, than their hardware RAID counterparts.

The SoftRAID software is very easy to use and also comes with some very advanced features, including the ability to monitor all of the hard drives in any and all RAID configurations, a much faster rebuild time than a normal hardware RAID if a hard drive fails and the software also has the ability to predict when a hard drive is about to fail so you can replace it before you get into too much trouble. One of the issues with using a program like SoftRAID is that you have to be careful when updating your computer operating system (like MAC OSX). Luckily, the fine folks at SoftRAID are super attentive to these issues and you can check their website for the latest info on when it is safe to update your OS, and if there are any issues, how to deal with them.

Aside from the stellar RAID configurations and the incredible versatility of the ThunderBay 4, it is also a very well built enclosure with all metal construction—save for the rubberized feet. It also includes an excellent cooling fan that keeps the HDs cool and is very quiet for how effective it is. I have these three enclosures sitting on my desktop just next to my monitor and while they are audible, they are a quiet whisper compared to my older RAID 5 enclosures. And since most hard drives fail due to overheating, I am pretty impressed with how cold these enclosures keep the hard drives. The ThunderBay 4 enclosures also have vibration dampening built into the enclosures, which helps to keep the hard drives safe from vibrations that might affect the needle reading the hard disks inside the hard drives.

In terms of speed, my 12 TB RAID 0 enclosure achieves read/write speeds of around 600 to 750 MB/sec, which is blazing fast for a non-SSD hard drive set up. My 24 TB RAID 5 enclosure clocks in with read/write speeds in the neighborhood of 490 to 580 MB/sec. For comparison, a normal hard disk drive on it’s own has read/write speeds anywhere from 80 to 180 MB/sec depending on the hard drive and the connection. With the blistering read/write speeds of my ThunderBay 4, I can easily work with high bit-rate, 1080p video in real time. These units also make data transfers a very quick and painless process.

As I said earlier in this review, there are many good options out there for storage, what you choose comes down to your needs, what you can afford and the options you need. Hence, this isn’t one of those reviews saying this is better than everything else out there. This is a review discussing how the ThunderBay 4 is more flexible than most other options and just as robust and secure. All of the other options I found on the market for RAID enclosures offered either RAID 0 or RAID 5, and a few offered JBOD as a separate option, but none of them were flexible enough to offer JBOD and RAID in the same enclosure. Besides flexibility, the hardware RAID options were anywhere from 50% to 110% more expensive than the OWC ThunderBay 4. In the end, it was a no-brainer to go for the ThunderBay 4. For more information on the OWC ThunderBay 4 visit the Other World Computing website.