RAID configurations

[SusanV]

Hi,

Anyone here up to speed on RAID configurations for hard drives, and how it
relates to image editing?

I was ordering a new PC today, and got 2 hard drives. Asked that they be
configured to RAID 0, which is supposed to use both drives in such a way as
to increase processing speed. ( I have a third external drive that I back-up
to) The sales guy checked with the tech guy who said that unless I'm into
heavy-duty gaming, RAID 0 was a waste. When I said I was a photographer
and artist and will be running CS3, the techie said, "oh, image editing? Nah,
you don't need that much speed". hmmm... does he know what he's talking
about?

I've been making my little 8 x 10 digital positives from scanned medium
format... Now that I've got the photogravure thing kind of figured out, I"m
going to be using the 4x5 view camera, scanning those, and eventually
outputting 16 x 20. Those file sizes are going to be pretty big. I thought I
was going to need as much speed as possible.


Anyone know about this stuff?

Susan
ps... i got the puter anyway, without the raid config. I'm just curious about
this and would like to learn more about computer power as it relates to this 2D
editing we do.

 

[jd callow]

I use raid 0 for my apps drive and Raid 1+0 for my data

I'd say he is wrong. If you back-up often than you would benefit by Raid 0.

Raid 0 is Striping and it means the data is split across multiple drives. theoretically increasing drive access as you add drives. This doesn't work with all drives so be carefull. Raid 0 is fast and high risk for those who don't back up.

Raid 1 is Mirroring. This is where 2 drives act and look like 1 drive, because they are duplicates of each other: each keeps a back-up of everything written to the raid. this means if you lose a drive you don't lose any data. This is no slower than a regular single drive, but far more secure.

Raid 1+0 uses both Stripping and Mirroring it is the best of both worlds.

I receantly had a drive fall out of my raid 1+0 configuration. I was warned by the drive manager, rebuilt the raid and never lost a step -- it was lovely!

There is also Raid 5 (I think) which is a higher level version of 1+0, but requires a controller smart enough to manage it.

In the pecking order of bang for the buck speed wise I go by the following
1) memory Size
2) Drive access Speed
3) CPU Cache size
4) Bus Speed

 

[donbga]

Hi,

Anyone here up to speed on RAID configurations for hard drives, and how it
relates to image editing?

I was ordering a new PC today, and got 2 hard drives. Asked that they be
configured to RAID 0, which is supposed to use both drives in such a way as
to increase processing speed. ( I have a third external drive that I back-up
to) The sales guy checked with the tech guy who said that unless I'm into
heavy-duty gaming, RAID 0 was a waste. When I said I was a photographer
and artist and will be running CS3, the techie said, "oh, image editing? Nah,
you don't need that much speed". hmmm... does he know what he's talking
about?

I've been making my little 8 x 10 digital positives from scanned medium
format... Now that I've got the photogravure thing kind of figured out, I"m
going to be using the 4x5 view camera, scanning those, and eventually
outputting 16 x 20. Those file sizes are going to be pretty big. I thought I
was going to need as much speed as possible.


Anyone know about this stuff?

Susan
ps... i got the puter anyway, without the raid config. I'm just curious about
this and would like to learn more about computer power as it relates to this 2D
editing we do.

Susan,

AFAIAC, not getting the RAID config was a good move if you are talking about the Nvidia on board controllers and software. I've had nothing but problems with two completely different mobos using the RAID controllers and software. The internet is littered with messages from people describing similar experiences.

Get 3 external hard drives, and do son, father, grand father backups and off site storage if possible. Backup image data separately on DVD. You might also consider using A DLT tape drive which are very reliable.

My 2 cents,

Don

 

[MAGNAchrom]

I would argue that the single slowest bit of architecture in desktop computers is the hard disk setup. You can go crazy with fine-tuning the system with SATA, multiple controllers, striping, etc. But perhaps the biggest single bit of advise is this: keep your hard disk less than 67% full and never let it get above that and you will have a happy system that performs better than most other "tweaked" systems that approach their capacity. The reason is simple: the fastest tracks are on the outside of the hard disk platter and once you get beyond 50% full you are reading/writing to the inside tracks which can be as much as 50% slower.

In short, buy yourself plenty of hard disk space first and worry about RAM second (which is somewhat contrary to conventional wisdom). I like the suggestion of primary (fast), secondary, (medium speed), and tertiary (network backup drives which are technically quite slow) as a reasonable system, but bear in mind that sooner or later you will fill it all up anyway, so designing it up front and sticking to deliberate backup strategies will serve you well for years to come.

Speaking of which, offline harddisks (e.g. removable 250Gb drives) are probably cheaper per megabyte and more feasible than DVD backup for most serious photographers.

Cheers,

 

[jd callow]

You should have at least two internal drives. Having one internal disk for both OS pagefile and PS scratch disk is not a good way to go, as the OS and PS will fight for access.

If all you have on your system is two hard drives than a raid won't be helpful, because the OS will see the raid as one drive and you'll have the problem I described above.

 

[clay]

I have just finished setting up a new Mac Pro. I already had an external RAID system that I just transferred to the new system. That is a nice way to go, since it makes the migration process painless. This external setup has 5 disks, and I have it arranged with one striped RAID 0 array of 3 disks, and one of 2 disks. I backup the big array to the small array every night with a backup program that essentially clones the volume. When the 3 disk array starts approaching capacity, I will just get another 5 disk array, and make it a 5 disk RAID array, transfer the data to the new box, and then reconfigure the old box to a 5 disk array. Bottom line is that it is fast. You can really notice a difference when working on a large file with a lot of layers.

 

[SusanV]

Hi Michael, ... the box I've ordered is huge, and has room for 4 hard drives.
As ordered it will have 2 250GB serial ATA 3Gb/s, 7200rpm drives. As money
allows I will add drives. I also already have an external drive that I have set
up to back-up image files every evening. I agree that the price of external,
and/or removable drives makes them seem like a good value for using as
backup. I will keep in mind what you said about drive capacity and speed.
Didn't know that!

Jd... regarding bang for buck, I got 2GB dual channel ddr2 @ 667MHz, and
there is room to double that. Intel core 2 duo E6420 with 4MB L2 cache,
2.13 GHz, and a front side bus speed of 1066. Good point about the 2 hard
drives and raid, and how that wouldn't work well for a ps scratch disk... that
never occurred to me. In fact... that single point might be the deal-breaker
as far as using a raid 0 configuration when the primary task of the machine is
photoshop, as mine will be. I was thinking that later on when I add other
drives I might set them up as raid 0, but since PS always "looks to" it's
scratch disk, THAT is where speed would be a good thing... as in a 10000 rpm
drive. (?) I'm kinda confused about this now... the scratch disk and how it
relates to raid config.

See this is the thing when dealing with tech people about my (our)
computers... I have a hard time finding people who understand the photo
editing process that we do. When most of them hear "photo-editing" they're
thinking of fixing red-eye in family snaps. They don't understand the "need
for speed". haha... Right now I'm using an OLD pentium 4, and when I apply
unsharp mask, I go play fetch with the dog while it's working at it. Man o
man... the dog is gonna hate this new computer

Clay, it took me a couple of reads to visualize your system, but I got it, and
wow that sounds like a great way to do it.

ok gotta go clear my head of all this techie stuff for a while and make some
art to pay for my new machine

thanks for the education... keep it coming!

Susan

 

[clay]

One thing I failed to mention on this new system was that it had 3 extra internal SATA drive slots. I shopped around and found a deal on three 7200RPM 80GB drives (around $65 per drive) and installed them into the empty bays. I configured them as a RAID 0 striped array (it looks like one large virtual disk). I keep this drive basically empty and use it as the photoshop scratch disk. Because of the striping and the fact that it is hooked right into the system bus, my photoshop speed has improved radically. Just for grins, I loaded a 600 MB image file from a large format drum scan, added five layers and then did a few editing things and USM etc.. This would have left my old system panting for breath, but the dedicated scratch disk array makes this feel like a 6megapixel jpeg in photoshop.

Hi Michael, ... the box I've ordered is huge, and has room for 4 hard drives.
As ordered it will have 2 250GB serial ATA 3Gb/s, 7200rpm drives. As money
allows I will add drives. I also already have an external drive that I have set
up to back-up image files every evening. I agree that the price of external,
and/or removable drives makes them seem like a good value for using as
backup. I will keep in mind what you said about drive capacity and speed.
Didn't know that!

Jd... regarding bang for buck, I got 2GB dual channel ddr2 @ 667MHz, and
there is room to double that. Intel core 2 duo E6420 with 4MB L2 cache,
2.13 GHz, and a front side bus speed of 1066. Good point about the 2 hard
drives and raid, and how that wouldn't work well for a ps scratch disk... that
never occurred to me. In fact... that single point might be the deal-breaker
as far as using a raid 0 configuration when the primary task of the machine is
photoshop, as mine will be. I was thinking that later on when I add other
drives I might set them up as raid 0, but since PS always "looks to" it's
scratch disk, THAT is where speed would be a good thing... as in a 10000 rpm
drive. (?) I'm kinda confused about this now... the scratch disk and how it
relates to raid config.

See this is the thing when dealing with tech people about my (our)
computers... I have a hard time finding people who understand the photo
editing process that we do. When most of them hear "photo-editing" they're
thinking of fixing red-eye in family snaps. They don't understand the "need
for speed". haha... Right now I'm using an OLD pentium 4, and when I apply
unsharp mask, I go play fetch with the dog while it's working at it. Man o
man... the dog is gonna hate this new computer

Clay, it took me a couple of reads to visualize your system, but I got it, and
wow that sounds like a great way to do it.

ok gotta go clear my head of all this techie stuff for a while and make some
art to pay for my new machine

thanks for the education... keep it coming!

Susan

 

[frugal]

My understanding is that RAID performance can really depend on the hardware configuration and how it's controlled.

RAID 0 gives you incredible write performance because it can send 1 block of data to the 1st drive and the controller doesn't have to wait for that to finish writing before it sends the next block to the next drive (although whether your computer can take advantage of this will depend on how the RAID works). Now my understanding is that read performance isn't going to be any better than a single drive because you're waiting for 2 drives to seek to the right spots and the data probably has to be sent back sequentially anyway.

For RAID 1 your write performance can suffer since you have to write to 2 disks before you move on. If you have each drive on a separate channel or controller though you should be able to do this simultaneously so you should have the same performance as a single drive. If they're on the same channel you may have to wait for the 1st disk to finish writing before the 2nd disk is written too which will make things really chug.

As mentioned, RAID 1+0 gives you the best of both worlds.

If you use a good backup strategy, then I'd be more inclined to go with RAID 0 just because of the economics (more bang for buck in terms of storage space).

Personally though? I'd say the best 2 drive setup for a personal computer is 1 drive for OS and a separate drive for data. That way your OS drive can completely fail and your data should be fine, you can even take that 2nd drive to another computer if you need to. And it makes reinstalling your OS a lot easier. I probably wouldn't do a RAID setup unless I had 3 drives (1 drive for OS, 2 drives in a RAID for data).

 

[jd callow]

To my knowledge everything Frugal said is true.

Personally though? I'd say the best 2 drive setup for a personal computer is 1 drive for OS and a separate drive for data. That way your OS drive can completely fail and your data should be fine, you can even take that 2nd drive to another computer if you need to. And it makes reinstalling your OS a lot easier. I probably wouldn't do a RAID setup unless I had 3 drives (1 drive for OS, 2 drives in a RAID for data).

This is my layout. I have 6 sata drives. 2 80 gig's set up as raid 0 for the OS and Apps and 4 500 gigs set-up as raid 1+0 for data. It is fast and reliable.

You need to buy drives that are built for raids. Many consumer level drives are not suitable and at the other end many of the 10k rpm drives tend to be shorter lived. I bought better quality 7k drives. They were30 to 50% cheaper than the 10k drives and about or slightly higher than the cost of the avg consumer 7k drives. One last consideration is case temp. Over heated equipment, including HDD's are an issue so good ventilation is important.

 

[clay]

The information I found on this indicated the opposite for the software RAID on Mac OS X. Here is a link:

Dead Link Removed

It seems to show that reading is considerably faster than writing. Whatever, there is a noticeable performance improvement.


Here is a link to the Mac OSX instructions for creating a RAID 10 set. I thought about this, but since I had the external hardware RAID setup, this seemed like making things a little too cute. But if you had four internal drives and that was it, this would definitely be a way to get the performance benefits of striping with the safety of mirroring.

Dead Link Removed

My understanding is that RAID performance can really depend on the hardware configuration and how it's controlled.

RAID 0 gives you incredible write performance because it can send 1 block of data to the 1st drive and the controller doesn't have to wait for that to finish writing before it sends the next block to the next drive (although whether your computer can take advantage of this will depend on how the RAID works). Now my understanding is that read performance isn't going to be any better than a single drive because you're waiting for 2 drives to seek to the right spots and the data probably has to be sent back sequentially anyway.

For RAID 1 your write performance can suffer since you have to write to 2 disks before you move on. If you have each drive on a separate channel or controller though you should be able to do this simultaneously so you should have the same performance as a single drive. If they're on the same channel you may have to wait for the 1st disk to finish writing before the 2nd disk is written too which will make things really chug.

As mentioned, RAID 1+0 gives you the best of both worlds.

If you use a good backup strategy, then I'd be more inclined to go with RAID 0 just because of the economics (more bang for buck in terms of storage space).

Personally though? I'd say the best 2 drive setup for a personal computer is 1 drive for OS and a separate drive for data. That way your OS drive can completely fail and your data should be fine, you can even take that 2nd drive to another computer if you need to. And it makes reinstalling your OS a lot easier. I probably wouldn't do a RAID setup unless I had 3 drives (1 drive for OS, 2 drives in a RAID for data).

 

[frugal]

The information I found on this indicated the opposite for the software RAID on Mac OS X. Here is a link:

Dead Link Removed

It seems to show that reading is considerably faster than writing. Whatever, there is a noticeable performance improvement.

Right, should've mentioned that all of what I said is theory the implementation can introduce several different variables, particularly if we're talking about software RAID.

 

[SusanV]

Hey... interesting stuff. I just popped in for a sec and caught up with the
discussion. Will be back later, but for now go look at this : http://www.barefeats.com/scratch.html

It's a test of read/write times with different raid and scratch configs using
Photoshop. Got it off a quick google of "raid photoshop". doh

later,
susan

 

[clay]

Well, what you say makes sense to me. I am wondering if he just got the labels wrong on his graph. I did a little more digging, and most other benchmarks support what you were saying: writes are faster than reads. I dunno, however it is, I know it is fast.

Right, should've mentioned that all of what I said is theory the implementation can introduce several different variables, particularly if we're talking about software RAID.

 

[frugal]

Well, what you say makes sense to me. I am wondering if he just got the labels wrong on his graph. I did a little more digging, and most other benchmarks support what you were saying: writes are faster than reads. I dunno, however it is, I know it is fast.

Yeah, something weird is going on there, my first guess would be something wrong with the test, my 2nd would be that there's something very strange with the software RAID implementation that he was testing.

Susan, the link you posted illustrates something else we didn't mention, nothing beats more memory when it comes to Photoshop performance. I know there's all kinds of different formulae you can find for how much memory you should have for Photoshop, if memory serves it's usually something like you should have as much available memory equal to around 4-5x the biggest file you intend to work with. That will really help keep Photoshop from hitting the scratch disk and make things much snappier.

 

[jd callow]

at one time the rumor was that your memory should be 3x the largest image. Because of the way Windows uses memory I can see why the multiplier has grown. It is my opinion that you have the max you can afford, the system can manage, or the motherboard will take. Memory is the #1 accelerator, followed by drive access...

 

[SusanV]

Still following along and learning...
Just found this online at Adobe. How to Optimize Photoshop CS3 for XP and
Vista. There is a little info on RAID 0 (good for PS scratch disk), but much
info about how PS uses RAM and other good optimization info. When my new
Dell 710 gets here, I'll be setting it up according to this...

http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=kb401088#resources

Susan

 

[Sean]

I'm a sucker for computer geek talk I only use PS for web type stuff so don't have smokin systems like you guys (just a little souped up macmini). I don't have much to add but I did get burned with raid a few yrs ago and lost the entire array. I've avoided consumer grade raid ever since. The APUG/Hybrid server has a beefy raid1 setup which has been rock solid. Now that I'm using a mac I just do a clone of my system to an external disk every few days. My clone tool is "SuperDuper" and it has a smart update feature so updating my spare disk takes only minutes. If one disk dies I can boot right up to the spare disk and not skip a beat. I'm not sure if Windows has such a clone tool other than ghosting the disk? Can windows boot off of a firewire disk like the macs can? I've been out of the loop with Vista..

 

[lenny]

I just finished building a 5 TB RAID. Works like a charm. I use cheap 2 TB Bigger disks from LaCie (which are RAID's themselves) for the backup system. I wouldn't recommend them except for backup, but they are Firewire 800, which is nice.

The 5 TB RAID was 5 Hitachi 1 TB drives set up as RAID 0 (striped, for speed) I set up 3 different partitions, the first one is usually the fastest. Cost about 2K (just under).

I have another RAID that is 4 160 gig, 10K drives for speed.

You can save a lot by being willing to buy the components yourself. I also used SOFTRaid, a mac application, and SATA cards from Sonnet and their latest 5 Bay case, very sweet.

All you have to do is screw the internal drives into a bracket, and slide the bracket into the case and you can save thousands.

It is very safe as long as you back up every night (I use the $15 Deja Vu, whicih works fine), and can recover easily from a drive failure - all drives fail at some point... I've seen lots of hard drives cook over time.

I just replaced one of my drives, the manufacturer sent it out within a couple of days and all I did was slide it in. Since I was using the LaCie drives for backup, I felt almost no drop in production...

 

[jag2x]

I tried this route with a few SCSI drives at once stage. Having 2 controller each having 2 drives with RAID-0 across them all. What I did was use a benchmark tool to calculate the READ/WRITES on the drives. I ended up only using 3 drives, and was getting around 95-110 MB read/write per sec. When using 4 drives I found it got a lot slower?
I stopped using the drives and just went on using my blank IDE drive as a scrath drive, because the SCSI drives I had would generate soo much heat you could practially fry an egg on it!
Though I suggest try getting a benchmark tool to check your throughputs...

One thing you might want to look into is SSD, solid state drives. I havent used one but reading articles on them they seem the way to go for exteremly fast read/writes. Actually i'm sure you can get them but be prepared to pay top dollar for them. Its more like a USB drive but attached to the BUS, so the throughput is the same speed the BUS is. Try googling it.
Another one I've seen is iRam by Gigabyte, which goes cheaply, it can only handle up to 4GB. Seems to me a good way to go, that is if your swap file doesnt get more than 4GB! I know mine goes to around 2GB at times, but thats when I have too many files opened with several layers...

I also had the brainy idea of setting up a RAM DISK, on an external computer and attach to it via a shared drive...the throughput for that was slow for some odd reason? Even when I used a gigabit network card between them...

One thing I do want to get going is to be able to use voice commands to call certain parts of photoshop up, because of all this EXTRA clicking i'm sure to get RSI any time soon! I did try a trial version of Dragon Dictate and it did call a few of the commands in Photoshop, though it was tricky to call the integrated commands. Perhaps now that I think about it, the best way would be to associate a key command or action and call those commands via Dragon Dictate.
my two cents...

 

[cmo]

Some words about what I use...

The whole workstation cost ca. 1200 Euros (ca. 1800$) including Windows XP Home, it was assembled from pieces by a mailorder house. I could have paid twice that price for a Mac Pro, right. I use this PC for one year and had no trouble, no data loss, and not one single crash.

Nvidia
Board: DFI LANparty UT nF3 Ultra-D. This is actually a board that gamers love. Based on nVidia nForce 3 Ultra with integrated RAID controller.
CPU: AMD X2 3800+
Graphics card: Matrox G550
Power: Enermax 365Watt
RAM: 2x 1024MB MDT plus 2x 512 MB = 3000 MB

The hard disks are the interesting part:

1. 4x 80GB SATA Hitachi as a RAID 0+1 using the RAID controller on the mainboard. It has two partitions, "C" for Windows and the programs and "D" for the data that I currently use, e.g. a part of my archive

2. 1x 320 GB IDE as internal backup.

3. In one slot I have an eSATA controller. This is for my external backup disk, a 1 TB Samsung Spinpoint in an external Icy Box case that allows USB2 and eSATA. In fact, eSata is lightyears ahead of any USB disk. Probably the fastest backup I ever had.

I spare you the description of the case (huge) and fans (many).

But one word about the performance: a b/w scan from my Imacon is pretty big. One 35mm negative as a 16 bit Tiff has ca. 75 MB. This is opened in Photoshop CS2 in about 2 seconds. Not too bad.

I spent ca. 45 Euro on a wonderful program named Acronis True Image which allows disk images. In case of a crash I can easily retrieve the whole setup and my data, much better than any other backup software I ever used (Retrospect was the worst by far).

What really annoys me is that I paid something to Microsoft. If Photoshop were available for Linux or if GIMP had 16 bit Tiff processing I would have a Linux (Ubuntu) computer.

 

[Carl1]

Susan,

If I was you I would get no less than 6 drives using a Raid 0 setup. If you can buy 10k rpm drives and use a separate Raid controller that has at least 512kb controller cache and plugs into a Pci-e slot that would help a lot. If your going to scan 4x5 your files will be ~1gb in size or so which means your going to run out of ram as soon as you open up PS and start on creating layers. The speed of the raid array is essential for that. Set the raid array as your primary scratch PS disk. Check out my setup online at www.mondragonfineart.com

 

[Dismayed]

Make Sure You Back Up

Susan,

If I was you I would get no less than 6 drives using a Raid 0 setup. If you can buy 10k rpm drives and use a separate Raid controller that has at least 512kb controller cache and plugs into a Pci-e slot that would help a lot. If your going to scan 4x5 your files will be ~1gb in size or so which means your going to run out of ram as soon as you open up PS and start on creating layers. The speed of the raid array is essential for that. Set the raid array as your primary scratch PS disk. Check out my setup online at Dead Link Removed

RAID 0 with 6 drives? Just be sure to back up your data because you'll lose your data with the failure of a single drive.