A TB of storage

Curt · Mar 5, 2010

Now that I'm making large scan files and have used a portion of my external disk drive that I also use to backup my computer I've decided to get another external drive for image files. I am looking at 1 and or 1.5 Terabyte drives.

What storage system are you using?

Curt

pellicle · Mar 5, 2010

I use my negatives as my primary storage.

My scanner is my optical drive to read this.

I have found that since I started scanning (in 1996 or so) that my technique and available equipment has advanced so that its better to go back to the negs and scan again if I want to print.

what will you do when your drive dies? Will you have redundant backups?

nope, negs will be around in 60 years, drives wont (Ohh, and I'm a database analyst and work for a national library in a mass digitization [news paper archiving] project in my present day job)

here is a lovely instance of the reasons why I currently stand by my above recommendation

http://vivianmaier.blogspot.com/

This was created in dedication to the photographer Vivian Maier, a street photographer from the 1950s - 1990s. Vivian's work was discovered at an auction here in Chicago where she resided most of her life. Her discovered work includes about 100,000 mostly medium format negatives and a ton of undeveloped rolls of film. Born February 1, 1926 and deceased on Tuesday, April 21, 2009.

reckon anyone would be fishing her HD's out of the garbage?

PS: I have 2 USB TB drives and 3 PC's (two XP one Redhat) with various sizes of internal drive. I presently cross back up to 2 of them and the TB storage ... I have another one "off site" on my desk at work. Disaster recovery.

TSSPro · Mar 5, 2010

Its not a matter of if your external drives will fail, but when. They will not last forever and technology will march on without them. I tend to go for drives that have an external power source and an on-off switch so that your drive wont spin even if its not being used.

I have been using iomega drives lately. They are fairly inexpensive and have the on-off switch which can help in limiting their overall wear.

Best of luck with your choice.

pellicle · Mar 5, 2010

Hi ... just to add to the content for the OP to consider

TSSPro said:
Its not a matter of if your external drives will fail, but when.

this is 100% correct. Further you need to think in statistical evaluation, mean time before failure could mean it dies in ten days or dies in 10 years

you just never know

sometimes bearings make horrible noises for months before it goes, other times internal controllers die and its kaput.

I have been using iomega drives lately. They are fairly inexpensive and have the on-off switch which can help in limiting their overall wear.

I believe that we get best life from the drives which are running permanently and never switched off. Certainly high rates of on off will lead to an early death. I've got a nice buffalo brand drive here which powers down automatically as the machine shuts down or you dismount it.

a nice touch ... but it does not seem to like being inactive and goes into some sort of strange noisy holding pattern when its inactive. I'm concerned that this indicates that when the head is "parked" its rubbing on something.

another issue in backup is that you actually verify the data ... copy does not verify itself (though some utilities do) thus (for us) an issue is having a file format which is fault tolerant. If you corrupt a bit in a TIFF you mostly will only loose a pixel, but if you do that with a JPG you'll loose more.

Marco B · Mar 5, 2010

TSSPro said:
Its not a matter of if your external drives will fail, but when. They will not last forever and technology will march on without them. I tend to go for drives that have an external power source and an on-off switch so that your drive wont spin even if its not being used.

I think the risk of modern drives failing is over-rated. I have a 6-7 years old Maxtor 200GB 7200RPM disk from a former computer in my also not so young replacement computer, that is still going strong after many start-ups. Running chkdsk with the /R option has shown me not even a single bad sector. I do make proper backups of course to an external hard drive via USB2. Total Commander has a great synchronizing option for that.

Most people will have replaced their computer and copied their data way before the hard-disk fails.

I think most cases where people assume a true hard-disk failure like a crash, are nowadays attributable to some form of small soft- or hardware glitch and the messing up of the file system as a consequence of that. Running chkdsk will in most cases solve this.

The worst thing in 15 years I ever had to deal with was when my nephew (the sun of my sister), tried downloading something and it failed.

Couldn't even start-up in safe-mode!

I than created a bootable BartPE CD with a limited XP SP3 installation on it, and was able to restore the PC by running chkdsk multiple times against the messed up drive. PE builder is really great for these kind of things.

That computer has been running safely for at least six months now since that issue...

And if all had failed, and I hadn't been able to bring back to life the operating system, the BartPE CD would have allowed me to copy data from the offended disk if not truely crashed. Re-installation of the operating system would than be safe.

Bob Carnie · Mar 5, 2010

Curt

Currently we use TBite drives to store images,
I am now considering a larger type unit Raid which can handle more info.

We store and scan large files of Colour Negative images because, these negatives do fall apart in dark storage and we want to make sure we have a record of critical images shot on colour negative.
We have seen colour negatives that are 15 years old , that have serious problems, and this is from more than one photographers work.

Pellicle makes a good point about storage, but I have to disagree when it comes to Colour C41 type negatives.
Black and White, Transparancies seem to have a longer shelf life and we are not as worried, about image shelf life , therefore are more confident to store images in negative or transparancey form in sleeves in boxes.

We do make scans *low rez of all film worthy of a first glance and over the years these scans do take up a lot of space and planning for the future is important.

As we scan, rather than network the files to another hard drive we burn dvd's and load to the drive from the dvd. This way we have original scan on the Scanner TB hard drive, the scans on DVD which we save, and then on new Hard Drive for storage on another computer.
This is as far as we have gotten with our strategy and basically waiting for the technology geeks to come up with a way to move with ease our images from these TB drives.

R Shaffer · Mar 5, 2010

I have had my external drive fail. It was a WD My Book, 500gb and it was my primary storage for my photos. Fortunately, I had my most valuable stuff, family images from vacations, holidays, ect on CD's as well. But I lost a lot that I really wish I still had.

So when I upgraded the computer, which I use for 100% of my work as well as my photos, I installed a 3- 750gig raid 5 array and have a spare matching drive in the closet, for hot swap if one dies. I can loose one drive and still have an operable system, if two die, then I am relying on my back-up. My back-up is a 1Tb Seagate External that ports into my router.

So I would 100% agree that you really need to PLAN your storage based on "when the drive fails" not "if the drive will fail".

Marco B · Mar 5, 2010

R Shaffer said:
So I would 100% agree that you really need to PLAN your storage based on "when the drive fails" not "if the drive will fail".

I totally agree, that is why I make backups to an external harddrive from my main drive on my PC. Still thinking of getting a second backup drive, to store outside my house, in case it burns down once

I was just saying, hard drives nowadays are quite reliable, compared to 10-15 years ago.

And I still think that many people who write-off their hard-drive as broken because "the system won't boot" or "start-up", are actually having resolvable file system issues.

They just don't have the knowledge to solve these. Buying a new laptop is the easy way out than...

Marco

pellicle · Mar 5, 2010

Bob

bob carnie said:
but I have to disagree when it comes to Colour C41 type negatives. Black and White, Transparancies seem to have a longer shelf life and we are not as worried, about image shelf life , therefore are more confident to store images in negative or transparancey form in sleeves in boxes.

thanks for providing that ... I am so far into digital that I almost only use black and white negative anymore. I completely was not thinking of colour, so when you said that I had a "douh" moment.

I have some E-4 stuff at home which is nearly clear now.

To the Curt:
Chemistry quality is of course critical and bucketloads of labs are skimping on that badly, so scanning colour neg and E-6 slides may provide an excellent option for archiving.

But don't underplay the storage failure, its like all those people who say "I've never had a car accident". You may never have one (good), but don't bank on that.

I personally buy a new drive every year or two and migrate all that is on the smaller one onto the bigger one. Since 2001 I've not lost any data even though some of the drives (re tasked to other purposes) have failed.

PS ... definitely do not rely on DVD-R or CD-R as backup

pellicle · Mar 5, 2010

Bob

bob carnie said:
This is as far as we have gotten with our strategy and basically waiting for the technology geeks to come up with a way to move with ease our images from these TB drives.

gigabit ethernet to a SAN or NAS?

As we scan, rather than network the files to another hard drive we burn dvd's and load to the drive from the dvd. This way we have original scan on the Scanner TB hard drive, the scans on DVD which we save, and then on new Hard Drive for storage on another computer.

which is (while expensive) an interesting strategy ... we recenly had a D#LL server go down and with the rescue disk inserted something dreadful happend ... software saw a number of disks attached and said "hey, I'll just write onto that fiber channel"

50 or so TB later .... took down two weeks of the scanning teams work.

you can bet that didn't go quietly in the organisation either...

Marco B · Mar 5, 2010

pellicle said:
I personally buy a new drive every year or two and migrate all that is on the smaller one onto the bigger one. Since 2001 I've not lost any data even though some of the drives (re tasked to other purposes) have failed.

With current low hard disk prices, this is probably not a totally ridiculous thing, but still I think many people would do better to learn basic computer and hard disk management, than buy a new drive even before it shows signs of issues (I DO say you need to back-up as well!!!).

Being able to fix and prevent issues by using basic operating system commands like "checkdisk(chkdsk)" with all of its options, is a great thing.

I regularly defragment and chkdsk my drives, do you?

Do you also buy a new car if the ash tray is full? Just joking here, don't feel offended... I can see your logic for a new drive to some extent, especially considering the price tags.

pellicle said:
PS ... definitely do not rely on DVD-R or CD-R as backup

Other nice anecdote... I recently needed a file from an old CD-R. This was one big exception of which I had NO backup. Even though the CD looked pretty pristine, with only a few minor scratches, I couldn't get it to read in my DVD-drive... the dreaded "cyclic redundancy error" The top layer of the CD, which contains the actual data, was OK.

I had read before about "polishing" the CD to get it readable again. Since there was nothing to loose, I first tried it with "toothpaste", per internet recommendations ... that only scratched more. Still unreadable.

Out of curiosity, if it COULD be done, I than bought a tube of some car polishing cream and rubbed it on with a soft cloth and cleaned the CD under the water tap. It came out with less visible big scratches, but with an overall misty appearance of very fine ones...

Reluctantly and sceptically, I than inserted it in my drive, to see what would happen. The drive chucked away on a low speed trying to read it...

I COULDN'T BELIEVE MY EYES, IT COPIED THE FILE I NEEDED!!!

Yes, it was solved, I had it back! What a bit of "tooth-paste" can't do

Bob Carnie · Mar 5, 2010

I think I should add that this thread, un-doubtley is about one of the most pressing problems facing photographers, labs, archivists now and in the future.
Any and all thoughts of strategy , from my perspective would be greatly appreciated.

I am about to purchase a big ass drum scanner to start making high resolution multiple separations of images and believe each image may require a couple of gigs storage space.

I do plan to output final size all the files to black white film as a permanent record, but all the work getting there will need to be stored for future use, which may be years from now.

This project will be for photographers who want the type of prints I am entertaining working on and the physicality of this is scaring the crap out of me.

Curt · Mar 5, 2010

Yes it's a problem that will come up, whether a dozen big files or hundreds of big files. They have to go some where. I started out with computers when I brought a Tandy 1000 home. It had the green screen and didn't do anything but consume energy. It's been a long trip in a short time since then, Thirty years and multiple iterations later the energy consuming computer does a whole lot more.

Disk drives do fail, no question there, I personally have not had one fail and I've had many of them. Anyone remember the Winchester drives? We had them to store and backup CAT scans. Now that was a nightmare but it worked reasonably well for the time. What is long term? I need to have the files I work with, a master as Sandy King mentioned and an adjusted, corrected, edited, working file or two. Now at my age long term storage is going to be less than a younger person who will have more time to worry it out. Given that my forty year old negatives, and the family negatives that go back a Century are in perfectly printable condition, I have the them as backups as long as they are physically safe, the physical safety of equipment is the same.

Modern disk drives are the current solution to storage unless one wants to use an online storage on someone else's drives with their backup strategy. Thinking about it for a while I came to the conclusion that when I make a scan and save the files, that's not it, I make the scan in order to make a print. When the print or prints are made I have done the work I intended to unless I want to make another one later. So the risk is losing the stored file between the time of storage and the time it is needed to make a print, soon or later. Again the original is a backup, not perfect, as anyone who has spent hours trying to recreate a file for the desired image look. Matching is not the same as the identical original unless the complete instruction set is recorded and that's not reasonable in the case of dodging and burning etc..

So I get the big drive, for me it's big, scan and store and print and in the back of my mind is that voice saying the equipment can fail what will I do.

Worker 11811 · Mar 6, 2010

I have two TB drives in my Mac Pro on top of the 500 GB that is the startup drive. I have a 1 TB desktop USB drive and two smaller desktop drives. One is 100 GB and the other is 250 GB.

The main, 500 GB drive holds all my day-to-day stuff. The first 1 TB internal drive is for storage and to back up some of the day-to-day stuff on the main drive. The second 1 TB drive is used mostly for scratch space and for holding source files for video editing projects and for Photoshop projects while I am working on them. When I am done working on a project I decide how important the working data for that project is. If it is important I archive all the data. If it is not very important I will save only the final output files and dump the rest.
The 1 TB external drive is used for Time Machine backups. I have Time Machine set to back up the main, 500 GB and the first 1 TB drive. The second 1 TB drive is NOT backed up via time machine. That drive contains mostly scratch. The source files from most of my projects are already archived. e.g. - The DV tapes for video projects are stored separately and I store the negatives for my photo projects in another place too. If that data is lost it can easily be recaptured.

Right now, Time Machine still has 200 GB of space to work with on that external 1 TB drive. That keeps all the backup data of everything done on this computer since last October, 2009.

The 100 GB portable drive is used for shuttling large amounts of data back and forth between home and work. (Things like video files.) The 250 GB drive is used to back up those things I think are REALLY important that I don't want to lose. That drive is not connected to the computer unless I am using it to back up or restore data. At all other times it sits idle and unplugged.

As it stands, I have almost 4 TB of data storage and I have backups of everything and double backups of things I think are important. I can't think of any way to be more secure than that.

pellicle · Mar 6, 2010

Bob

bob carnie said:
I think I should add that this thread, un-doubtley is about one of the most pressing problems facing photographers, labs, archivists now and in the future.
Any and all thoughts of strategy , from my perspective would be greatly appreciated.

indeed, the view of our organisation is that all the technology exists now, it is only the will to pay for it. Costs are a problem as is the issue of dealing with archival detail. In our project (and I rub shoulders with many national library digitization project officers in many countries) the arguments pivot around how much do you want to archive, at what level is it to be preserved (reprographics vs display graphics).

The needs are in my view clearly case by case. For instance in the newspaper archival group they wished to provide A3 prints of each page for clients, clearly a big ask. While on the digitization of medieval parchment fragments the historians only sought smaller sizes for screen viewing, and were not interested in assaying what the paper type was. Projecting the needs of future historians is difficult, and the digitization team wanted 2400dpi scans of the parchments as 16 bit TIFF.

I'm on the storage team and I had other things to say, as did the budget controller.

If you have not already, start doing some some projections of growth in storage requirements and cost.

Meta data is also important to archive along with the original. Look into METS for that.

Curt

each of my 4x5 images is about 500Meg (for black and white 16 bit 2400dpi) so perhaps you can see why I justify them as their own storage.

Ask yourself a question, do you need to store all of it or only what you like? Clearly determining that is harder than you may anticipate. For 35mm archival make sure you pick a reasonable number. I find that 2400dpi is reasonable and can make an acceptable print but 1200dpi may be all thats needed. You can store ~4 images of 1200dpi that one 2400dpi image will take.

getting back to your question, the 1TB drives are a good solution for small scale things, and its likely that larger solutions will arrive sooner than your need does.

pellicle · Mar 6, 2010

Curt

Curt said:
Thinking about it for a while I came to the conclusion that when I make a scan and save the files, that's not it, I make the scan in order to make a print. When the print or prints are made I have done the work I intended to unless I want to make another one later. So the risk is losing the stored file between the time of storage and the time it is needed to make a print, soon or later. Again the original is a backup, not perfect, as anyone who has spent hours trying to recreate a file for the desired image look.

yep, that's about what I do too ... though Bobs point about colour materials loss is a significant one so I do recommend scan and backup. I found 1350dpi from my Nikon LS-20E was quite enough to show people and make a print.

If nothing else its diversification.

BTW, the reason I like my EPSON 4990 is I can lay strips down, batch scan the lot in strips of 6 and save each as a file at 1200dpi.

does me

Bob Carnie · Mar 6, 2010

Pellicle

Would you say that the TB raid systems that bundle multiple storage system together up to 18TB of storage is current state of the art, or would you say another system is better , bigger ??

I too now have about 6 TB of storage on my Mac including external. Three years ago I thought this would be ample for my operation.
But we are expanding our scanning service , specifically in the colour realm for reasons I explained. My clients are now being advised about their colour needs and we are getting ready for a serious few years of scanning.

It will take me longer to facilitate the output to BW Film as final pieces , due to my learning curve adjustments , testing of appropriate archival colour materials, and basically getting my skill level to the challange.

Like any craft this area of expertise will take me 5 years to satisfy my ego and current printing offerings.
So for 5 years I will be scanning, editing and storing crazy amounts of data, and as you say planning and having a budget to do this is of importance.
It would be great to concentrate 100% on this project but I have to pay the bills and staff with regular work therefore the timeline.

Ultimately my clients will be responsible for storage of their own data of each image, but that I cannot control so I need a plan that will get me through the next 10 years of scan,edit and output archival film.

Are there devices, coming in the near future that may be in the hundreds if not thousands of TB capabilities that us mere mortals may not be aware of ??

pellicle · Mar 7, 2010

Bob

weather has been perfect this weekend, so skiing came first ... sorry mate

bob carnie said:
Would you say that the TB raid systems that bundle multiple storage system together up to 18TB of storage is current state of the art, or would you say another system is better , bigger ??

we've got something over 80TB on our SAN, but I'm not the hardware guy. I'm a DBA and software - middle ware sort of specialist in the organisation. However I normally work with stuff which has BIG data repository and system growth.

I too now have about 6 TB of storage on my Mac including external. Three years ago I thought this would be ample for my operation.

it was probably even over kill

Are there devices, coming in the near future that may be in the hundreds if not thousands of TB capabilities that us mere mortals may not be aware of ??

I think that it will be hard to go past sata drives plugged into a controller for the foreseeable future. I don't think I could feel safe in suggesting a home-brew concoction of USB drives.

I would seriously examine gigabit switched network between workstations and the "host" which has a fiber channel to a SAN. You can throttle connections and dish out access to LUNs on the SAN. SAN management is sort of a specialist area in itself but you can get quite scalable and high speed stuff that way.

I can get some more details on a smaller operation my mate has running and see if that works for you or improves on what you're currently planning.

Since the level of TLA is getting pretty dense here so I'd better leave it here and offer you to PM me for more details as I can provide them.

Bob Carnie · Mar 7, 2010

I just downloaded a ton of info on SAN, I got some reading to do , thanks for the info I will follow with email later this summer with ques.

regards
Bob

pellicle · Mar 8, 2010

Bob

just make sure you're networks are switched, something like this. You do not want collisions on the network as they slow it down. One cable from any given workstation to the switch. Since the switch will be handling and managing the traffic between it and the server (and again only one thing on that cable run) it needs to have a faster connection to the server. Thin in and fat out ... gigabit in and fiber channel out. The key to FC is that it is deterministic unlike ethernet which just blurts out data and determines after that if everything worked ok (the CD of CSMA/CD) ... some other reading

http://en.wikipedia.org/wiki/Fibre_Channel

toyotadesigner · Mar 9, 2010

I have one 1 TB and one 2 TB WD MyBook Studio connected to my iMac for more than one year now. They are always on because - as Pellicle already mentioned - lots of switching will let them die faster.

They fill up pretty quick because a hi res scan of a 6x9 chrome @ 4.000 ppi and 48 bit result in a 660 MB file.

As Pellicle said: I have the original slides, so I don't worry too much because of improved scan software and know how of the operator

I've just checked prices - the 2 TB WD today costs just a tad more than my 1 TB a year ago <sigh>

About the panic of a data loss: two years ago we had a burglary and our computers and backups had been stolen. First it was a shock, but after purchasing a new machine I just scanned the most important images again.

That's one of the many advantages of working with film.

Curt · Mar 9, 2010

Is there a best recommended place to buy a TB drive?

pellicle · Mar 10, 2010

Curt said:
Is there a best recommended place to buy a TB drive?

I like the Buffalo brand, so if there is a better place to buy the same product it would be the one with the better price or the better return policy if it dies in the first month and the one which is the most convenient to get to.

toyotadesigner · Mar 10, 2010

If you live in Europe you might check Dead Link Removed, in CanUSA http://www.frys.com, they offer the WD MyBook Studio 2 TB for only 229 $
(<sigh> I wish we had those bargains in Europe)

donbga · Mar 10, 2010

pellicle said:
To the Curt:
Chemistry quality is of course critical and bucketloads of labs are skimping on that badly,

That's just BS Chris, there is no easy way for a lab to skimp on their C-41 chemistry, in fact it's not in their best interest to do so. They might do so out of neglect but those that do aren't in business for long.

A TB of storage

Privacy & Transparency

Privacy & Transparency