Doing color negative inversions manually

[koraks]

This is something I've brought up a few times on the forum, also because the issue of "how do I get those damn colors to come out OK on my C41 scans" keeps popping up (accompanied with much pulling of hair and gnashing of teeth). When it comes to getting consistent colors (frame to frame, film to film), I find that manually doing the inversion and color balancing based on a raw/positive scan of the film works best - at least for me.

About a year ago, I wrote a blog about it, which I've linked to from time to time on the forum: https://tinker.koraks.nl/photography/flipped-doing-color-negative-inversions-manually/
Recently, @Andrew O'Neill (re?)kindled an interest in color negative film and has done a few videos on it. In one particular thread, we got to talk about the inversion/color balancing process, with an eye on comparing different types of film and different exposures on the same roll: https://www.photrio.com/forum/threads/first-time-using-120-gold-200-portra-160-ektar-100.210141 Andrew suggested I do a video on the scanning and manual inversion process - and while I'm more of a writer than a vlogger, I caved...

So here's the video version of pretty much that same piece, where I show how I use Epson Scan software (which came with my old 4990) to make a 'raw' scan, and then use GIMP to invert and color balance some color negatives.



While this approach works very well for me, I admit it's not perfect, neither is it fool-proof, and it still relies in subjective eyeballing to get everything right. I'm sure there are also many improvements possible to this workflow - I'd like to invite anyone to offer them as suggestions, to point out the aspects that don't work particularly well in how I've shown it here. In short, feel free to discuss, comment and make your own spin on this.

 

[_T_]

Very good post. Very well put together video. My process is very similar.

There are some cases in which one might find that this procedure proves inadequate: low CRI subject lighting, mixed subject lighting, expired film, improperly developed film, etc.

But cases such as these have a plethora of unique solutions that range from very simple compensations for crossover to individually masking and adjusting the color of each subject in the photo.

I appreciate you giving an example of some problematic frames with those over exposures, it can be very informative to see how the procedure differs for cases outside of the ideal. I'm not certain that going further down the rabbit hole of suboptimally produced negatives would be as fruitful, because the worse the negative is to begin with the more unique and specific the solutions tend to become.

 

[Mick Fagan]

Welllllllll, that was really interesting.

I have the Epson V800 scanner, and the scanning setup looks to be identical to what I have, so I'm going to give colour negative scanning a go.

I've never scanned colour negatives, in fact I've only used my scanner for B&W 4x5" film, and I've been very happy.

I have a few thousand rolls of colour negative 35mm film and about 100 sheets of 4x5" colour negative film to sort of go through. I had been colour negative printing in my own darkroom for around a 25 year period, but stopped using colour negative film as the darkroom time was so long and to be honest, getting quite expensive.

I've never before contemplated doing colour negative scanning, but after seeing how you did the three colours individually, I now see how simple it can be. It was interesting to see you adding contrast and effectively adding density by changing the curve(s). It also looked a bit like the density change with RA4 printing whereby making the print darker by more exposure, adds red to the image.

I also use Gimp, but to be honest I'm was very unsure of how things could, or would work with colour negatives. In 1982 we purchased our first Chromacom scanning system, which from memory was approximately $1,000,000 USD and the process sort of worked just like you displayed. We really have come a long way with regard to cost reduction.

Many thanks for that video.

 

[koraks]

Thanks for the kind comments, guys!

@_T_ yes, you're absolutely right in that specific cases sometimes require specific solutions. I find that when I scan negatives the way Epson (&co) intend it, i.e. scan as color negative so you get a positive image with the mask removed etc., I end up doing a lot of tailoring to get the result I want. With properly processed negatives, the process I showed in the video (and that apparently you use, too) gets me very close pretty much all the time. It works for C41 and ECN2 (the strips I showed in my video were actually Vision3, so ECN2), and even for oddball films like maskless Aerocolor, super-contrasty Phoenix, very low-saturated/weirdly-masked Shanghai 400 or very expired film with a dense mask. A little odd color balance or even a decent dose of crossover don't really affect this process; you compensate for it automatically. I do find it really helps to keep several frames of the same film in view, preferably frames with different subject matter, contrast etc, so you can average out any casts/biases on the fly.

I have the Epson V800 scanner, and the scanning setup looks to be identical to what I have, so I'm going to give colour negative scanning a go.

Yes, this should work for the V800 just fine! In case you hit upon any differences in the scanning process, please post back; it'd be interesting to know.
And indeed, my corrections resemble a lot what you'd do in the darkroom. If you noticed that S-curve contrast adjustment - well, that's basically what color paper does, too. I just added a toe and a shoulder, which is what you get when you print the negatives, optically. And the color adjustments are indeed very much the same as what you'd do through filter adjustments. The main differences is that it's more flexible in digital space, since you can introduce (and thus, correct) things like crossover. Maybe I should do another video on this one of these days.

 

[jejes]

I like your way, But i like too this way
I use Negative Lab Pro and Negmaster with success. Just give it a look.

 

[_T_]

I used to use a process similar to alex burke's until I realized that removing the orange mask from the image by inverting the color of the base is an entirely unnecessary step.

The exact same results can be had by skipping that step and adjusting the black and white points of the RGB curves because the mask is in the histogram with the information of the image and gets cancelled out automatically by properly setting the black and white points.

 

[koraks]

i like too this way

That's conceptually similar to what I do, but he's a bit more careful/specific about avoiding clipping. The basic principle seems the same, though. It won't work in GIMP (yet) because this approach relies on dynamic adjustment layers. That's slated for the near future in GIMP.

I use Negative Lab Pro and Negmaster with success. Just give it a look.

I'm aware of it, but it won't give consistent results on consecutive frames. It can be satisfactory and very user-friendly on a frame-by-frame basis if consistency is not a requirement.

The exact same results can be had by skipping that step and adjusting the black and white points of the RGB curves because the mask is in the histogram with the information of the image and gets cancelled out automatically by properly setting the black and white points.

Exactly!

 

[Alan Edward Klein]

I used to use a process similar to alex burke's until I realized that removing the orange mask from the image by inverting the color of the base is an entirely unnecessary step.

The exact same results can be had by skipping that step and adjusting the black and white points of the RGB curves because the mask is in the histogram with the information of the image and gets cancelled out automatically by properly setting the black and white points.

How do you set the black and white points when you scan? Can you scan flat and set them in your post-processing program also?

 

[_T_]

When inverting manually I find it best to change the scanning software's settings so that it makes the fewest changes possible to the information coming out of the scan head. I think that's what you're referring to when you say scanning flat. I want as many of the decisions regarding color to be made manually by me at the post processing phase as possible.

I use the raw unprocessed 48bit HDRi data from the scanner with all the settings that could alter the information turned off. I use silverfast to produce these files which means I also have to be certain to turn off negafix which will apply color tranformations even to the 48bit HDRi raw.

So if you're setting the black and white points during the scanning phase they should be at their default positions. The black point should be set as low as possible and the white point should be set as high as possible so that none of the information is clipped. Or, if possible, it would be even better to turn off the black and white point settings completely.

 

[koraks]

How do you set the black and white points when you scan? Can you scan flat and set them in your post-processing program also?

For consistency, what you want is to scan flat, and most importantly: scan the same time, every time. This means that if you set black & white points during scanning, they should always be set at the exact same points. I circumvent this by scanning as shown in the video, which effectively disables all white/black-point setting during the scan process. This means I get the same scan every time regardless of what kind of film is on the platen. This gives me a consistent starting point for adjustments, and it means that if I work out the adjustment for one roll of film, I can then duplicate the exact same adjustment to all other film of the same type that's processed in the same way and get the same color balance every time. By contrast, if you manipulate contrast or color balance during the scanning process itself (and/or let the scanning software automatically determine something), you'll get a different result every time. The result may still be perfectly satisfactory, but it'll be difficult or even impossible to get consistent results from frame to frame and roll to roll.

 

[Alan Edward Klein]

When inverting manually I find it best to change the scanning software's settings so that it makes the fewest changes possible to the information coming out of the scan head. I think that's what you're referring to when you say scanning flat. I want as many of the decisions regarding color to be made manually by me at the post processing phase as possible.

I use the raw unprocessed 48bit HDRi data from the scanner with all the settings that could alter the information turned off. I use silverfast to produce these files which means I also have to be certain to turn off negafix which will apply color tranformations even to the 48bit HDRi raw.

So if you're setting the black and white points during the scanning phase they should be at their default positions. The black point should be set as low as possible and the white point should be set as high as possible so that none of the information is clipped. Or, if possible, it would be even better to turn off the black and white point settings completely.

CORRECTED: By scanning flat using my Epsonscan software, I mean that the black and white points are at 0 and 255. All other settings are Off. Then when I get the scanned file, I set the black and white points manually in Lightroom or Elements right next to the ends of the data. . Then scan,
Alternatively, I leave all settings off except I set the black and white points in Epsonscan before the scan. I place the black point a little before on the left and the white point a little after data on the right.

Someone once said you get more data the first way by setting point before the scan. But I have never been able to confirm that it makes a difference doing it before the scan or in the editing program. Do you know?

 

[Alan Edward Klein]

For consistency, what you want is to scan flat, and most importantly: scan the same time, every time. This means that if you set black & white points during scanning, they should always be set at the exact same points. I circumvent this by scanning as shown in the video, which effectively disables all white/black-point setting during the scan process. This means I get the same scan every time regardless of what kind of film is on the platen. This gives me a consistent starting point for adjustments, and it means that if I work out the adjustment for one roll of film, I can then duplicate the exact same adjustment to all other film of the same type that's processed in the same way and get the same color balance every time. By contrast, if you manipulate contrast or color balance during the scanning process itself (and/or let the scanning software automatically determine something), you'll get a different result every time. The result may still be perfectly satisfactory, but it'll be difficult or even impossible to get consistent results from frame to frame and roll to roll.

Those are good points. See my last post about the quantity of data using the two methods.

 

[albireo]

This is something I've brought up a few times on the forum, also because the issue of "how do I get those damn colors to come out OK on my C41 scans" keeps popping up (accompanied with much pulling of hair and gnashing of teeth). When it comes to getting consistent colors (frame to frame, film to film), I find that manually doing the inversion and color balancing based on a raw/positive scan of the film works best - at least for me.

About a year ago, I wrote a blog about it, which I've linked to from time to time on the forum: https://tinker.koraks.nl/photography/flipped-doing-color-negative-inversions-manually/
Recently, @Andrew O'Neill (re?)kindled an interest in color negative film and has done a few videos on it. In one particular thread, we got to talk about the inversion/color balancing process, with an eye on comparing different types of film and different exposures on the same roll: https://www.photrio.com/forum/threads/first-time-using-120-gold-200-portra-160-ektar-100.210141 Andrew suggested I do a video on the scanning and manual inversion process - and while I'm more of a writer than a vlogger, I caved...

So here's the video version of pretty much that same piece, where I show how I use Epson Scan software (which came with my old 4990) to make a 'raw' scan, and then use GIMP to invert and color balance some color negatives.



While this approach works very well for me, I admit it's not perfect, neither is it fool-proof, and it still relies in subjective eyeballing to get everything right. I'm sure there are also many improvements possible to this workflow - I'd like to invite anyone to offer them as suggestions, to point out the aspects that don't work particularly well in how I've shown it here. In short, feel free to discuss, comment and make your own spin on this.

Wow, great video @koraks. I don't use Epsonscan but this is great. Nicely presented too, well explained and clear - this is the youtube content I like. More of this please.

 

[koraks]

Someone once said you get more data the first way by setting point before the scan. But I have never been able to confirm that it makes a difference doing it before the scan or in the editing program. Do you know?

I'm not sure, but I'd expect that the person you cited is right. But we need to differentiate between having 'more data' vs 'more information'. The reason is that in principle, you're acquiring a 16-bit per pixel image stream from the scanner in both scenarios. So the total amount of data is the same in both cases. But this doesn't mean that both streams contain the same amount of actual information. In other words: it's possible that in one case, part of the pixel data is effectively padded with ones or zeros, resulting in a lower effective bit depth than 16. Or, put yet differently: while 16 bits allows for distinction of 65535 brightness levels, a padded 16-bit data stream may be able to differentiate significantly less. Whether this actually happens, I don't know, and it's possible that both answers are true, to an extent, at the same time.

Consider that the scanner can scan a dynamic range of let's say 4.0logD (I think Epson claims something like that). Setting the black and white points to the extremes during scanning would logically map this entire range to the 65535 levels the 16-bit analog-to-digital converter (ADC) is capable of resolving (give or take a little noise). If you instead truncate the range by shifting the white and/or black point, there are two possible ways in which this truncation translates into the digital domain. Either, the same 16 bit sample is acquired, but this time the highest and/or lowest values are simply lopped off, the remaining center bit is multiplied so it again occupies the full 16 bit range, and the intermediate values of this integer multiplication are simply padded with zeros (these would be the leas significant bits). The resulting data would nominally be 16 bits per pixel, but in reality, it would hold less information. In this scenario, it's still possible that the scanning software doesn't actually pad the data with zeroes, but instead interpolates intermediate values more or less intelligently. This would result in an in-between situation where 16 bits of information seems to be present, but closer analysis would still demonstrate that some of the data are effectively made up.

Alternatively, when you set the software to scan only a part of the full dynamic range, the scanner could use analog gain (there are a few ways for accomplishing this) to actual amplify the signal before it's fed into the ADC. An analog offset might also be included so that both the adjusted black point as well as the white point are taken into consideration already in analog signal conditioning. This would theoretically result in 16 'real' bits of image information. The penalty is obviously more noise, which would in fact reduce the amount of real information a little - but it may (probably will) still be better (i.e. more effective information) than in the digital-padding scenario above.

I haven't looked into it deeply, but I suspect both mechanisms are used at the same time - both analog and digital gain (probably with some 'smart' padding) are used if you adjust the white and black points. This means that you gain some real information compared to the 'flat scan' (of the entire scannable density range) and then a dramatic curve adjustment. The penalty is, obviously, that any such black & white point adjustment will have to account for the actual film densities you're trying to scan, and this automatically means that you'll sacrifice some degree of consistency. It's still possible to strike a compromise and set the black & white points in such a way that they'll plausibly cover the density range of any media you're trying to scan (let's say, super-high contrast maskless Phoenix 200 film all the way to rather soft ECN-developed Kodak Vision3 film). I've not explored this option, because I find that the actual image data resulting from a straight/flat scan as I demonstrate in the video is amply sufficient to get good positives from most of the film I've handled so far. But there may be room for improvement, based on your argument.

Wow, great video @koraks. I don't use Epsonscan but this is great. Nicely presented too, well explained and clear - this is the youtube content I like. More of this please.

Thanks, that means a lot to me; much appreciated! I'll keep it in mind!

 

[_T_]

In silverfast the gain control is separate from the black and white point.

I’m fairly certain that setting the black and white points in silverfast does exactly the same as it would in gimp or photoshop and simply crush all the information (or lack of information) into pure black and pure white after the scan is complete resulting in a loss of data though maybe not of any real information. Either way the controls are less precise so I would still prefer to do these transformations in post. I don’t think you could achieve a higher effective color depth by setting your black and white points in silverfast.

What epson scan is doing would require a bit of testing. I don’t have epson scan.

 

[koraks]

Epson Scan doesn't have a dedicated gain setting, so I expect it has an in-built algorithm that adjusts gain based on the more 'user friendly' (but somewhat opaque) settings in the Epson Scan user interface. But that's a guess on my part. I've not done specific testing on this.

 

[cerber0s]

Very nice @koraks ! I actually made a video a while back about almost the same thing, but never posted it here. Mine compares three different methods. The first is a manual conversion using Affinity, the second is manual conversion using "Negadoctor" in Marketable, and the last method is "automatic" in Smartconvert.

When doing manual conversion I have a similar method to yours, but I think yours is much more elaborate and probably yields more consistent results. I will definitely try it.

One difference for me is that I scan with a digital camera.

Here's the video. Each image is only converted to a "starting point" to keep doing final corrections from.

 

[koraks]

Great, thanks for adding this @cerber0s! Especially the fact that you start from a 'camera scan' will be useful to many people.

 

[Alan Edward Klein]

I'm not sure, but I'd expect that the person you cited is right. But we need to differentiate between having 'more data' vs 'more information'. The reason is that in principle, you're acquiring a 16-bit per pixel image stream from the scanner in both scenarios. So the total amount of data is the same in both cases. But this doesn't mean that both streams contain the same amount of actual information. In other words: it's possible that in one case, part of the pixel data is effectively padded with ones or zeros, resulting in a lower effective bit depth than 16. Or, put yet differently: while 16 bits allows for distinction of 65535 brightness levels, a padded 16-bit data stream may be able to differentiate significantly less. Whether this actually happens, I don't know, and it's possible that both answers are true, to an extent, at the same time.

Consider that the scanner can scan a dynamic range of let's say 4.0logD (I think Epson claims something like that). Setting the black and white points to the extremes during scanning would logically map this entire range to the 65535 levels the 16-bit analog-to-digital converter (ADC) is capable of resolving (give or take a little noise). If you instead truncate the range by shifting the white and/or black point, there are two possible ways in which this truncation translates into the digital domain. Either, the same 16 bit sample is acquired, but this time the highest and/or lowest values are simply lopped off, the remaining center bit is multiplied so it again occupies the full 16 bit range, and the intermediate values of this integer multiplication are simply padded with zeros (these would be the leas significant bits). The resulting data would nominally be 16 bits per pixel, but in reality, it would hold less information. In this scenario, it's still possible that the scanning software doesn't actually pad the data with zeroes, but instead interpolates intermediate values more or less intelligently. This would result in an in-between situation where 16 bits of information seems to be present, but closer analysis would still demonstrate that some of the data are effectively made up.

Alternatively, when you set the software to scan only a part of the full dynamic range, the scanner could use analog gain (there are a few ways for accomplishing this) to actual amplify the signal before it's fed into the ADC. An analog offset might also be included so that both the adjusted black point as well as the white point are taken into consideration already in analog signal conditioning. This would theoretically result in 16 'real' bits of image information. The penalty is obviously more noise, which would in fact reduce the amount of real information a little - but it may (probably will) still be better (i.e. more effective information) than in the digital-padding scenario above.

I haven't looked into it deeply, but I suspect both mechanisms are used at the same time - both analog and digital gain (probably with some 'smart' padding) are used if you adjust the white and black points. This means that you gain some real information compared to the 'flat scan' (of the entire scannable density range) and then a dramatic curve adjustment. The penalty is, obviously, that any such black & white point adjustment will have to account for the actual film densities you're trying to scan, and this automatically means that you'll sacrifice some degree of consistency. It's still possible to strike a compromise and set the black & white points in such a way that they'll plausibly cover the density range of any media you're trying to scan (let's say, super-high contrast maskless Phoenix 200 film all the way to rather soft ECN-developed Kodak Vision3 film). I've not explored this option, because I find that the actual image data resulting from a straight/flat scan as I demonstrate in the video is amply sufficient to get good positives from most of the film I've handled so far. But there may be room for improvement, based on your argument.



Thanks, that means a lot to me; much appreciated! I'll keep it in mind!

From what I experienced, the scan is the same regardless of the settings. Settings are all applied after the scan. In effect, Epsonscan has two functions: first, it scans everything. Second, it applies edits just like any other editing program to the scan file and creates and saves the final edited file. You see this in Prescans. As you adjust the edit settings, the displayed data changes. Obviously that's editing changes because the prescan is over. The final scan operates essentially the same way.

If my theory is true, then one could wait to apply the black-and-white points and all other edits to the scanned file with one's favorite editing program.

One caveat. ICE requires a second IR scan to determine and reduce dust. But the first "primary" data scan is still a "flat" scan.

 

[koraks]

That's possible @Alan Edward Klein ; I've never tested it specifically. It wouldn't surprise me if the scan really is the same and that only software like VueScan really manages to influence hardware gain.

 

[Alan Edward Klein]

That's possible @Alan Edward Klein ; I've never tested it specifically. It wouldn't surprise me if the scan really is the same and that only software like VueScan really manages to influence hardware gain.

The problem with non-Epson programs using hardware gain to amps is that one has to assume that Epson designed their scanners for maximum dMax with minimum distortion. Raising the gain will distort just like audio speakers driven too high with volume. Why would Epson set the gain less than its best maximum with no distortion? That's one of their advertising elements, giving the dMax value, which provides the best details and penetration through dense areas of the film.

In any case, wasn't the speed of the scanner that affected the revised dMax? Slower would allow more data in shadow areas, or so the claim went. Or was it a change in light intensity or amp gain? I;ve never seen anyone show a comparison that made a difference versus normal scans.

You can't squeeze blood out of a turnip.

 

[_T_]

I'm not sure that epson software is changing the hardware gain during scanning. I'm pretty certain that silverfast is not in my system. But I believe that vuescan claims to be able to do so, or at least offers a software option that appears it could be doing something like that.

All of this of course could be dependent on the scanner you're using. Maybe some scanners are able to alter the hardware gain (or even exposure) and some are not.

 

[avandesande]

Why not use a color head to illuminate the negative and digitize with a camera? Instead of correcting curves in software you can use the color head to correct the image. Isn't this the way a Fuji scanner works?

 

[koraks]

Raising the gain will distort just like audio speakers driven too high with volume. Why would Epson set the gain less than its best maximum with no distortion? That's one of their advertising elements, giving the dMax value, which provides the best details and penetration through dense areas of the film.

As explained in my earlier, lengthy post, there can be a net information gain by applying analog gain before the signal is fed into the ADC if media are scanned with a density range less than the scanner's dmax.

In any case, wasn't the speed of the scanner that affected the revised dMax? Slower would allow more data in shadow areas, or so the claim went.

If this is the case, and it may very well be, then analog gain is indeed used, it seems. One of the ways to amplify the signal on a photo site is to increase exposure time. This logically results in a longer scan time. It also increases noise, but again, the net information gain can outweigh the disadvantage of the noise generated.

Why not use a color head to illuminate the negative and digitize with a camera? Instead of correcting curves in software you can use the color head to correct the image. Isn't this the way a Fuji scanner works?

This is discussed in a few other threads on camera scanning and the best light sources to use for this. What this doesn't address is the issue of frame-to-frame or roll-to-roll consistency, or that of color balancing. Using a color head doesn't automatically mean the scans come out perfectly balanced. Also, (quasi-)modern Fuji Frontier scanners use an RGB LED array for illumination - is this what you understand to be a 'color head'? Keep in mind that when printing color the old-fashioned way, you still adjust filtration based on the film used and usually for exposure conditions, and you may have to account for deviations in development if that happened to be a little sloppy. There's no auto-magic when using a color head.

 

[brbo]

I'm not sure that epson software is changing the hardware gain during scanning. I'm pretty certain that silverfast is not in my system. But I believe that vuescan claims to be able to do so, or at least offers a software option that appears it could be doing something like that.

All of this of course could be dependent on the scanner you're using. Maybe some scanners are able to alter the hardware gain (or even exposure) and some are not.

I'm yet to come across a scanner where you are able to adjust sensor gain (in user software). Vuescan's naming of the feature ("analogue gain") is probably misleading people into thinking this is hardware (iso) gain as known in many digital cameras. It is not. It acomplishes this by extending the exposure times. This is documented in their manual so I'm really puzzled by how persistent that misconception is.