Silverfast 8: 16bit grey scale?

[alanrockwood]

It's generally not a great idea to do dynamic range or noise analysis on image data that has been conformed to a color space as you have to contend with the gamma encoding and the raw to colorspace transform and what effect that has on the resulting image samples.

So...

http://m.avcdn.com/sfl/epson_v850_raw_black_frames.zip

This contains two files: raw0001.tif and raw0002.tif. Both are a ~36x24mm black frame at 6400 dpi from the center of a completely blocked off capture area (except the scanner calibration area). Vuescan was used to capture and output the raw data with all the controls that vuescan can control (analog gain, etc) zeroed out and locked so they are exactly the same between scans with the only difference being raw0001.tif was captured in 16 bit mode and raw0002.tif was captured in 8 bit mode immediately after raw0001.tif.

Anybody can download these raw files and do whatever analysis they want on them.

I've done a very basic and preliminary look using some tooling that I use for my own internal software. The 8 bit samples were scaled up to 16 bit samples to make the comparison a little easier (using sample/255, then result * 65535, all in float to preserve precision).

The per color channel numbers:

raw0001.tif (captured in 16 bits, saved in 16 bits)
resolution: 9064x6046 px
bits per sample: 16 bits
analyze window_size: +-1024, x1: 3508, x2: 5556, y1: 1999, y2: 4047
max_r:1817, max_g:1911, max_b:1714
min_r:0, min_g:0, min_b:0
avg_r:334, avg_g:350, avg_b:313

raw0002.tif (captured in 8 bits, saved to 8 bits, scaled to 16 bits for analysis)
resolution: 9064x6046 px
bits per sample: 16 bits
analyze window_size: +-1024, x1: 3508, x2: 5556, y1: 1999, y2: 4047
max_r:59904, max_g:53248, max_b:32639
min_r:0, min_g:0, min_b:0
avg_r:1699, avg_g:1666, avg_b:1644

Just for edification, the same values, but scaled to 8 bits

raw0001.tif
resolution: 9064x6046 px
bits per sample: 16 bits (scaled to 8 bits)
analyze window_size: +-1024, x1: 3508, x2: 5556, y1: 1999, y2: 4047
max_r:7, max_g:7, max_b:6
min_r:0, min_g:0, min_b:0
avg_r:1, avg_g:1, avg_b:1

raw0002.tif
resolution: 9064x6046 px
bits per sample: 16 bits (scaled to 8 bits)
analyze window_size: +-1024, x1: 3508, x2: 5556, y1: 1999, y2: 4047
max_r:233, max_g:207, max_b:127
min_r:0, min_g:0, min_b:0
avg_r:6, avg_g:6, avg_b:6

Make of it what you will, though, again, I think it's pretty obvious that 16 bit capture has a significantly lower noise floor and therefore more dynamic range than 8 bit capture. Whether that makes it through the raw to colorspace operations and gamma encoding, is harder to tell as the analysis @alanrockwood has done so far, though sound from an analysis perspective, is being done on data that's already been conformed to a color space. Also whether that lower noise floor matters, depends on what you're capturing, so 8 bit capture very well may be adequate for lower contrast materials.

Either way, the raw files are available for anybody to do their own analysis and draw their own conclusions.

Finally, I come to a discussion of these scans by Adrian. They are scans of a black slide, so in a perfect world the output would be all zeros.

As Adrian noted, the 8 bit scan is noisier than the 16 bit scan. I am uploading the images and histograms to show this. For these histograms I zoomed in on the horizontal scale to 10% for each of the two scans. The horizontal scales are comparable for the two scans because they are both shown in a percent of full scale. (In this case full scale displayed here is 10% of full scale of the original scans.) The first image is the 16 bit histogram, and the second image is the 8 bit histogram.

Just eyeballing the histograms I estimate the noise in the 8 bit result to be about 2.5 times that of the 16 bit result. That is a big difference, but nowhere near the 257 fold difference one would naively expect based on the difference in bit depth alone. (As an aside, yes, 257 is the correct scaling difference between an 8 bit word and a 16 bit word, not 256 or 255.)

I have previously shown from fundamental considerations that if there is much noise showing up in an 8 bit scan then a 16 bit result has to have almost as much noise relative to full scale. At most the 16 bit histogram will show only a little less noise, not the two or three fold difference that we are seeing here.

Also, looking at this from the other direction, based on the width of the distribution in 16 bit mode (from which I estimate that the sigma of the 16 bit result on an 8 bit scale would be about one half bit), there should be a little noise showing up after conversion to 8 bits. There should be peaks at a few but only a few ADC step increments. Therefore, these results are hard for me to explain. There is an explanation for the difference. I just don't know what it is, but it can't be due to the difference in bit depth alone.

Also, it is worth pointing out that even just looking at the 16 bit histogram we can see that there is a lot of noise in a dark image acquired from the scanner, orders of magnitude more than the ADC step size at 16 bits. This implies that 16 bits has far far more bits than required to acquire the full dynamic range available, given the noise level inherent in the instrument. There is nothing wrong with this, but it does show that the scanner is relatively noisy.

I will shortly be posting some of my own dark scans. I already posted some, but I have repeated them with various parameters. As a teaser, I'm not seeing a lot of difference in noise my scans between 8 bit and 16 bit mode. I don't have a good explanation of why my results are different from Adrian's.

 

[alanrockwood]

Next I will discuss my re-scans of a dark region. Please refer back to my post #118 to see what the original full image was. This scan was performed an an Epson V/750 scanner, which is a comparable model but an older version compared to the V/850 that Adrian used.

For what I am posting below I took a small crop from the lower left hand corner of the fill image. In this part of the image the light is blocked by a combination of a book, a black piece of paper, and the opaque frame of the film holder. Based on the geometry I used there should be almost no scattered light reaching the sensors, so what I am seeing is the dark response of the system.

These were gray scale images, taken from the green channel. They were acquired in image mode using Vuescan. I don't think that Vuescan applied any profiles to the images. Anyway, that parameter was set to “default”, and I don't think it would make any sense to apply a color profile to a gray scale image, so I assume that “default” in this case means no color profile.

For the histograms shown below I cropped the left most 2% of the scans. I used photoline for the processing.

Please bear with me because there are several image to compare, so don't be overwhelmed by the number of images. I will give the commentary following the images.

The first two images were scans acquired in 8 bit mode and stored by Vuescan in 8 bit mode. The two scans were performed identically. I am showing these two in order to illustrate that scanning is not perfectly reproducible using this system. Note that there are two peaks showing. The relative abundances differ slightly between the two nominally identical scans. Also, if you look carefully at the two images (to the left of the histograms) they look similar, but not quite the same, another indication that scans are not perfectly reproducible on this scanner.

The peaks are spaced by one 8 bit ADC increment. There is actually a third peak present positioned at zero on the horizontal scale, but it is obscured by the axis. Also, for some reason photoline does not allow me to place the cursor there to get an abundance. However, from other experiments (which I will not describe here) I know that there is a peak at the leftmost axis, and I have reason to believe that the graph is scaled so that the obscured peak is at full scale.

The fact that there are two peaks showing in the 8 bit scan means that there is a little bit of noise in the system, enough to register non-zero counts in the lowest few ADC increments.

The third image was acquired in 16 bit mode and stored in 8 bit mode by Vuescan. It is extremely similar to the first to scans.

The fourth image was acquired in 16 bit mode and stored in 16 bit mode. Just by eyeballing the images it looks to me like the width of the distribution is about the same as in the scans stored in 8 bit mode. If anything it might actually be slightly broader and shifted slightly to the right

The fifth image was the same scan as the fourth image. However, it was converted to 8 bit using photoline before the histogram operation was performed. Note that it looks very similar to the other 8 bit results. If anything the distribution might be slightly wider and shifted slightly to the right.

All of this adds up to the following conclusions. There is noise in both the 8 bit and 16 bit images, and the amount of noise is almost the same regardless of whether they are in 8 bit or 16 bit mode.

The sixth and last image is one that was acquired in 8 bit mode and stored in 16 bit mode. There is a sparse set of peaks, as expected for an image stored in 8 bit mode, but compared to the other 8 bit images the peaks are spaced closer together. I don't know why this is happening. Perhaps the image was gamma encoded twice. Anyway, other than the fact that the peaks are spaced closer together, the image is pretty much similar to all of the others.

Here's what I conclude. The Epson scanner produces scans that are slightly noisy, and the noise is down at the range of about one bit relative to an 8 bit ADC. Furthermore, there is little or no difference in the noise, regardless of whether one store the results in 8 bit or 16 bit mode. Because there is some noise, there will not be banding in 8 bit mode, assuming that if one does subsequent image processing it is done in 16 bit mode. Also, because of the similarity in noise level in 8 bit vs. 16 bit mode the difference in dynamic range will be small-to-negligible.

I can't say what will happen with another scanner (Canon, Nikon, etc.), and I can't say for sure that the results will be the same if performed on another Epson scanner. I also can't explain why my results are different from Adrian's scans of dark images.

 

[alanrockwood]

Here's a good discussion of noise in CCD detectors.

https://hamamatsu.magnet.fsu.edu/ar...variety,capture of accurate image information.

Part of the article could be interpreted to mean that 12 bit ADCs are often appropriate. However, I'm sure this depends on many factors, and it is unclear how closely this relates to scanner applications. In any case, the ultimate decider is experiment. As my PhD dissertation used to say "Theory proposes. Experiment disposes." And he was a theoretician.

One thing the article emphasizes is the roll of temperature in affecting dark current (and noise that is associated with it.) As I'm sure many of you know, CCDs are often cooled for astronomy applications in order to reduce noise.

 

[Adrian Bacon]

I will shortly be posting some of my own dark scans. I already posted some, but I have repeated them with various parameters. As a teaser, I'm not seeing a lot of difference in noise my scans between 8 bit and 16 bit mode. I don't have a good explanation of why my results are different from Adrian's

Mine is a V850Pro with vuescan. What is yours, and which software? If using vuescan, are you zeroing out and locking the controls? If not, that can mess things up.

 

[Adrian Bacon]

I can't say what will happen with another scanner (Canon, Nikon, etc.), and I can't say for sure that the results will be the same if performed on another Epson scanner. I also can't explain why my results are different from Adrian's scans of dark images.

hmm… I suspect that there’s a setting difference somewhere. Did you save as a raw tiff? If not, whatever profile you have set for your output is applied. Also, unless you select lock exposure and lock color, vuescan will change the gain, black point, and white points to what it thinks is best, so if you want any consistency, you want to lock those controls and manually zero out the gain and appropriately set the black and white points, then save as raw.

 

[alanrockwood]

Mine is a V850Pro with vuescan. What is yours, and which software? If using vuescan, are you zeroing out and locking the controls? If not, that can mess things up.

Mine is a V750 Pro. I'm using vuescan. I believe the only difference between the 750 and 850 is the light source.

I'll have to check my settings. I know that some of them are locked by me.

 

[alanrockwood]

hmm… I suspect that there’s a setting difference somewhere. Did you save as a raw tiff? If not, whatever profile you have set for your output is applied. Also, unless you select lock exposure and lock color, vuescan will change the gain, black point, and white points to what it thinks is best, so if you want any consistency, you want to lock those controls and manually zero out the gain and appropriately set the black and white points, then save as raw.

I did not save as raw. I saved as TIFF without checking the RAW option. TIFF profile is unchecked, which I assume means that there is no profile applied to the TIFF image. I did not lock exposure. However, as can be seen in the scan of the full image, it includes full light and full dark regions, so if there is an automatic exposure taking place it should have scaled it to get maximum dynamic range to capture both ends of the exposure range.

I will try locking exposure and locking image color. I will also set the RAW mode output. In fact, I will set it to save in both TIFF and RAW. I will also set the white points at 1 and the black points at 0.

 

[Alan Edward Klein]

Let us suppose you have scanned 10,000 color images and saved them as 16 bit images. Further suppose these were 35mm images scanned at 4000 dpi. That's ~128 MB per image or ~1.28 TB of storage for the set. You are going to keep those images as an archive, unmodified. Scanning in 8 bits would save you ~0.64 TB of storage.

Now what happens when you start manipulating the images. You will be doing this one image at a time, so during the manipulation phase you are really affecting the archival storage requirement, even if you do a temporary save of the image you are manipulating. (I will assume you will be doing image manipulation in 16 bits regardless of the original bit depth of the archival images.)

Once you are completely finished with the manipulation (not saving the temporary intermediate results you made during manipulation) you have the choice of saving the results in either 8 bit or 16 bit format. It is generally accepted that once the image is in final form there is no perceptible difference between a 8 bit and sixteen bit image. Some output devices can't even deal directly with 16 bits.

Anyway, if you start with a 16 bit archival images, process them one at a time, and save them as 16 bit images in their final form (not saving the temporary intermediate results) then your total storage requirement is 2.56 TB.

On the other hand, if you start with 8 bit archival images, and do exactly the same thing (manipulating them in 16 bit form, but not saving the intermediate images permanently), and save them as 8 bit images in their final form you will require 1.28 TB, for a savings of 1.28 TB. The difference in storage may or may not make a difference to you. That's a personal decision.

Even if you start with 8 bit and save as 16 bit in final format to preserve the possibility that you might want to do further manipulations later, even though you thought you had already reached final form, you will still save about 0.64 TB if the original archival images are in 8 bit rather than 16 bit format.

Regardless of all of this, some scanner/software combinations only allow the operator of the scan to save in 8 bit results, so in that case there is no choice. I believe the Leaf scanner is one of them when using Silverfast. Then the question becomes, does it matter that you can only save in 8 bit format? If the 8 bit images are noisy/grainy enough then it doesn't really matter that 16 bit is not available.

Zeroing in on the Leaf scanner, one of the complaints about that ancient scanner is that it's a little noisy in the dense part of the film. That automatically means that there would be no point in saving the scans in 16 bit form, even if 16 bit saves were available because the noise level of the scanner alone is sufficient to make 16 bit saves superfluous, regardless of the amount of grain in the film.

This (i.e. the inherent noise level) may or may not apply to other scanner/software combinations.

So saving on storage of 1.28tb for 10,000 scans is the only advantage of scanning in 8 bit? Doesn;t seem worth the time. First off I don;t have 10,000 scans. But even so, what's the cost of that storage? $20? What happens if someone develops a better program in the future that needs 16 bits?

It just seems like a lot of effort for little gain if any. It's an interesting discussion from a technical standpoint. But from a practical one, not so much.

 

[alanrockwood]

I did a little sleuthing and I learned that when I save the same scan in 8 bit mode in both raw and tff, the two results are nearly identical. What I mean is that when I zoom in on a histogram region the peak pattern is nearly the same.

However, when I save a scan in 16 bit mode in both raw and tiff mode they histograms look different when I zoom in on a histogram region.

Assuming that vuescan saves regular tiffs in gamma encoded mode, the results above confirm what the vuescan documentation says at the Harmrick website, which is that when you save an 8 bit file in raw mode it actually saves in gamma encoded mode. This kind of throws the comparisons for a loop, by which I mean that you can't compare a 16 bit raw to an 8 bit raw because vuescan's 8 bit raw saves aren't really raw. They are gamma encoded.

 

[alanrockwood]

So saving on storage of 1.28tb for 10,000 scans is the only advantage of scanning in 8 bit? Doesn;t seem worth the time. First off I don;t have 10,000 scans. But even so, what's the cost of that storage? $20? What happens if someone develops a better program in the future that needs 16 bits?

It just seems like a lot of effort for little gain if any. It's an interesting discussion from a technical standpoint. But from a practical one, not so much.

Correct, there is nothing wrong with saving in 16 bit mode, if your system lets you do that, and most (but not all) systems do let you do that. And if storage is not problem then saving in 16 bit is not problem. On the other hand, if the scans have noise in them (including grain) there is nothing to be gained by saving in 16 bit mode either.

So in a sense, it's a wash, that is unless a person can produce scans that are grain-free and noise-free, and in that case 16 bit is and advantage.

 

[Adrian Bacon]

I did not save as raw. I saved as TIFF without checking the RAW option. TIFF profile is unchecked, which I assume means that there is no profile applied to the TIFF image. I did not lock exposure. However, as can be seen in the scan of the full image, it includes full light and full dark regions, so if there is an automatic exposure taking place it should have scaled it to get maximum dynamic range to capture both ends of the exposure range.

I will try locking exposure and locking image color. I will also set the RAW mode output. In fact, I will set it to save in both TIFF and RAW. I will also set the white points at 1 and the black points at 0.

that’s what I put my settings at. Lock color, lock exposure, set the white and black points, and set the exposure and analog gains. If you don’t do that vuescan will change those settings to what it thinks is best every time you scan.

I don’t know what the set profile checkbox does, but I never set it and if outputting to standard tiff in full color, the samples are always conformed to the output color space. It’s possible that the checkbox just tells vuescan to embed the ICC profile, because when I open the resultant tiff file in photoshop it pops up and asks me what profile it should use.

also, the Epson does actually have two sets of lenses, and two sets of sensors with different resolutions, it’s entirely possible that each sensor has its own ADC. It might be useful to see if there’s a performance difference between the two. My raw samples where on the 6400 dpi sensor at the native sensor resolution. You could also potentially have different performance characteristics depending on the resolution you’re capturing at.

per your comment about having ADCs for each bit depth, many newer digital cameras capture 14 bit when taking stills with a shutter and 12 bit when using electronic shutter and capturing video (and different DR for each mode), given the price point of the Epson, why wouldn’t it have two different ADCs? Keep in mind that one of its core features is digitizing documents where 8 bits is more than enough for paper.

 

[alanrockwood]

...per your comment about having ADCs for each bit depth, many newer digital cameras capture 14 bit when taking stills with a shutter and 12 bit when using electronic shutter and capturing video (and different DR for each mode), given the price point of the Epson, why wouldn’t it have two different ADCs? Keep in mind that one of its core features is digitizing documents where 8 bits is more than enough for paper.

The only reason I can think of for including both an 8 bit and 16 bit ADC is if the 16 bit ADC were too slow to keep up under certain operating conditions. Otherwise, a 16 bit ADC can be used as if it were an 8 bit ADC just by truncating the lower order bits (or alternatively, by rounding to 8 bits). It's just an extra complication and expense that would seem to serve no useful purpose, other than possibly speed. I doubt if a scanner, even one using 1999 technology, would have a 16 bit digitizer that would be speed limiting.

Have you looked into the issue I mentioned that vuescan does not save 8 bit scans in linear form even if RAW is specified? Both the Hamrick documentation and my sleuthing indicate that even if you specify RAW it saves 8 bit scans in gamma encoded form. This would make it difficult to compare a 16 bit RAW scan with an 8 bit scan. At least on the software I usually use it would not be easy (Photoline). I do have Photoshop, but I seldom use it for personal reasons, so I don't know how easy it is to compare two histograms, one in 16 bit linear and one in 8 bit gamma, using Photoshop.

My suggestion is to save in 16 bit TIFF and 8 bit TIFF. That way the comparison would be "apples to apples" in terms of the data format, other than the bit depth that is.

By the way, I did some scans setting the parameters as you suggested. I may get around to posting the the results eventually, but the short answer is that it didn't significantly change anything compared to the settings I used before, except for that issue that 8 bit RAW does not save the data in linear form.

 

[Adrian Bacon]

My suggestion is to save in 16 bit TIFF and 8 bit TIFF. That way the comparison would be "apples to apples" in terms of the data format, other than the bit depth that is.

I don't recommend doing that in any way shape or form. If you're not saving raw, what happens is:

raw sample -> scale to 1.0-0.0 -> 3x3 matrix -> XYZ connection space -> 3x3 matrix -> output color space -> output gamma encoding -> scale to output bit depth -> save

The first 3x3 matrix converts the raw samples to XYZ coordinates in the CIE 1931 color space, the second 3x3 matrix converts the XYZ coordinates to the RGB values for the output color space (i.e. sRGB, ProPhoto, Adobe RGB, etc). There is nothing linear about those operations and I can't think of any good reason to try to characterize ADC noise levels on samples that have been run through those operations first. All it's going to do is muddy the waters, and that's if you're just doing a naive conversion with no real color management. Often times, there's a whole color model in there that tries to do a perceptual transform so the hues, saturation, and perceived brightness matches between the various color spaces. Many color engines allow you to select perceptual, relative, or absolute in the rendering intent, and it's not clear what Vuescan does, so in short, that's a terrible idea and better to just avoid if possible.

Have you looked into the issue I mentioned that vuescan does not save 8 bit scans in linear form even if RAW is specified? Both the Hamrick documentation and my sleuthing indicate that even if you specify RAW it saves 8 bit scans in gamma encoded form. This would make it difficult to compare a 16 bit RAW scan with an 8 bit scan. At least on the software I usually use it would not be easy (Photoline). I do have Photoshop, but I seldom use it for personal reasons, so I don't know how easy it is to compare two histograms, one in 16 bit linear and one in 8 bit gamma, using Photoshop.

I assumed you were going to. if the samples are raw and the only operation being performed is gamma encoding (which technically doesn't make them raw at that point, but whatever... raw means not conformed to a color space, i.e. the operations I described above), then either apply the reverse on the 8 bit samples to re-linearize them, or apply the same gamma on the 16 bit samples. If I'm acquiring 8 bits and saving 8 bit raw, applying a gamma is a dumb thing to do and frankly I'd consider that to be a bug in Vuescan if it actually does that because all it does is eat up a whole pile of least order fidelity.

 

[alanrockwood]

I don't recommend doing that in any way shape or form. If you're not saving raw, what happens is:

raw sample -> scale to 1.0-0.0 -> 3x3 matrix -> XYZ connection space -> 3x3 matrix -> output color space -> output gamma encoding -> scale to output bit depth -> save

The first 3x3 matrix converts the raw samples to XYZ coordinates in the CIE 1931 color space, the second 3x3 matrix converts the XYZ coordinates to the RGB values for the output color space (i.e. sRGB, ProPhoto, Adobe RGB, etc). There is nothing linear about those operations and I can't think of any good reason to try to characterize ADC noise levels on samples that have been run through those operations first. All it's going to do is muddy the waters, and that's if you're just doing a naive conversion with no real color management. Often times, there's a whole color model in there that tries to do a perceptual transform so the hues, saturation, and perceived brightness matches between the various color spaces. Many color engines allow you to select perceptual, relative, or absolute in the rendering intent, and it's not clear what Vuescan does, so in short, that's a terrible idea and better to just avoid if possible.



I assumed you were going to. if the samples are raw and the only operation being performed is gamma encoding (which technically doesn't make them raw at that point, but whatever... raw means not conformed to a color space, i.e. the operations I described above), then either apply the reverse on the 8 bit samples to re-linearize them, or apply the same gamma on the 16 bit samples. If I'm acquiring 8 bits and saving 8 bit raw, applying a gamma is a dumb thing to do and frankly I'd consider that to be a bug in Vuescan if it actually does that because all it does is eat up a whole pile of least order fidelity.

I am not able to reverse the 8 bit gamma encoding to create a linear file. I also don't have a way to convert a RAW 16 bit file to a gamma encoded 16 bit file.

Therefore, things seem to be in a bit of an impasse, since a true RAW file can't be directly compared to one that is gamma encoded. If you are able to convert a gamma encoded 8 bit file to a linear one (or convert a RAW 16 bit file to a gamma encoded file) then that would enable a comparison to be made. Otherwise comparisons can't be made, and we are just spinning our wheels.

As far as color profiles are concerned, I strongly suspect that because I did not check the box "TIFF Profile" in the output window a color profile was not applied to my results, especially because all of my work on this problem has been in gray scale. (I don't see why a color profile would be applied to a gray scale image, but maybe I am missing something.)

I agree that gamma encoding an 8 bit "RAW" file is a dumb idea, and confusing as well. I wonder if this whole issue could be bypassed by using Silverfast. Unfortunately, I am not competent with that program, and in any case the version of Silverfast that come with my V750 won't run under Windows 10.

One more thing, it is already implicit in our discussions so far, but it's worth explicitly noting that I saved my files in Grayscale, and I think yours were save in color, so that is another point of difference between what you are doing and what I am doing.

Actually, despite the apparent impasse due to incompatible file types, I think this discussion is making progress, but I don't see much in the way of future progress without breaking through the impasse.

Also, if you look at the last set of gray scale images I posted you will see that when I took a file saved in 16 bits and converted it to 8 bits within Photoline the results were practically the same as the file I saved in 8 bits, so I think that gets around some of the issues you raised with regard to gamma encoding, color profiles, etc. Any thoughts on that?

 

[alanrockwood]

Here's what the vuescan documentation says about the TIFF profile setting in the output tab.

"Output | TIFF profile
This specifies whether to embed an ICC color profile into the TIFF file. This is primarily useful if you’re using Photoshop(TM). You can specify the profile to use by setting Color | Output color space.

Professional Option: This option is displayed when Output | TIFF file is set."

So by not activating TIFF profile one can avoid having a color profile applied to the file. That eliminates one possible source of variability that concern's Adrian.

 

[Adrian Bacon]

As far as color profiles are concerned, I strongly suspect that because I did not check the box "TIFF Profile" in the output window a color profile was not applied to my results, especially because all of my work on this problem has been in gray scale. (I don't see why a color profile would be applied to a gray scale image, but maybe I am missing something.)

Typically for greyscale there would be something like 20% dot gain applied or something similar. It's not clear what Vuescan does for grayscale, and there's no settings for that. If the gamma of the color profile is being applied to an 8 bit grayscale image, that's an even more egregious bug on the part of Vuescan.

Therefore, things seem to be in a bit of an impasse, since a true RAW file can't be directly compared to one that is gamma encoded. If you are able to convert a gamma encoded 8 bit file to a linear one (or convert a RAW 16 bit file to a gamma encoded file) then that would enable a comparison to be made. Otherwise comparisons can't be made, and we are just spinning our wheels.

I can write some code to do it, but before I do that, I'd rather explicitly ask Hamrick what the behavior is if acquiring at 8 bits and saving 8 bit raw, for both color and if doing single channel grayscale. If it really is applying a gamma or dot gain to the 8 bit output if the input is also 8 bits, I'd be requesting that he fix that as that's just dumb and negates the point of saving raw.

 

[Adrian Bacon]

So by not activating TIFF profile one can avoid having a color profile applied to the file.

Not quite. I read that as "embed an ICC profile or not", not "apply a color profile to the samples or not". If that controls whether or not the samples are conformed to a color profile, then what's the difference between that and telling Vuescan to save as raw? If the samples haven't been conformed to any profile, then aren't they raw? I just don't see that as controlling the behavior of what Vuescan does with the samples and instead see it as controlling whether to embed an ICC profile into the file.

Whether or not you copy an ICC profile into a tiff chunk in the file has very little to do with whether or not the samples have been conformed to that profile. I never select output tiff profile and the samples as saved do conform to my color profile (if not saving raw) as all I have to do in photoshop is "apply profile" (not "convert to profile") and the colors are then correct. In photoshop, "apply profile" tells photoshop that the samples in the file conforms to the color profile you're applying, so interpret it that way. That lets you deal with a file that doesn't already have an embedded ICC profile, or has the wrong ICC profile embedded into it (which does actually happen a lot), then when you save the tiff file in photoshop it replaces the ICC profile with the one you applied, thus allowing correct rendering the next time you open the file.

 

[Adrian Bacon]

I've reached out to Hamrick and explicitly asked what the behavior is if acquiring at 16 bits and saving 16 bit raw, acquiring at 8 bits and saving at 8 bit raw, and if the behavior is any different between acquiring full color and acquiring monochrome from one channel. I'm awaiting a response and will post it here when I get it.

In the meantime, just to do a sanity check, I captured 8 bit color and saved out to just a standard tiff file with the exact same settings as the raw with the only difference being saving as a tiff instead of a raw tiff.

http://m.avcdn.com/sfl/epson_v850_raw_black_frames_v2.zip

The same two raw files as before plus an 8 bit color tif file of the same capture area with same blacked out configuration as before. If the raw 8 bit file has any gamma encoding, it should be pretty close if not exact to the 8 bit tif file that has the color space and gamma applied to it.

The same simple analysis as before:

The per color channel numbers:

raw0001.tif (captured in 16 bits, saved in 16 bits)
resolution: 9064x6046 px
bits per sample: 16 bits
analyze window_size: +-1024, x1: 3508, x2: 5556, y1: 1999, y2: 4047
max_r:1817, max_g:1911, max_b:1714
min_r:0, min_g:0, min_b:0
avg_r:334, avg_g:350, avg_b:313

raw0002.tif (captured in 8 bits, saved to 8 bits, scaled to 16 bits for analysis)
resolution: 9064x6046 px
bits per sample: 16 bits
analyze window_size: +-1024, x1: 3508, x2: 5556, y1: 1999, y2: 4047
max_r:59904, max_g:53248, max_b:32639
min_r:0, min_g:0, min_b:0
avg_r:1699, avg_g:1666, avg_b:1644

2022-03-26-0003.tif
resolution: 9064x6046 px
bits per sample: 16 bits
analyze window_size: +-1024, x1: 3508, x2: 5556, y1: 1999, y2: 4047
max_r:19789, max_g:64384, max_b:32639
min_r:0, min_g:0, min_b:0
avg_r:3236, avg_g:3297, avg_b:3052

I had my output color space set to ProPhoto (with a 1.8 gamma). They're not even close, which would seem to indicate that the raw sample doesn't actually have any gamma encoding applied to it.

When I get the time I'll poke around to see if there's any difference when capturing single channel and saving grayscale.

@alanrockwood The only things I can think of that would cause what you're seeing with your scans is: 1. the monochrome behavior is different. 2. photo line is doing something you're not aware of. 3. The v850 has better electronics than the v750. 4. There's something in the Vuescan settings your not aware of. 5. any combination thereof.

 

[alanrockwood]

It will be interesting to see what he says.

For 8 bit saves I have been acquiring in 16 bit and saving in 8 bit.

Interesting comment about V850 electronics being better than V750 electronics. Can you document that? I have done extensive searches for any differences between those two models over a period of years, and the only difference I have seen mentioned is that the two scanners use a different light sources.

 

[Adrian Bacon]

For 8 bit saves I have been acquiring in 16 bit and saving in 8 bit.

Let me get this straight. You're on here claiming that 8 bit mode is just as good as 16 bit mode and saying you can see no difference between 8 and 16 bits, and making claims that 8 bits is all you need, but you're acquiring in 16 bits for everything? And you wonder why you're not seeing much of a difference between the two? SERIOUSLY!?!?!?!

I'm sorry, but if you're trying to be credible here, that's a heck of a great way to shoot yourself in the foot. I've already shown that capturing in 16 bit mode is superior to 8 bit mode in terms of DR and overall noise performance, regardless of how it's saved. I've also showed that acquiring in 16 bit and saving in 8 bit mode is better than acquiring and saving in 8 bit mode. I'm having trouble seeing much of a reason to continue putting effort into this conversation. Color me bent out of shape.

 

[alanrockwood]

Let me get this straight. You're on here claiming that 8 bit mode is just as good as 16 bit mode and saying you can see no difference between 8 and 16 bits, and making claims that 8 bits is all you need, but you're acquiring in 16 bits for everything? And you wonder why you're not seeing much of a difference between the two? SERIOUSLY!?!?!?!

Please review what the input and output parameters mean in Vuescan. The input parameter is the number of bits that Vuescan receives from the scanner. The ouput parameter is the number of bits that Vuescan writes to disc. I consider it an 8 bit a scan if the data written to disk is 8 bit data. If you don't agree with that terminology then what would you call it when the data is stored in 8 bits?

I never advocated setting the input parameter in vuescan to be 8 bit if 16 bit input is available. However, if the scans were to be saved in linear format then it should not matter if you set the input parameter to be 8 bits or 16 bits. The results should be the same, except possibly how the scanner handles the conversion from 16 bits to 8 bits (e.g. rounding vs. truncation of the low order bits).

On the other hand, if the results are saved in 8 bit gamma format then there can be a slight difference between acquiring in 16 bits vs. acquiring in 8 bits because of roundoff error in the intermediate steps in the conversion process. Roundoff error affects the two types of calculation slightly differently, i.e. gamma encoding 16 bits and then converting to 8 bits vs. gamma encoding starting with 8 bits. However, the differences will not be huge. My scans confirm this, i.e. there is little difference between setting the input parameter to 8 bit vs. 16 bit if you are saving in 8 bit. I already showed this in post #152, and I just did it again on my V750 scanner.

 

[alanrockwood]

Let me get this straight. You're on here claiming that 8 bit mode is just as good as 16 bit mode and saying you can see no difference between 8 and 16 bits, and making claims that 8 bits is all you need, but you're acquiring in 16 bits for everything? And you wonder why you're not seeing much of a difference between the two? SERIOUSLY!?!?!?!

I'm sorry, but if you're trying to be credible here, that's a heck of a great way to shoot yourself in the foot. I've already shown that capturing in 16 bit mode is superior to 8 bit mode in terms of DR and overall noise performance, regardless of how it's saved. I've also showed that acquiring in 16 bit and saving in 8 bit mode is better than acquiring and saving in 8 bit mode. I'm having trouble seeing much of a reason to continue putting effort into this conversation. Color me bent out of shape.

As far as showing that acquiring in 16 bits is superior to acquiring in 8 bits, you have not yet done that for reasons already discussed, i.e. your 8 bit scans are not directly comparable to your 16 bit scans because your 8 bit "raw" scan are actually gamma encoded and your 16 bit raw scans are linear. This makes the two types of acquisition not directly comparable in histograms without appropriate software to do the conversions to comparable data formats.

 

[alanrockwood]

Returning to the fact that I did my scans in gray mode, I set my input in gray mode and the output in gray mode, and I disabled TIFF profile on output. Is it your position that vuescan forces a color profile on data that is set to gray scale on input and gray scale on output with TIFF profile turned off on output? I don't understand why vuescan would force a color profile on an image that contains no color anywhere in the data handling chain. Can you explain the purpose of forcing a color profile onto gray scale images, and can you cite any documentation from vuescan that it actually does force a color profile onto gray scale images?

 

[Adrian Bacon]

I never advocated setting the input parameter in vuescan to be 8 bit if 16 bit input is available

Oh please. You should have disclosed that you were acquiring in 16 bit mode and saving 8 bit files from the very start.

Please review what the input and output parameters mean in Vuescan. The input parameter is the number of bits that Vuescan receives from the scanner. The ouput parameter is the number of bits that Vuescan writes to disc. I consider it an 8 bit a scan if the data written to disk is 8 bit data. If you don't agree with that terminology then what would you call it when the data is stored in 8 bits?

I've been a Vuescan user since the early days of Vuescan, likely way longer than you. I'm well aware of the input and output options. Writing an 8 bit file is exactly that. The source could have been any number of bits, and if you're acquiring at a different number of bits than what you're writing you should disclose it, especially if you're having a discussion about the dynamic range differences between 16 bits and 8 bits. It's disingenuous to call an 8 bit tif file an 8 bit scan if you acquired in 16 bits. How it's stored and how it was acquired are not the same thing and never have been. If you acquired at 16 bits and saved at 8 bits then you say it's a 16 bit scan saved as an 8 bit file. If you acquired in 16 bits and saved as 16 bits, then you say it's a 16 bit scan. If you acquired as 8 bits and saved at 8 bits then you say it's an 8 bit scan. If you acquired in 8 bits and saved in 16 bits then you say it's an 8 bit scan saved as 16 bits, not a 16 bit scan like you just inferred with your 8 bit logic you just said. You're dancing around the fact that you should have disclosed that information way sooner than you did but didn't.

Leaving now. I'll update when I hear from Hamrick, but otherwise I'll leave you to continue your nonsense without me. See you around.

 

[alanrockwood]

The question has come up why I didn't specifically specify a scanning workflow where the input tab in vuescan was specified as 16 bits and the output was specified as 8 bits. However, if the output of a scan is 8 bits then I consider it an 8 bit scan. I don't know what else one would call it other than an 8 bit scan.

However, in the context of my earlier posts it seems self evident to me that it would be a (slightly) preferred acquisition scheme to acquire in 16 bit and save in 8 bit rather than specifying 8 bits for the input tab and 8 bits for tab. For example, in post #150, which is part of my discussion for using only a single ADC in a scanner (e.g. 16 bits) regardless of whether one ultimately wants an 8 bit results I said the following "...if the engineer wants to have the option for an 8 bit output then any signal processing done in the scanner it will be done in 16 bit mode, preferably using floating point arithmetic (or possibly some integer bit depth greater than 16 bit), and only converted to 8 bit mode for the final output. Otherwise the programmer would have to program two complete and separate signal processing paths when that is neither necessary nor desirable."

This statement is based on how a good engineer would design an instrument, but it is also based on a well-known principle that in a calculation it is best to do any rounding off at the very last stage of the process, keeping more significant figures during the intermediate stages of a calculation. This reduces the overall error budget in the calculation. This is also evident in my comment to Adrian when I lauded his use of floating point for intermediate calculations.

Also in post #152 I pointed out by experimental results that as a practical matter this typically doesn't make much difference. In particular, I pointed out that scans #1 and #2, which were acquired in 8 bit mode and saved in 8 bit mode, were very similar to scan #3, which was acquired in 16 bit mode and saved in 8 bit mode.

I apologize if I didn't make this clear enough before. Perhaps I should have been more explicit in pointing out that if you are saving 8 bit scans then the preferred workflow is to acquire the data in 16 bit mode (specified in the input screen in vuescan) and saving in 8 bit mode (specified in the vuescan output screen), but as a practical matter it doesn't really make much difference.

As I mentioned above, I thought this would be self evident, especially considering the context of my other writings in this thread, but I guess it was not.

Anyway, if a scan is saved as an 8 bit file then I call it an 8 bit scan, and I think most other people would also call it an 8 bit scan. I really don't know what else it would be called.

Also, gamma encoding isn't actually much of an issue. It can be shown that the following two workflows do not produce identical results, but the differences between them are small. Workflow 1: digitize in 16 bit mode. Do the gamma transformation. Round off to nearest integer. Convert to 8 bits. Round to nearest integer. This produces a gamma encoded result by one algorithm. Workflow 2: digitize in 16 bit mode. divide by 257 to produce a generally non-integer intermediate result. Round to nearest integer to produce an 8 bit result. (At this point it is the same as if you simply digitized with an 8 bit ADC, so if one actually had an 8 bit digitizer in the instrument one would start here.) Apply the gamma transformation. Round off to nearest integer. This produces a gamma encoded result by a different algorithm. The two results are not identical, but they aren't much different either. Here's a plot of the results of the two different workflows overlaid on the same plot, one it black and the other in red.

The difference becomes even less if one then converts back to linear signal space. Here's a plot of the results of the two workflows overlaid on the same graph, one in black and the other in blue.

I think most people can agree that based on the graphs, the two workflows produce very similar results. Therefore, although it is true that if one wants to save 8 bit scans then the best workflow is to acquire in 16 bit and save in 8 bit (provided the option is available), and this is the workflow I used, the end result won't be much different if one simply acquires in 8 bit and saves in 8 bit.

One slight clarification about the figures in this post. The x axis is labeled as "ramp (255 arbitrary units)." This means that the numbers run from zero to 255, not that there were 255 separate values in the ramp. The ramp has 65535 separate values, so it is quasi-continuous.