comparison of results digital processing 1896 film (startling)

[jtk]

https://www.thisiscolossal.com/2020/02/ai-upgrades-classic-film/?mc_cid=61f655f1ac&mc_eid=6822972151

 

[pbromaghin]

Beautiful video. Thanks for posting.

 

[jtk]

Be sure to explore this article's internal links.. Topaz ai Gigapixel might be hammer in coffin for MF/LF
..it's so cheap even I might buy...have not yet tried free sample with high detail crowd scene... terrifying!

 

[PhilBurton]

Be sure to explore this article's internal links.. Topaz ai Gigapixel might be hammer in coffin for MF/LF
..it's so cheap even I might buy...have not yet tried free sample with high detail crowd scene... terrifying!

Does anyone know how much processing power was required to do this enhancement? What kind of machines? How much time?

 

[jlbruyelle]

Does anyone know how much processing power was required to do this enhancement? What kind of machines? How much time?

You can find out yourself, by downloading and using the trial version of the software that was used to make this: https://topazlabs.com/gigapixel-ai/. It can run on a ordinary PC, and since this is not a real-time process you don't need a very powerful PC, it's just a trade-off between the power of the machine and the processing time. Since there is no real programming involved (you just use a pre-trained neural network) , it is really not difficult to do.

Please note - and this is a very important point that most people have overlooked completely - that the original sequence is not at all the one shown in the OP, so the comparison is bogus. According to his own description, Shiryaev ran the software on a far better copy of the original film (available there), so the only achievement here is 2x upscaling and frame rate conversion, which is routinely achived by specialized chips in your TV set, in real time and with very comparable results and artefacts. The viral spread of Shiryaev's video was probably mostly a result of the AI buzzword, given that similar techniques, AI-based or not, have been in use for several years now.

It is also noteworthy that, contrary to the pitch on Colossal, there was no need to upscale a 720p youtube video to produce a 4K version of this film, since Institut Lumière had already scanned the original footage in 4K (a 720p version of this scan is available on Youtube there, a HD version is available on bluray). Let's say that the main interest (in addition to advertising a piece of upscaling software) is to offer an opportunity to compare the actual high-resolution image with a "reconstruction" of it by AI - I use quotes because it is not an actual reconstruction, what the software really does is make up high-resolution details that look realistic but are not necessarily real.

In short, - sorry jtk - you cannot really replace your LF camera with a smartphone and Gigapixel AI, you can only expect a twofold increase in perceived resolution before you start seeing strange artefacts.

As final note, I have found a youtube video giving a short overview of how this super-resolution AI stuff is done. You may find it interesting, it's there.

 

[StepheKoontz]

Wow, this is why the internet can't be used at face value.

Someone takes a frame capture, totally blows out the highlights using levels (or a horrible scan) and uses it as a "before" to make this look like it did some sort of super magic, recovering lost information. If you look at the actual before and after in the video, I actually think the before version shows more details. As you can see, the difference between the source and upscaled is nothing like the extreme falsely created difference in the images at the top of the article as shared in the OP. I will say the addition of the sound track does more for the video than any of the visual improvements.

 

[Cholentpot]

Be sure to explore this article's internal links.. Topaz ai Gigapixel might be hammer in coffin for MF/LF
..it's so cheap even I might buy...have not yet tried free sample with high detail crowd scene... terrifying!

I've seen you grinding this ax in other threads for a while now. I don't understand, you're under the impression that film of any sort was still viable as a commercial option?

I'm sure there are some corporations that still use typewriters and some CEOs that write memos with a fountain pen but for all practical purposes anything not written on a word processor is extinct. I would have thought this of film at least a decade ago if not more. Who out there that is using photography as a means for making a day to day living is using exclusively film? Yes, I'm sure there are a few people left but for the most part the world has shifted to digital. Unless you're living under a rock or something.

I'm shooting film for the pleasure of it. No new digital process is going to make me stop using film. Maybe if someone came out with a swapable 35mm full frame sensor that I can load into my cameras across platforms. I'm not sure where you're going with this MF/LF is dead.

It already is dead. I can get 25 shots for $250 before processing and scanning? What a great way to make a buck. Oh, you're not in it for the money? Then it's not a living any more and it's now art or a hobby. Great. No problem with that.

I genuinely don't understand the reason for these threads. We're in 2020, this stuff sounds like it's out of 2007.

 

[dutchsteammachine]

I have done frame interpolations for about a week now, messing around mostly with NASA footage. Here are some results.

(No Upscaling though, I hate that...)

 

[StepheKoontz]

I have done frame interpolations for about a week now, messing around mostly with NASA footage. Here are some results.

(No Upscaling though, I hate that...)

Doing frame interpolations to very low frame rate, choppy looking video makes a lot of sense, and it does make it more enjoyable to watch. The OP showing an image clearly edited (or very poorly scanned) to have massively blown out highlights, next to one that doesn't as "proof" that his pet software is amazing is disingenuous at best.

 

[NortheastPhotographic]

Be sure to explore this article's internal links.. Topaz ai Gigapixel might be hammer in coffin for MF/LF
..it's so cheap even I might buy...have not yet tried free sample with high detail crowd scene... terrifying!

Medium and large format are not in a coffin and no one would stop shooting an 8x10 camera because some plug in can upres a little better than what PS can natively do.

 

[Cholentpot]

Medium and large format are not in a coffin and no one would stop shooting an 8x10 camera because some plug in can upres a little better than what PS can natively do.

I'd like to add.

I don't think anyone is shooting film these days for the resolution or sharpness or any of these things. They're shooting for the joy of shooting, or maybe for a look that is not easily replicated.

Film is as dead as writing a quarterly report in ball point pen. You can do it, but why? However, I'm not sending a letter to my Grandmother that's typed up and printed. I'm going to sit down with pen and ink to write it out, and no matter how good a processor gets it won't take that pen and ink away from me.

 

[removed account4]

i like restoration imagery, but while it is nice to see it done, and a great way to preserve historic photographs and moving picture footage
to me it looks a bit unreal. as nice as it is, there is something about it that my eyes don't like.
if david lynch did though, it would be a different story (from lumiere and company done with the brother's original camera &c)

 

[dutchsteammachine]

Doing frame interpolations to very low frame rate, choppy looking video makes a lot of sense, and it does make it more enjoyable to watch. The OP showing an image clearly edited (or very poorly scanned) to have massively blown out highlights, next to one that doesn't as "proof" that his pet software is amazing is disingenuous at best.

This is why I always try to find the highest possible resolution scan.

Most of the time, a scan where you are able to clearly resolve grain has a higher resolution than some faux amalgamation using a low-resolution source.

Anyways, I have tried out these upscale programs on some of my old photos and films from the early 1900s, and it mostly just gets confused by the grain. Downscaling it to avoid the grain usually results in a lesser-quality upscale than just using the full-resolution scan.

I can see upscaling work out if all originals are lost, and only a 360/720p scan is known to exist. But it needs to be really high bitrate, as compression artefacts also confuse it...

The AI I use to interpolate frames can handle grain really well actually, it acts as a kind of filter to lessen film-grain also, and as a weak stabilizer.

 

[jlbruyelle]

Doing frame interpolations to very low frame rate, choppy looking video makes a lot of sense, and it does make it more enjoyable to watch.

Good job, it certainly looks good on the moon footage. I'd say this is mostly owing to the facts that unlike Sheryaev you limit interpolation to sensible values and simple cases, with motion that remains comparatively slow and mostly constant over the image. In fact I don't see a big difference between the original and interpolated footages, and I'm perfectly happy with the former, but maybe it's just me.

OTOH the result is completely different with 60 fps and/or slightly more difficult cases such as spinning wheels or rapid walk, such as this NYC footage also processed by Sheryaev. The wheels seem to be slightly moving back and forth instead of spinning (especially visible 1mn20 into the video) while people seem to regularly stop in mid-air for a frame or two, which is the usual behaviour of interpolation algorithms - and one of the main reasons why I don't accept it. I find all this painful to watch, and I utterly fail to understand the YT comments insisting that 60 fps is "closer to life" than the original 16-20 fps. It's not any closer to life, it's just a lame, and very noticeably failed, attempt to restore temporal information that was lost forever in the process of filming the scene.

Incidentally, as a reply to jnantz, I'm glad to see that there are at least two of us that don't like this sort of treatment. Trying to "cure" low frame rate (or low resolution for that matter) is by no means restoration anyway, you will make any restorer scream if you say this in front of them. When you know how it's done you understand that it's not even a proper "reconstruction" of the original scene, just a way to make the images more attractive. Some documentary producers openly admit that they use it just as a way to make sales easier.

The OP showing an image clearly edited (or very poorly scanned) to have massively blown out highlights, next to one that doesn't as "proof" that his pet software is amazing is disingenuous at best.

To be fair, the switch-over between the actual and ridiculously poor "original" images was not jtk's fact, but Colossal's. Jtk's fault, and many others' as well, was to not check the sources that Sheryaev himself gave along with his demo video.

 

[dutchsteammachine]

Good job, it certainly looks good on the moon footage. I'd say this is mostly owing to the facts that unlike Sheryaev you limit interpolation to sensible values and simple cases, with motion that remains comparatively slow and mostly constant over the image. In fact I don't see a big difference between the original and interpolated footages, and I'm perfectly happy with the former, but maybe it's just me.

OTOH the result is completely different with 60 fps and/or slightly more difficult cases such as spinning wheels or rapid walk, such as this NYC footage also processed by Sheryaev. The wheels seem to be slightly moving back and forth instead of spinning (especially visible 1mn20 into the video) while people seem to regularly stop in mid-air for a frame or two, which is the usual behaviour of interpolation algorithms - and one of the main reasons why I don't accept it. I find all this painful to watch, and I utterly fail to understand the YT comments insisting that 60 fps is "closer to life" than the original 16-20 fps. It's not any closer to life, it's just a lame, and very noticeably failed, attempt to restore temporal information that was lost forever in the process of filming the scene.

Incidentally, as a reply to jnantz, I'm glad to see that there are at least two of us that don't like this sort of treatment. Trying to "cure" low frame rate (or low resolution for that matter) is by no means restoration anyway, you will make any restorer scream if you say this in front of them. When you know how it's done you understand that it's not even a proper "reconstruction" of the original scene, just a way to make the images more attractive. Some documentary producers openly admit that they use it just as a way to make sales easier.



To be fair, the switch-over between the actual and ridiculously poor "original" images was not jtk's fact, but Colossal's. Jtk's fault, and many others' as well, was to not check the sources that Sheryaev himself gave along with his demo video.

Yes, it depends a lot on the shooting rate of the source and the type of scene. Sometimes 4x interpolation looks fine, sometimes even just 2x messes up.
Given most Apollo footage was shot in 12fps, 2x brings it up to 24 which is a huge difference in my eyes. Most other footage is 6fps or 24fps. The former needs 4x interpolation, haven't had a hand at that footage yet. The latter is already at 24fps, so really doesn't need it in my opinion.

I think frame interpolation looks good most of the time when done at reasonable rates. What I can't stand are the ones with AI-added colour. They look absolutely horrendous.
I am fine with audio too, it really helps me imagine being among those people... I can't imagine how much time all the mixing takes.

The problem I have seen with a lot of attempts at AI-upscaling is that they use a crappy low-resolution source when a lot of these films can be found online or at archives scanned at 4K... so I don't see the point.
Again, AI-upscaling doesn't handle grain well. The times I tried it, mostly just gives everything a fuzzy spiderweb-texture.

 

[jlbruyelle]

I am fine with audio too, it really helps me imagine being among those people... I can't imagine how much time all the mixing takes.

Not much, really. What they did is just adding a couple of library sounds, quite often the same sound repeatedly (such as the horn on the New York sequence), overlooking such basic things as distance, reflections, relative levels or synchronism. Even such obvious elements as the opening carriage doors in the La Ciotat sequence are utterly missing. This is very basic ambiance stuff done in an hour, tops. I have to admit I haven't been convinced by the sound any more than by the image.

OTOH some people do professionally this job of reconstituting the sound atmosphere of past cities, and the result can be very convincing indeed. See for instance https://lejournal.cnrs.fr/articles/ecoutez-le-paris-du-xviiie-siecle (google translate will help if you don't read French), and watch the video. There is no comparison, but of course these are very different amounts of work, and skill levels.

The problem I have seen with a lot of attempts at AI-upscaling is that they use a crappy low-resolution source when a lot of these films can be found online or at archives scanned at 4K... so I don't see the point.
Again, AI-upscaling doesn't handle grain well. The times I tried it, mostly just gives everything a fuzzy spiderweb-texture.

Yes, this is a common problem with neural networks. Whenever you try to give them something they were not trained for (for instance noise, of which grain is a good example), they still try to make up something and end up with false recognitions or just plain nonsense. This behaviour is called "hallucinating" (I especially like this term).

 

[mmerig]

I'd like to add.

I don't think anyone is shooting film these days for the resolution or sharpness or any of these things. They're shooting for the joy of shooting, or maybe for a look that is not easily replicated.

Film is as dead as writing a quarterly report in ball point pen. You can do it, but why? However, I'm not sending a letter to my Grandmother that's typed up and printed. I'm going to sit down with pen and ink to write it out, and no matter how good a processor gets it won't take that pen and ink away from me.

Please think again. I am using a 4 by 5 for a project (going on 5 years now) where authenticity, easy archiving, and resolution were primary objectives, and the sponsors were fine with film. Also, the orthochromatic look was important, so I used such film, as well as pan and color transparency. Film is a very small part of the total cost, as I am not taking hundreds or thousands of images and then sorting through them later. Each image, ideally, is a keeper. Nearly all have been, but I screw-up sometimes.

Lugging around a heavy pack, sometimes for many miles and days is the downside, although a medium format digital kit or scanning back would probably be comparable in weight. Some travel is on horses and mules, so having super expensive camera gear prone to damage packed on them is a risk. My camera did get damaged when blown over by a wind gust (and almost by a mule), and I was able to fix it with parts from the local hardware store and was up and running in a few days. Also, no batteries (I use a selenuim cell meter, or sunny-sixteen).
Low-tech is low-hassle.

 

[Cholentpot]

Please think again. I am using a 4 by 5 for a project (going on 5 years now) where authenticity, easy archiving, and resolution were primary objectives, and the sponsors were fine with film. Also, the orthochromatic look was important, so I used such film, as well as pan and color transparency. Film is a very small part of the total cost, as I am not taking hundreds or thousands of images and then sorting through them later. Each image, ideally, is a keeper. Nearly all have been, but I screw-up sometimes.

Lugging around a heavy pack, sometimes for many miles and days is the downside, although a medium format digital kit or scanning back would probably be comparable in weight. Some travel is on horses and mules, so having super expensive camera gear prone to damage packed on them is a risk. My camera did get damaged when blown over by a wind gust (and almost by a mule), and I was able to fix it with parts from the local hardware store and was up and running in a few days. Also, no batteries (I use a selenuim cell meter, or sunny-sixteen).
Low-tech is low-hassle.

Fine and dandy.

But you're riding a mule.

Again.

You're riding a mule.

Do you understand how fringe that is? Yes, someone out there is still using a pack animal. For the rest of us 99.9999999999% riding a mule is slightly more out there than taking a rocket to the moon. I don't think your statement bolsters your argument.

You're riding a mule and taking photos using a 4x5. Also, you don't have clients, you have sponsors. I'm taking photos of foreclosures, Billy's barmitzva, Chuck and Nancy's extended family and photos of Johnny's dog kennel. These are not sponsors, they are clients. I don't get to dictate a 'look' or a 'format' they want photos.

That being said, I'd love to wander the hinterlands riding a horse setting up camp and taking photos in the clear frosty morning and then tramping back into civilization once in a blue moon to give m'howdies and to re-supply and get 'ol Betsy re-shod. As it is by shooting film that puts me way out of line with my peers which is fine by me. But I don't think I'm showing up to Clara's engagement on a donkey. I may show up with a 4x5 but It'll have to be a very understanding client.

 

[mmerig]

Fine and dandy.

But you're riding a mule.

Again.

You're riding a mule.

Do you understand how fringe that is? Yes, someone out there is still using a pack animal. For the rest of us 99.9999999999% riding a mule is slightly more out there than taking a rocket to the moon. I don't think your statement bolsters your argument.

You're riding a mule and taking photos using a 4x5. Also, you don't have clients, you have sponsors. I'm taking photos of foreclosures, Billy's barmitzva, Chuck and Nancy's extended family and photos of Johnny's dog kennel. These are not sponsors, they are clients. I don't get to dictate a 'look' or a 'format' they want photos.

That being said, I'd love to wander the hinterlands riding a horse setting up camp and taking photos in the clear frosty morning and then tramping back into civilization once in a blue moon to give m'howdies and to re-supply and get 'ol Betsy re-shod. As it is by shooting film that puts me way out of line with my peers which is fine by me. But I don't think I'm showing up to Clara's engagement on a donkey. I may show up with a 4x5 but It'll have to be a very understanding client.

Sure it's fringe. But you said "I don't think anyone is shooting film these days. . .". It only takes one person to refute that argument, and there are probably others besides me on the fringe. "Hardly anyone. . ." is a more accurate way to put it. Given the objectives of the project, it made sense to use what I am using, just as it does to use digital for documenting foreclosures etc.

By the way, I don't own the mules, and it's not an unusual preference for trail riders. Mules are smart, sure-footed, and can have a smoother gate than horses. The mule use is just a travel convenience, and I mentioned it as part of the equipment risk I am dealing with.

 

[Cholentpot]

Sure it's fringe. But you said "I don't think anyone is shooting film these days. . .". It only takes one person to refute that argument, and there are probably others besides me on the fringe. "Hardly anyone. . ." is a more accurate way to put it. Given the objectives of the project, it made sense to use what I am using, just as it does to use digital for documenting foreclosures etc.

By the way, I don't own the mules, and it's not an unusual preference for trail riders. Mules are smart, sure-footed, and can have a smoother gate than horses. The mule use is just a travel convenience, and I mentioned it as part of the equipment risk I am dealing with.

I mean if you're going to nitpick and be like that. I DID say that it is also used to the specific look, and you did day you shoot for the orthochromatic look. So if we're playing this game you're not shooting 4x5 strictly for the resolution. Therefore my statement was correct.

Thank you, I'll let myself out.

Give the jackasses a carrot for me.

 

[mmerig]

Sorry, I did not mean to offend you.

 

[Cholentpot]

Sorry, I did not mean to offend you.

Me? Offended? Never. 'tis the internet.

What's the mules name?

 

[mmerig]

The riding mule's name is Missy. (we actually had two mules named Missy -- a little confusing at times). Apple cores are just as good as a carrot.

 

[Cholentpot]

The riding mule's name is Missy. (we actually had two mules named Missy -- a little confusing at times). Apple cores are just as good as a carrot.

I had a mule for a week. It's name was Mo. We fed it everything we could find. Too bad I wasn't into photography then, I don't have any photos of Mo.

 

[Helge]

I've seen you grinding this ax in other threads for a while now. I don't understand, you're under the impression that film of any sort was still viable as a commercial option?

I'm sure there are some corporations that still use typewriters and some CEOs that write memos with a fountain pen but for all practical purposes anything not written on a word processor is extinct. I would have thought this of film at least a decade ago if not more. Who out there that is using photography as a means for making a day to day living is using exclusively film? Yes, I'm sure there are a few people left but for the most part the world has shifted to digital. Unless you're living under a rock or something.

I'm shooting film for the pleasure of it. No new digital process is going to make me stop using film. Maybe if someone came out with a swapable 35mm full frame sensor that I can load into my cameras across platforms. I'm not sure where you're going with this MF/LF is dead.

It already is dead. I can get 25 shots for $250 before processing and scanning? What a great way to make a buck. Oh, you're not in it for the money? Then it's not a living any more and it's now art or a hobby. Great. No problem with that.

I genuinely don't understand the reason for these threads. We're in 2020, this stuff sounds like it's out of 2007.

You are about as bad as the OP only for different reasons.
This kind if circle reasoning, cliched FUD has been prevalent since the day digital became even tolerable in quality.
It’s under a guise of being sensible, adult and rational.
Here’s the thing:
Professional photography is art, with very few exceptions.
Always has been always will be.
Maybe not high art, or profound art, though sometimes it is?
You sell yourself and folio as a whole, and as sort of a Gesamtkunstwerk.
Good images alone would make everyone a professional.
Very few images are as truly time critical as supposed and often pretended.
Not that long ago, people where fine waiting for development and scan or colour separation.

If you are in the bargain basement and the cost of film, development, and/or scanning and enlarging is truely a factor to the client compared to what you are payed as a professional, then maybe you should consider if you are sitting on a twig that might snap at any moment, and whether this is really your true vocation?

I find that a surprising amount of professionals and certainly a lot of the ones worth keeping an eye on, are using film.
Either they never left and are “holdouts”. Or they are newcomers, that use film as one of their differentiating factors. Either using it actively in their marketing or using it as a trade secret.