Of course demosaicing is a good guess, and overall an impressive feat, considering what the initial dataset looks like to the human eye.
But there is only so many ways you can improve it. Even new deep learning networks approaches, sees only very marginal improvements.
The main problem persists as smudging or waxiness of complex real world structures and textures. IE not USAF 1951 or picket fences.
And, a subtle but overall noticeable (if only subconsciously), different treatment of high contrast, clear geometric edges. Sometimes appearing as almost unsharp masking.
The trouble is of course, that people like "clean" and they love "sharp".
They've been taught to by the photo industry, and billions of examples, and the human visual system gobbles clean and sharp up as "free processing".
It's the equivalent of too much sugar and salt, on small samplers of food.
It's nice in small doses, but becomes cloying and too much all the time and in meal sized portions.
Of course, looking at images shouldn't be "eating your vegetables". So sure, sharpening and cleaning has its place, but everyone needs a healthy varied diet.
You don't see any problem with too clean and too sharp until you acclimatize, and look at how good big film prints used to look only a decade or two ago.
All very academic since your scans are probably terrific.
Again, what I'm talking about is that one in a hundred or a thousand frames, that you want to print at some size and present on your wall for years or use in a gallery.
Regarding luminance and bayer filtering:
I'm having trouble seeing what's puzzling you. You basically describe yourself what I'm trying to convey: "you are in fact sampling something at every sensel site and depending on the algorithm you’ve chosen to demosaic with, you’re not just naively filling in the holes".
Less aggressively saturated colour filters will make continuity and rapid changes in luminance dominated, monochrome and even very complexly detailed multi chromatic structures easier to "glean".
Stitching at higher macro settings will not only give you the needed resolution to get over Nyquist and grain aliasing, it will also also allow somewhat higher colour precision.