At least in part this is about particle size and to an extent also geometry. The silver halides (silver chloride and silver bromide) are white to very pale yellow; i.e. they reflect back most visible wavelengths, which also explains why they are inherently mostly sensitive to UV as they readily absorb this.
As the silver halide particles reduce to metallic silver due to photon capture, conglomerates of metallic silver form. Depending on their size and geometry they start to absorb certain wavelengths; as these particles grow, you generally see them progress from yellow towards orange, then brick red and ultimately purple brown hues.
The same effect of size and geometry also underlies the colorful nature of lith prints and Van Dyke Brown and kallitypes.
This is about what I recall from having looked into it for a while when I was exploring lith printing. I'm sure there's more information available on the underlying physics that can help understand how certain nanostructures can more effectively trap photons with specific wavelengths. I guess there's probably a few effects that interact when it comes to lumen prints.