Anyway, I've been referred to a book. Here's hoping I'll get my explanation there.
If the book is by Tadaaki Tani, It will most likely be to your satisfaction.
I don't understand, fundamentally, how it works so have no hope of explaining, but can give you several concepts to research.
First, the silver halide grains, the light-sensitive part of film, are inherently only "sensitive" to bluish light. As AgX mentions, sensitivity to other colors is added by the use of the "sensitizing dyes."
In both cases, the quantum efficiency tends to be fairly low - I'm thinking in the general range of 10 or 20% but don't rely on these numbers. Anyway, if you look at a piece of unprocessed film in the light, it generally appear to have a fairly light tan color. So you know right away that a lot of light is being reflected back, and is not absorbed by the film. And we know, historically, that a certain amount of light passes through the emulsion, because of something known as "halation." Some earlier films would get "halos" around bright spots of light. It was possible to coat the film, beneath the emulsion, with dark materials, and this would eliminate the halos. So it seems to show that the halos were caused by light, having passed through the emulsion, being reflected back into the emulsion by the film base.
There is another part to this. Even after light is absorbed, it is not necessarily adequate to form a developable image. It is generally considered necessary for a minimum of 4 (or possibly 3) silver atoms to come together in a single grain to form a stable, developable image. Anything less may disappear on its own, so even if a photon was initially absorbed, it might not be useful with respect to forming an image. There is an older book that can be found online, here, that may be useful.
https://archive.org/details/PhotographicSensitivity
In fairly recent times, the past dozen (?) years or so, a couple new technologies were developed to improve sensitivity. One is with "fragmentable electron donors," aka two-electron sensitisation, where it is possible to form a stable latent image from only two photons, instead of four. The other uses something called "antenna dyes," which are essentially dyes attached to the existing sensitizing dyes; the quantum efficiency may be roughly doubled by these. There have been technical papers with A. Muenter of Kodak as one of the authors, but probably have to be paid for.
Hope this will give you a good start.