I don't think it's either diffraction or diffusion that's causing the low contrast commonly seen in pinhole images. Rather, it's flare -- specifically, what large format photographers call "bellows flare", caused when a lens with too large an image circle is used; bright light then scatters from the inside of the bellows (or lens cone, or camera chamber) and produces an overall fog on the film that reduces contrast.
As I said before, I have one camera that doesn't do it at all (i.e. produces images of about normal contrast for the film and development), which I believe is due to a hyper-efficient internal baffle and de facto lens shade resulting from the conversion of the lens to pinhole. Additionally, my AutoPin Polaroid 210 shows very little flare and thus good contrast (with prints for which I have little or no contrast control), probably (I think) because the folds of the bellows capture more light than they scatter. By contrast, a couple of my other cameras that were originally made with lenses that barely covered the format, and converted to the ultra-wide coverage of a pinhole, give pretty normal contrast in controlled light, but flare terribly with a bright sky even just outside the frame.
Light fall-off can be an issue, but doesn't really affect perceived contrast, which isn't really the full-frame brightness ratio but rather on a scale just above the micro-contrast that influences perceived sharpness. In addition, reduced perceived sharpness tends to reduce the perceived contrast as well -- scenes with the same brightness range will look less contrasty if slightly fuzzy than if perfectly sharp.