I am reminded of the Mark Twain admonition: "All generalizations are false, including this one."
Only 58,000 M4s were produced, as compared to 225,000 M3s and 83,000 M2s, so M4s are more rare. For the M4, add in a rewind crank rather than a knob, framelines for the popular 35mm focal length, often considered normal for rangefinder shooting (the M3 needs "frog eyes" or auxiliary finder for a 35mm lens, but has greater finder magnification), a self-resetting frame counter, all making the camera friendlier for someone who actively uses it, then consider that the build quality was more akin to the M3 and earlier camera, more brass gearing, smoother film advance, etc, and fewer corners cut to meet a price point as in later models, and it all adds up to a more expensive camera, one preferred by both collectors (rarity) and users because of the features. So its used price is higher.
I guess some might consider this irrational. It seems like reasoning pretty well grounded in reality to me. It's up to the buyer to decide if the differences are important. As Roger Hicks has pointed out, the cost of a Leica M amortized over its working life is pretty low, so you might as well get the features that fit your needs if it's within your budget.
Lee
