Yep. This is Leica's strength and weakness at the same time. Each viewfinder only works well for one focal length. The 0.85 is awesome for 50mm, 0.72 is for 35mm and (I suspect but never tried) the 0.58 is great for 28mm. Everything else is a compromise. In a way it helps curb costly M-GAS: it is hard to justify owning more than just two lenses.
If money were growing on trees in my back yard, I would have completely thrown this silly concept of an interchangeable lens rangefinder out the window, and just had three dedicated M-bodies with different viewfinders for 28mm, 35mm and 50mm lenses. Also, since I'm dreaming, I would have two copies of each lens: fast+heavy (Lux), and slow+compact (Summarit/Elmar).