Alright after playing around today I finally figured out what the problem is.
I always shoot in DOF preview mode. I've just enjoyed composing and seeing what's in focus, and I've never had a problem doing this being too dim with the nice big viewfinder. Only, not only does the lens tell the meter what aperture it's shooting at, but it also meters based on how much light's hitting the prism finder. So stop down, and every stop you're doubling the problem.
I feel like such a muppet, but at the same time I can see how easy this is to do.
Now I'm actually looking forward to shooting slides again, and I should be getting much better results with B&W too. Argh, the months I've wasted!