I believe you answered your own question, which is that you'll have to stay fast enough to keep camera shake to a minimum, and not try to slip under the ASA (i.e. don't think that you can compensate for underexposing by a half stop or so). Check your enlarging lens for fungus, dust, or separation, and then maybe try something like Tri-X/HP-5/Delta-400 in Microphen, for a little extra effective speed.
I have 8x12s from 400 negs or slides, and the deciding factor on whether it looks good at that size is camera shake, followed by sufficient depth of field for the subject; except when I've used Delta-3200, it's never the film. You may just want to look into improving your shooting stance by using doorframes or walls for support, pulling in your elbows, or shifting how you balance your camara.