I have a bronica s2 and recently acquired a RZ67. In order to get sharp focus on these cameras I use the pop up magnifier. My technique is to frame the shot at waist level then lift the camera to my eye for sharp focus through the magnifier. Then return to waist level and shoot This seems wrong because of the possibility of not getting the focus right at large apertures. I am fairly new to all this and I am looking for how experienced users go about this.
You don't really need to lift the camera to your eye at any point -you can frame your subject with the magnifier folded away,and then flick it into position - you'll see a highly magnified area you can use for focusing.
Another good reason to use a tripod, I guess. Nevertheless, it seems you've already had a go at this, so how did the prints turn out? Just try a few rolls and check out the results. Then if you need to change your technique you'll know what you need to do. For what it's worth, I do the same as Ben with my 645. Haven't noticed any problem so far.
If nothing else, a few lifts and you'll have done your biceps a world of good.
Second the tripod thing, though it is not your question. It can force more careful contemplation, framing, leveling, etc (which may not be what you want, depending on your style), and eliminates your issue. For me, this is much more important for 2 1/4 than 35mm. But my 120 camera is a Rollei TLR, not an SLR.