So, I'm at a loss of what I should actually get...I mean, all I really use with 35mm is Tech Pan and HP5+. 4x5" Tech Pan is...a lot. And with LF I have no need for fast film.
I've been shooting 5x4 for over six years now. I think you are wrong about not needing fast film. What you don't need, is slow film.
Graininess ceases to be a consideration with LF. After all, a 10x enlargement from 5 x 4 film results in a print that's about 125 x 100 cm (about 50 x 40 inches). Few people make prints this big, so graininess is of little concern.
What is of concern is the considerably smaller f/stops you'll be using to control depth of field. You will seldom be shooting with an aperture as large as f/16, f/22 and f/32 are the most used apertures in 5x4. This results in very slow shutter speeds. Many LFers seldom see shutter speeds above 1/15.
In the field, motion blur from the breeze is a problem at these low shutter speeds. In portraiture subject motion is a similar problem.
I'm must sayin' that you'll probably learn that you want as much real speed as you can get with LF. That's one of the reasons I use 400Tmax and develop in XTOL 1:3. Gives me a real personal EI of 500. And 400Tmax has excellent reciprocity characteristics also. This is important in maintaining shadow detail when shooting at small apertures and long shutter speeds.