Dust is a pain, I'll grant you that.
What I do is load in my darkroom, which is clean. As in, I take everything out at least twice a year and use a damp sponge to clean every surface - ceiling (you'd be surprised), walls, shelves, under shelves (very surprised), doors, etc. It also has an electrostatic air cleaner running most of the time - I turn it off when I need total darkness (damn LEDs).
I clean my film holders with an anti-static brush. Clean both sides of the dark slides, the film guides, the light trap, all of it, with just an anti-static brush. All away from where I'm going to be loading. And I leave the darkslides in the holders - I don't stack them separately. That way dust doesn't fall on the inside while they are waiting to be filled.
Then I stack the film holders on the table, put the fresh film box on the table, and lights out. I open the film box, and make sure the film is emulsion side down - so that any dust that falls out of the air while I'm working falls on the back of the film. I pick up the film a sheet at a time, flip it over, shove it in the film holder and slide the darkslide home as close to a single motion as I can. Flip the holder over and repeat for the other side. Stack the filled holders in a known clean corner.
When I'm done I close up the film box, turn on the lights, and immediately start loading the film holders into zip lock bags. The only time film holders aren't in zip lock bags is when they are being unloaded/cleaned/reloaded. I don't reuse the ziplocks either - my wife uses them for sandwiches.
Using this method I get very little dust on my film prior to exposure.
In the field I follow a similar method using a Harrison pup tent (swab out the tent with a moist towel the night before you use it so it can dry out - every couple of days). This method works for me even in the arid desert southwest which is supposed to be dust city. At least it worked until I said that ;-)