If the loader is in fact phenolic (aka Bakelite) freezing won't hurt it a bit -- it's just as brittle at room temperature as it is straight from the freezer.
The issue of freezing unsealed film is one that seems to have many answers -- the biggest problem seems to be condensation, and on that count I'd be inclined to suggest it's better to freeze the cassettes (in a sealed bag) after loading than to freeze the loader -- the loader has much more air space inside, hence more humidity available to condense and water damage the film. Even if you get no ferrotyping (on color film, likely the case since the emulsion is so hard), water can remove sensitizing dyes, possibly even redeposit them elsewhere, and leave you with streaks and blotches.
So, what I'd suggest is to load it all into cassetted, bag the cassettes a half dozen or fewer per bag, and freeze them that way (with dessicant in the bags if you're in a damp climate). I've done that with the Tri-X I bulk load and get excellent results.