With a diffused head enlarger, technically, you should change the light mixing box for the different formats, practically, people don't.
With my various enlargers I have used, with all of the diffused head types I have always left the larger mixing box in the head for all formats.
When I had my LPL 7450/1 enlarger, I always left the 4x5 mixing box in the head. The same goes for my DeVere 504 today.
There is a difference and it is measurable, usually it is in the order of about 1/4 to 3/8 or slightly higher, of a stop, which is unimportant, for all intents and purposes.
There is at least one exception to this rule. If you are going to push the format to an extreme, say maximum enlargement of a 35mm neg, then any help in illumination when the enlarger is at it highest, may be better for your eyesight, when deciding grain and/or sharpness parameters.
Mick.