Using a diffusion chamber matched to the size of your negatives has two advantages:
1) you will maximize the amount of light going through the negative, which means your printing times will be minimized. That is obviously most important for large enlargements and dense negatives;
2) you will minimize the potential for extraneous flare - something that may not matter much for enlargers designed for professional use, but possibly more important for enlargers that aren't as carefully put together.
It is reasonably likely that your enlarger will perform very well with just a 4x5 diffusion chamber. My Omega D6 with an Ilford Multigrade 400 head certainly did - the 6x7 and 35mm diffusion chambers stayed in the drawer. But If I had seen the need to make lots of prints from 35mm, in a relatively short period of time, I would have happily swapped the diffusion chambers.