Yes, it is normal.
As the enlargement size goes up the exposure time goes up. However, the amount of stray light from the enlarger stays the same. The stray light times the longer exposure time will lower the print contrast. The stray light comes from light leaks around the enlarger and from light reflected from the paper on to the walls and then back on to the paper. Wrapping up the enlarger's light leaks and painting the walls around the enlarger either deep red or black will cut down on the stray light.
And, as Matt has stated, there are subjective factors at work.