The easy and inaccurate explanation is that lens size is dependent both on focal length and maximum aperture. A 50mm f2 lens needs a maximum opening of 25 mm, a 80mm f4 lens a maximum opening of 20mm.
The larger the size of the film, the larger the focal length needed to give the same angle of view. Where on 35mm 50mm is considered the standard focal length, on 4.5x6 this will be somewhere near 65 mm, on 6x6 it is 80 mm and on 4x5 it is around 150mm. All these lenses give more or less the same angel of view, comparable to the diagonal of the film format.
The consequence is that a f2.8 80mm for 6x6 needs to be bigger then a f2.8 65mm for 4.5x6. But then: a f2 65mm for 4.5x6 needs an opening of 32.5 mm and is bigger then the f3.5 80mm with its opening of 22.9 mm you can find in many TLR's. Of course there are many exeptions on this basic rule, for instance when you need retro focus contructions to clear for a mirror.