Scootermm, this sounds like a neat project. And the link provided by jnanian regarding the Mandallete camera was interesting.
I've built several pinhole cameras using the "falling plate" principle, which may also be a solution to your need for multiple large format negatives in one box. My version of this design uses matteboard film holders, stacked in the rear of the camera. Each holder has a notch, or series of notches, along the top or sides, which alternate with each holder. A light-tight mechanical sliding linkage allows the front-most holder to fall face-down into the bottom of the camera, with a little encouragement by tilting the box forward. The next film holder is thus revealed, ready to shoot.
My 8x8 version, using 8x10 paper negs or sheet film, holds 12 sheets. It has slots on both sides of the holders, interlocked by sliding linkages on both sides, operated by knobs. The problem with this design is if you don't operate both knobs simultaneously (up or down direction), the front holder falls crooked and gets hung up.
My 5x8 version, which holds 10 sheets, corrected this fault by using a single sliding linkage on top, which interlocks slots cut into the top of each holder.
Other operating limitations with the falling plate design is that your focal length has to be longer than the film holders are tall. Which wouldn't really be much of an issue with your panoramic format. The other issue is you've got to transport and handle the box in an upright position, otherwise the fallen film holders could get all topsy turvy in the box and jam the rest of them from falling properly.
Overall, however, I've gotten used to these issues, and find that having a dozen LF negatives already loaded in a box to be a refreshing way to do pinhole, especially scenic locations, where you are far from home.
As to the optimal pinhole size: making it too small - that is, going beyond optimal - doesn't necessarily make the images blurrier, as the increased diffraction from the too small aperture is being made up for by improved geometric effects of the too-small hole. My experience is going moderately beyond optimal you'll find the sharpness platues off, with no real improvement or loss. The real problem is that your exposure times just suffer too much, as reciprocity gets to be a real bear.
Good luck, keep us posted.