I toured several months with a 4x5. I put the lens in the front pannier, right side, thinking to protect that side if I went into a crash. Yeah, right -- the one big crash I took had me over the handlebars and the front wheel/handlebars ended up rotated180 degrees! But everything ended up okay.
I had the camera in its pack on top of the back rack -- pretty well protected by the panniers.
On short trips with only the Rollei, I use an over-the-shoulder bag that works well in keeping the vibrations down somewhat. The bag ends up on the middle of by back, so my balance is fine. I just hope I don't crash and burn.