To answer your original question, it mostly matters for ease of use. If the physical axis of tilt or swing is not located at the optical axis of the lens, then it may make altering those adjustments a bit less intuitive, and cause you to have to refocus the lens every time you do. However, in the end, you can probably still come away with the same shot. Now, another problem is if location of the physical axis is off from the optical center axis, then you run the risk of running out of image circle quicker to project onto the film plane. This is assuming you don't have rise/fall/shift controls as well to compensate for the lenses movements away from physical center.