Usually the stitcher has options to enter the camera motion: planar, rotation, and the sequence (vertical, horizontal, zig-zag or serpentine) so that are all settings to minimize problems.
When you are taking frames by hand there are certainly also differences in rotation (horizon), so the most ideal setup is to take the photo's from a tripod with a panorama head. If that is not an option you could pre-process your frames in an editor to level them out before feeding them to the stitcher.
The bottom line is that if you have frames that are nicely lined out relative to each other and with sufficient overlap you will almost certainly have a good stitch.