I'd keep fooling around with masks...but I wouldn't use litho film unless you wanted to completely block out a certain area or areas during a second exposure, as opposed to doing the whole print with one exposure. I'd use a standard in-camera film. This will give you easy and consistent continuous tone and a higher quality base and emulsion. It also allows you to have a lot of control over your mask with development, which is what you need to finely "craft" good masks. It is much more expensive, but is worth it. I use Ilford Ortho, preferably, as it can be processed under red light. Litho film, not being designed for continuous tone, is very finicky to work with, and consistency can be hard to achieve. It is also easier to damage, the emulsion can be prone to tiny defects, and the base is generally much lower in quality.
Masking takes lots of time, and plenty of trial and error. You should not expect to get it right on your first try. Also, while I have made a few masks by visual registration only, it certainly was not fun or efficient. Even a basic homemade registration system is usually better than this. Pins are ideal, but at the very least, cut out four edge notches on the original frame, using an X-Acto knife – one near each corner. This will aid you greatly in achieving visual registration if this is the way you decide to register the sandwich.
Also, IMO, you should experiment with different levels of diffusion during exposure. The diffusion goes between the in-camera film and the mask when making the exposure. The more diffusion that is used, the less precise registration has to be, but the softer the edges in the mask will be, thus the "dirtier" the line you will get between the masked area and the rest of the print. The various levels of diffusion can be used to creative effect, or out of necessity. With some images, the film base of the mask alone provides the perfect amount of diffusion. With other images, this provides a mask that is too sharp, and you can get a weird wiry effect no matter how much futzing around you do with registration. You should look for a diffusion material that is thin, not easily damaged, completely patternless, and available in various different levels of transparency.
A densitometer helps, but comparing your mask with a step wedge, or anything transparent with a known density, will do the trick.
Here is a good Webpage to read through, and it has some good links at the bottom as well:
http://www.largeformatphotography.info/unsharp/.