Hi Rich - A couple of ideas for consideration. Not necessarily the gospel though!
Since you have been investing in pinhole for years, it may be worth switching from spending lot of time to spending a little money. If you buy just one commercially manufactured laser-drilled pinhole, then you will have a known "gold-standard" as a reference. The thin wall, blackened circumference and sharp edges of the commercial piece might save years of misplaced effort. If the commercial hole is sharper for you then you are further along the path. If not, then you still have a righteous pinhole for your next camera. Regardless of outcome, you will truly know how good (or lacking) your own pinholes are... and what's that worth?
Next, is it possible that contrast is an issue? Could light be bouncing from your film to the wall of the camera, and then back to the film? Assuming you are painting the inside of the box, some matte black paint can bounce or scatter light more than you might think. Maybe a switch to a black velvet lining inside the camera would help. Or baffling. Or both!
Also, are you totally sure that your box material is 110% opaque? Far out question, but one never knows.
Along the same lines, are you "smoking" the edge of your pinhole aperture? This can help to eliminate stray light that might degrade an image.
Crazy as it sounds, sometimes problems can be diagnosed by "seeing what the film sees." If possible, try getting an eyeball onto the film plane of the camera. This will of course require totally excluding all other light sources, so you may find yourself staggering around the yard with a camera duct-taped to your head! Leave either your nose or mouth uncovered!! In any case, once your eye adjusts you may see obvious problems such as internal reflections, light leaks, hot spots, etc. (This technique also works wonders with Feds and Dianas!)
Same logic in revese applies to film holders. Go into dark room. Install holder, pull slide, remove pinhole, shine flashlight in through front of camera and look for leaks from the outside.
Finally, have you tried a few varieties of film? Say, a controlled experiment with T-grain fast and slow film along with conventional fast and slow film.
It would be instructive to try a controlled test with supposedly undersized or oversized pinholes as well.
All of this will leave you with no time to actually shoot the damn thing, so I promise you will no longer be vexed with unsharp images!
Jon
I have been making phole images for a few years, and have several home-made boxes. I projected all the pinholes on the wall, got an exact measurement of size, did the math to figure out the optimum "focal length," and inspected the pinholes under a microscope for any defects...I'm happy with the images, yet they don't appear as sharp as some others I see...