A lot of folk blame light leaks, and those may be part of your problem, but if you will look inside your camera you will see a bigger issue.
Take the back off and peek in there. See the plastic sides? The SHINY plastic sides?
That's the biggest problem with Holga cameras and many Russian cameras of the same kidney -- cheap construction that involves shiny plastic. The light hits the insides of the cameras and bounces everywhere. What you see is the result. My Stereo Sputnik was horrible. My Holga pinhole stereo was unusable.
The cure?
Put a piece of masking tape on the inside of the camera so it covers the lens/pinhole, then spray the inside of the camera with matt flat black paint -- primer paint works nicely, the flatter and blacker the better. You want the inside of that camera to reflect no light at all. Or, also, you an cut out pieces of flat black paper, or black felt, and paste them to the walls of the inside of the camera. If you want to get really fancy you can construct baffles, but I've never tried those.
When the paint dries, try it again. My prediction: Vast improvement.