I want to second shutterfinger---use a drill press and not a hand drill. And secure the material down properly. The Dremel is a good idea, too. I had to hand grind an interior steel column to get another column to fit into it properly. Took a long time, but it worked and there was less of a chance of ruining the materials, tools, or myself in the process. A hand file will take time, too, but it does work.
I've only used a lathe to drill lensboards, including one someone tried to hand drill that was...hmm, how to describe it...ovalicious. That's the word. And I got it for free.