If the plastic is genuinely Bakelite, no amount of heat short of charring will hurt it -- but you do need to watch for thermal expansion of the brass pin chipping the brittle phenolic.
You might be ahead to use a Dremel or similar to machine a clearance to allow use of the glued-in bushing method Charles suggested. Whatever you do, beware the infamous brittleness of Bakelite -- it's plenty strong, but has absolutely no give; it won't bend or stretch, it'll just hold absolutely rigid right up until it breaks.