Might it be possible to find a bit of aluminum tubing that could fit? A pin or screw that goes through the collar and the shaft instead of a set screw? Or maybe an adhesive instead of the set screw if you never intend to or need to disassemble it?
As the cartoon girl in the image says, this is good! Gives me a great starting place to search. Thanks!
EDIT: an hour and a half later, I can only find SI options (none of which are for a ~9.53mm bore, go figure). Still, the search continues. Already checked MMC, SDP-SI, Grainger, Amazon, and eBay...