The plastic end is glued or pressed into place - the problem, usually, is keeping them on. You may damage it (or your spare) trying to seperate it from the metal lever.
Try this: Find a rubber band about a quarter inch wide. Loop it around the bezel. Carefully use a pair of pliers to break the bezel loose, with the rubber band protecting the bezel from scratches from the plier jaws. Make sure you have a good grip before you start turning the pliers - righty-tighty, lefty-loosey. Unless there is damage to the bezel from dropping the camera, it should take only a slight movement before it breaks loose and can be unscrewed the rest of the way by hand. This isn't fool-proof, of course - that's why they make those expensive copper clamp-wrenches Nicholas linked you to. Good luck!