Dividing a room into oppressors and victims is a terrible way to frame society. Some males are privileged, some are not. Some women are privileged, some are not. Attractive people are incredibly privileged compared to plain ones. Would we have beautiful people put a bag over their head to make the ugly ones feel less bad? Rich people give up their Leicas so iPhone photographers aren't jealous? I'm neither rich nor good looking.
Unfortunately, 'priviledge' is different than you describe. All males have male priviledge. It has nothing to do with oppressors and victims. It has to do with societal habits that give advantage to males over females, reconizing them, and leveling the playing field. Common examples of male priviledge; being able to take a walk by yourself on the beach at night. Not having to think twice about getting on an elevator that has just a man in it. Having ones sexual health care coverage decided by ones own gender. Having your imput given greater consideration than women's at business meetings, and so on.
Denying or ignoring male priviledge is a terrible way to maintain our society...but very easy for males to do. The staus quo is fine and dandy, eh?
There are many types of piviledge based on a society's particular biases. The priviledge of the rich, and as you mentioned, of the beautiful. Even being tall has its priviledges, and I have had to learn how not to take too much advantage of it (mostly the ease of being able to command attention).
Male priviledge, like white priviledge, is of a different kind in that it is divided by gender, or by race in the latter. Both kinds need to be reconized for what they are...biases. They can cloud ones judgement and limit the opportunities of others.
I am reminded of a scene in the movie
Money Ball. Baseball scouts were discussing a possible player -- who was looked upon negatively because he had an ugly wife. Obviously the player must have something wrong with him.