I tried to back the screw shown above out. Some residue (parts of glue?) came off. I put the screw back in again, while trying a few variations, but it still wiggles both laterally and horisontally, even when the screw is tightened down.
It seems to move from the underside so to speak, when the inner L-shaped lever of the release button moves. Obviously the socket and the release lever is part of the same mechanism, but I don't know how they sit together really.
Is there a risk that the socket would fall off? If not, maybe I can live with it.