You're right that the diopter only affects how you see the viewfinder image.
Focussing entails setting the lens so that the viewfinder image appears sharpest. When that is depends on the lens to subject distance only.
Using the wrong diopter on the viewfinder only means that you will see the viewfinder image a bit less sharp than it cold be. But it appears sharpest still depending only on the lens to subject distance. It does not depend on your vision of the viewfinder screen.
But when the diopter is wrong enough, you will not be able to judge the sharpness of the image on the screen anymore, and focussing becomes a matter of chance.
The moment the viewfinder image is sharp still only depends on the lens to subject distance. Not on how you see the viewfinder image.
The viewfinder image typically appears at an (optical) distance of 2 to 3 meters. A distance at which we strain our eyes least.
You need to match the diopter strength of the viewfinder to the prescription glasses you might need for that distance.
If you only need glasses for larger distances, not at all at about 2 meters, you need to change the diopter in the viewfinder to 'neutral'/0 diopter.