The light built-in light meter in my FM3A changes the shutter speed (when on aperture priority) as you rotate the filter, and the image gets lighter (or darker) in the viewfinder. How can it be the same factor? Or am I misunderstanding what you're saying?
It can be hard to see sometimes, but in actuality only parts of the image in the viewfinder darken as you rotate the filter, while others remain unchanged.
The filter factor used for polarizers is that which tends to result in proper exposure for the shadows and the non-specular midtones and highlights. In most cases, those are the parts of the image we want, and which the polarizer has little effect on, whereas the polarizers do remove unwanted glare and reflection.
Glare and reflections tend to distort meter readings, so when we remove them, we tend to improve the accuracy of the reading (unless of course the subject of the photo is the glare or reflection, in which case you probably wouldn't use a polarizer in the first place).
The reason polarizer's require a filter factor is that in doing their work, they also add some neutral density.
Hope this helps.
Matt