For ordinary glass filters, the focus shift is about 1/3 of the thickness of the filter. (This is because typical glass has index of refraction n ~ 1.5, air has n=1, and the optical path length is proportional to 1/n.)
A typical filter is perhaps ~1.5mm thick, so you have ~0.5mm of focus shift, which is very noticeable behind the lens (in image space). But 0.5mm is negligible in subject space unless you are working at macro distances. That is why you don't notice the focus shift with a filter on the front, and if you did notice you would just refocus and take it out.
For a thin flat piece of glass, if the cones of light passing through it are narrow angle and are not at a sharp angle relative to the glass, the aberrations it introduces are negligible. So filters of good optical quality have little effect on the image in the front, since the beam of light from a point on the subject to the filter is very narrow angle (very slow in effective f-ratio). The cone of light from the lens to the film is much faster (f-number equal to whatever aperture you are using), and the off-axis angle of beam to filter might be large, IOW the filter is tilted relative to the beam. This in principle introduces an off-axis aberration, I think it's astigmatism, but whether it is significant I'm not sure, and it likely depends on the lens.
If you have a lens that is designed to take rear glass filters, like certain telephotos and fisheyes, leaving the filter off will cause unacceptable focus shift and may also cause an off-axis aberration, since the lens was designed to work with the filter in place.