The film does not sit behind the ground glass-- you'd be taking a picture through a frosted piece of glass, and while that might look nifty, it's never going to be sharp or clear.
Yes, you focus on the ground glass. Then you insert a film holder, which pushes the ground glass out of the way, and sets the film at the same plane of focus as the ground glass-- but that's a predetermined location, and the film holder has no way to know that you've actually moved the ground glass from it's original plane of focus by inserting a fresnel (or any other shim, spacer, etc.) in the way.
Take a 4x5 camera, with a 150mm lens on the front, with no fresnel lens. The ground glass, at infinity, will be 150mm from the focal point of the lens. Ordinarily, when you put the film holder in, the film will also be 150mm from the focal point. But if you put in a 2mm thick fresnel, now the ground glass surface is 152mm away, and you have to shorten your draw by 2mm to compensate. But when you switch to the film holder, it's now 148mm from the focal point, and your photograph is out of focus.
Cameras like your Toyo that's designed to have the fresnel between the lens and the GG will set the film plane back by 2mm to compensate, so that you're at 150mm for both.