Macy,
In answer to your question about whether to use the 77 mm screw-in or the Cokin square, I suggest the screw-in ones.
I guess that the Cokin filters have plain uncovered edges. A filter with covered edges, like the mount of the screw-in or square filters covered in a filter holder, is preferable to one that doesn't have covered edges. It is better if there is no light entering the filter from the edges. This can lower the contrast. (In fact it is used to lower contrast.)
You want to make sure that no light hits the back of the filter, and that the gap between the rear of the filter and the front of the lens is closed. That should be easier with the screw-in, thanks to the lip.
The screw-in filters might be multi-coated or single coated, the Cokin resin one will not be.
If you have a lens hood that screws into the front of the 77 mm filters, you might be able to use that with the 80 mm. The absence of a lens hood is one of the biggest disadvantages of holding filters in front of the lens. I guess that you don't need to be told to shade the filter from direct sun whenever possible.
The effect of the differences I mention above will not always be evident, it will depend on the situation. How important these factors are to you will depend on how important it is to reduce the chance of having problems.
If, like me, you don't change filters very often, you could put polyester or gel filters (like Lee, Kodak Wratten and Wratten 2) behind the lens, attached with 'snot tape' (Scotch ATG tape). You aren't using fast lenses, so the slight focus shift should not be a problem.
Good luck,
Helen