It depends! If you basically want to leave your yellow parts of the scene at the same exposure as a non-filtered shot, and remove blue (darken a blue dress, sky, etc.) you could make no adjustment for filter factor. On the other hand, if you want to lighten yellow (contrast filtering) you might use 1 stop additional exposure to increase exposure of yellow and keep blue as it was without a filter. The kicker comes when you start to think about what you want to do with greens, orange, etc. Like much of photography, the "always do x" rules seem to get replaced by "what do you want to achieve on the negative?" approaches. Some of the older - 1950's - Leica manuals and Graflex manuals do a good job on contrast filter theory and applications.
Basic solution: experiment.
Ken