Let me correct myself before going on. The red saturation mask is made with a pre mask through a green (#61) filter and the principle is made through a red (#29).
Now a word about color filters. Color and saturation masking use color separation filters generally wratten #29 red, #61 green and #47b Blue. Within the color ranges the higher the number the "sharper cut" or in other words the more specific they are, a #29 is sharper cutting than a #25 and a #48 is sharper cutting than a #47b etc. The choice of filter will be affected by the colors you want to effect and the material you start with. Kodachrome benifitted by the use of a #24 instead of a #29 for instance.
To control the masking process you have to know the dynamic range of your system. Different enlargers have different contrasts and lens flair is different as is darkroom conditions. To do this you need a 21 step grey scale (Kodak or Stouffer) it is a piece of film with 21 equal (about .15) steps of grey from clear to black. If you have the kodak you will thank yourself for numbering each step neatly at the edge with a fine point sharpie or india ink starting with the light end from 1 to 21. The Stouffer comes numbered. Put the grey scale in your enlarger and mask off any open space around the scale with developed out film (lith film exposed and developed in the light works great) or black paper. Make a print onto whatever material you are using so that as many steps show as possible. You are looking for the spot where the steps disapear on each end of the scale. Note the number of the last visible step dark and light. Read those steps on a densitometer and subtract the smaller from the larger and that is your dynamic range. For demonstration purposes I will use a dynamic range of 1.90 yours will be different.
Now to the mask. Set up to make a contact exposure. You need masking film I used to use Kodak Pan masking film which is no longer made Kodak says that Tmax 100 will work. You will also need dilute developer such as HC-110 mixed 1/2 oz per quart of water stop and fixer. Tray constant agitation at 68 degrees. It is important that you do everthing in a repeatable fashion! Make a series of exposures starting at 1 second and progressing at twice the previous ie: 1,2,4,8, 16 etc. process the film at 1 1/2 minutes at 68 degrees. You are looking for the exposure that gives a density of .35-.40 at step 21. When you find it note the exposure time that produced it and read step 1. Subtract the smaller from the larger, take that difference and divide by the density range which gives a percent (gamma). It will be around 15%. Repeat the procedure with a 51/2 minute development time. (These are Pan mask times you might have to "fish" a little for the Tmax times)
Armed with this data get a piece of graph paper. Place the gamma up the left side, the development time accross the bottom and exposure time up the right side. Place a mark at the point wherre the gamma and 11/2 minute dev time intersect do the same with the gamma at 51/2 min and draw a strait line between them. Now place a mark where the exposure time for 11/2 min time intersects with that time. Repeat with the exposure for 51/2 min and draw a dotted line between them. ( I will get an example up soon)
Now for the why of all this.......Lets say you have a transpaency (an almost white wall with a dark vase of flowers say) measure the density of the wall and the vase and when you subtract the smaller from the larger you get 2.45. From your test of dynamic range you know that anything more than 1.90 is trouble. Take the density range of the transparency 2.45 and subtract the dynamic range 1.90 giving a difference of .55 divide .55by the original density range of 2.45 you get about 21 %. Now to your chart. Place your finger at 21 gamma and run your finger to the right until it hits the solid line then down to the deveopment time required. Now move your finger up to the dotted line and then right to the exposure time. You now have the exposure and development time to give the exactly right mask for what you are printing.
We had a chart for each filter and exposure set up we used.
Next the highlight mask.......