and at one point, I thought it might be hopeless -- that the scanner software did something that I wouldn't be able to reproduce optically.
Unfortunately, you may be right. Scanners do weird things with negatives and masks and such. I get great results scanning Slides because I've calibrated to a colour card. B+W negs look good, but there's not much to go wrong scanning them, just blackpoint, whitepoint, gamma and done.
But i've never been able to scan orange-mask negs properly. I've tried with-rebate, without-rebate, i've tried scanning as pos and inverting colours in GIMP, it just never works. (and for the record, it's a v750 on Silverfast, so it should work, but it doesn't).
I've heard people crap on about Ektar being beautiful and saturated, but it looks flat and dull on my screen; I'm blaming the scanner not the film.
I've tried scanning x-pro as well, both pos in c41 and neg in e6, they're both just as bad as scanning normal negs.
Actually, my total frustrations scanning negs is making me seriously considering trying RA4 now that I've got an enlarger, otherwise I may as well give up shooting colour negs all together.
For your problem, i'd suggest maybe scanning as a positive including the orange (green?) mask around the edge (or even just scan the leader). Then import into your editor of choice. invert the colours, and use the eye-dropper to get an RGB value of the mask. From there you should be able to calculate a rough CMY value to use in your enlarger, and tweak accordingly.
One question though, do you want to cancel out the mask exactly? Part of the fun of cross processing is the overall colour tint. So if you want to keep some of that then there's nothing else to do but keep on burning through test-strips...