I was not scrubbing as much as rubbing gently. I just kind of did what I read up to do on the internet, the chamois was one of the things suggested. It varies as to what side the blemishes are on. It just seems to be luck of the draw.
I will re do the stabilizer step and then soak, agitate and hang up to dry. Thanks guys.
Thanks for letting me know I can not remove and of the dyes as well. I was a bit worried about that but was not sure if it was possible or not.