I would try to get one of the newer fans that are super quiet. My older Delta fan is too loud to ignore. Vent Axia used to be available, and they made silent fans that move air—very nice!
A high quality baffle will allow a lot of air flow, and be light tight. Some of the cheaper baffles rely on foam inside to keep them nearly light-tight, but that method cuts drastically into air flow.
I think many of the newer bathroom fans are super quiet, and it might be worth rigging one up to make it light tight if necessary, or perhaps you can vent into a dark space.