Specks when looking through viewfinder??

How do I fix this?

Download the instruction manual http://www.cameramanuals.org/nikon_pdf/nikon_fe.pdf if you have not already.
Page 41 changing focusing screens. Follow the instructions and remove the focusing screen. Blow out the bottom of the pentaprism, the focus screen, then reinstall the focus screen as it was.
DO NOT get finger prints on the screen.
If the spots remain they're between the viewfinder eye glass and the pentaprism and requires the top of the camera to be removed.

I generally just ignore them.

Usually the dirt is on the viewfinder and does not effect the photograph. It likes to collect in the hardest locations to clean.