Thank you for your reply.
Since I posted this question , I did call around some camera and repair stores, one store checked the Nikon service department and unfortunately the 300 AF-S F4 D is not listed on their availability service menu, which means Nikon does not have the parts anymore.
The local repair shop mentioned since Nikon won't ship the parts to them, it depends what is required to be fixed, this doesn't look good neither. But I will try to bring the lens to them and have a look.
My last resort is to buy another AF-S F4 D (the 300 f/4E PF is out of my price range) and still have one manual only lens. So that I can use it on the F80, F100 and the older FE.
Still not sure what to do at this moment, problem is if I buy another AF-S 300 F4D , who knows when will the SWM acts up and back to square one again.
If I get it fixed, I know the lens will last for a while with the new SWM replaced. (Hopefully)