Considering the price you paid and the quality of the piece, I would say it's definitely worth shelling out for a CLA. Maybe not from Kanto if that's out of your budget, but from a reputable tech, for sure.
I paid quite a lot to Nippon Photo Clinic here in NYC to CLA my S3 (the much more common older brother to your S4) and it was well worth it. Nikon rangefinders are a joy to use and that 50mm f/2 will not disappoint.