Not all RTV’s are the same. Dow 3145 is a low-outgassing RTV (probably better to say silicone adhesive, but we all know how RTV acts) specifically designed for these types of applications. There are newer, better variants that are priced well outside the reach of mere consumers like us, but we use this at work on prototypes and low-cost production lens assemblies when the application of an RTV-like adhesive makes sense.
For epoxy, 3M 2216 and 3M 1838 are used a lot and are probably a better fit for this application, but those epoxies cost ~$500 a tube.