Agree... in fact that's what I did this past weekend. The fall colors this year have been way better than expected. Enough to convince me to buy my first roll of color film in 15 years (Ektar 100).
That makes it about 2006 when I last shot 35mm color. Just a few years prior I had finally purchased a polarizer. The quality of my prints immediately improved. Even with cheap consumer film the colors popped in a way they never had before. I have one print of an old ranch building in combined sun and storm, and another of an old mine shack among the fall leaves, that I still love today. Made me wish I had always used a polarizer.
The penalty is a ~2 stop increase in exposure. Well worth it, and 400 speed films in bright light will put you in an aperture/shutter sweet spot anyway.
Some polarizers are more effective than others. I have a cheapo 52mm that works a little better than a "nicer" 49mm. By that I mean it can remove almost 100% of a reflection from shiny surfaces. You may not always need or want that though.
Results will vary depending on light direction and amount of diffusion. In overcast or shade you might not see much difference, while blue skies, especially at high elevation, can look surreal.