Because of the oft-repeated advice, I kept a yellow filter permanently in place for several years. I did get some nice shots when the sky was important, and not just when there was blue sky between the clouds. Apologies that I can't show you this one without a filter, you'll just have to believe me.
View attachment 353217
But when the sky was not really related to the subject, it could be distracting. And because I live in England, a lot of my photos had overcast skies, and were none the worse for it.
Furthermore, on a very sunny day, it could be difficult to meter for the (deepened) shadows in a hurry, and a few chance-of-a-lifetime shots were spoiled by impenetrable shadows. (Of course, a properly experienced photographer would have allowed for this, but there isn't time left in my life to gather so much more experience.
)
So now I leave the yellow filter off, but carry one in case of need. It is gradually carving a groove in its plastic box. I never use orange or red filters because drama isn't what rocks my boat.