I can think of little else more frustrating than being somewhere for the first (or any other) time and missing a shot that I wish I'd been able to capture. I also sympathise with those who become frustrated with hordes of tourists wandering around shooting the most inane subjects almost as a reflex (no pun intended) action. Working adjacent to The Tower of London makes for fascinating lunchtimes.....
However, where I go, a camera comes with me - usually my F1-N, a 20-35mm f3.5 L, 50mm f1.4 and 85mm f1.8. Many days they stay in the bag but they do get used regularly and I generally get the shots I want.