I've had a couple of situations where l was confronted while taking photos, or where people would comment behind my back, thinking that l wouldn't understand them. Wearing a Nikon around my neck and looking around, makes me look like a tourist in my own city!
Anyways, a couple of months ago l was taking some photos of some building with a very interesting facade, and an older guy, maybe in his 60s crept up from behind me, waited for me to take the shot and then asked me why l was taking photos there.
I tried to explain what l was doing, but the guy insisted that l could see through the windows and that l was taking photos of people, rather than the building, even though l was standing like 100ft from the building. With a Kiev 4. I thanked him for the ''heads-up'' and went my way, there's no use in arguing with people like that.
The other day, l was taking a photo of a VW Samba, which was handpainted with various colors, like a hippie van, and a couple of girls, probably thinking that l wouldn't understand them, said something like ''look at this guy, he's probably taking photos for his instagram''. You should've seen their faces when l looked at them and told them that if they liked what they saw, they should drop a like or two. Normally l'd just look at them and say nothing, but that day l took out my brand new 50mm 1.8D, so l was feeling especially confident!
I've missed plenty shots because of the feeling that you described in your first post. Just take the photo if you like what you see, and if someone walks up to you and asks you what you are doing, politely explain and perhaps have some photos ready on your phone to support your story. Soon enough you won't even notice the fear, you'll just be snapping!