In the United States, You have no expectation of privacy in public. Anything you can see from a public place, you can take a picture of, even if the "victim" is in their own home but has the blinds open. If you are standing on a public sidewalk or street, you would legally be able to take a picture with certain exceptions.
An exception to this would be: if with just your eyes you can see into a private home, you can take a picture of that, but if you require a telescopic lense with some sort of IR adapter to "see through" blinds. That MAY be considered illegal.
For real world examples of this question, check out PINAC.
Another example is "Creepy Camara Guy", https://www.youtube.com/watch?v=vs6iLtl0BAw. This guy basically goes around recording videos of people in public in a VERY obnoxious way. But he is within his legal right. (Note: Video unavailable: "This video is no longer available because the YouTube account associated with this video has been terminated.")