Who, besides some recent immigrant groups, cares about the ethnic composition of neighborhoods when seeking to buy (or rent, for that matter) a house? If I can't afford a house, the ethnic composition of that neighborhood doesn't much matter. If a neighborhood is a dangerous place to live (due to thuggish residents), the ethnic composition of that neighborhood doesn't much matter.
But when some people hate a person, even their inconsequential actions acquire a reprehensible character to the eye of the hater.