My suspicion is that it's not a case of chairmen/owners thinking "I don't want to employ him because he's black", but more a case of an ingrained distrust of other races/people who are different, possibly because these owners/chairmen are from an older, more ignorant generation.
It's just a suspicion though, because as you say surely nobody would spite themselves in the name of bigotry.