I think it comes down to this:
If you have a bad experience with a Black man, you remember he's Black.
If you have a bad experience with a Caucasian man, you remember he's a jerk.
This used to show up all the time in television and newspaper reporting. "The suspect was described as Black, about 160 lbs, in his 30's". But if not AA, "The suspect was described as about 160 lbs and in his 30's".