It took me a long time to realize that, and to realize that taking a risk to share these accusations with an individual is a gesture of respect for that individual. When a white person does that to another white person, however, there is no such risk involved, as well as no reason to believe that the accusing white person knows what they are talking about. So these days, although I do listen and reserve judgment when a black person accuses me of racism, I am still skeptical of such accusations from white people. More on this here.