I think a lot of White Fragility comes from a lack of agreement about what the word “racism“ means. The traditional meaning probably needs a new name at this point, so I will call it Propositional Racism i.e. accepting the truth of racist propositions such as “Black people should not be allowed to swim in my town’s public swimming pool.” When most white people say they are not racist, what they usually mean is that they don’t accept the truth of this kind of racist proposition. When anyone says that all White people are racist, they usually are referring to racist habits, or systemic racism in which a white person’s behavior helps to reinforce, or is complicit with, some sort of racially based injustice. These habits are not necessarily correlated with any kind of propositional racism, i.e. one can vote regularly for progressive black candidates, make regular contributions to black causes, believe that affirmative action is just and necessary, and still possess these racist habits. An example of such habit is the inability to recognize the faces of different POC. This can be present even in people with no significant propositional racism, because it is embodied by a part of the brain that controls habits, rather than language. I discuss this further here.
If someone is not a propositional racist, and is accused of being a racist, they are naturally going to be angry if they think they are accused of being propositional racists, because the accusation as they understand it is both false and insulting. I think that’s really all there is to what we call “white fragility.”.