I agree with you on this, but there are lots of anti-racist who don't. In fact, most of the anti-racist writers I've read on medium insist that the only way that Racism can be eliminated is to force white supremacists to give up their privileges, because to do so requires sacrifice on white people's part with no benefits. I think that abolishing racism would benefit both white people and black people, although of course it would benefit black people more. But a lot of anti-racists think this view is unrealistically optimistic.