For both money and race, it's not the benefits that come from playing the game, but the dangers that come from not playing it. Nobody is going to accept your potatoes if you try to exchange them for other food at Safeway. If you tried to drink from a white water fountain in Georgia in the 1950s, you would get lynched.
Of course, If enough other people agree with you and stop playing the game, you-all could help slow the game down and perhaps grind it to a halt eventually. As I said in my other comment on this post, I think intermarriage is going to be inevitable now that enforced segregation is abolished. When 12% of the population intermarries with the rest, the physical differences that made segregation possible will be so diluted that both race and racism will have to disappear. So called "Blackfishing" and "Cultural Appropriation" will speed up this process by insuring that even people who don't have any of those physical and cultural characteristics will voluntarily acquire them.