Important question is: who did you get in trouble with? If it was a virtue signaling white woke SJW, I think you should reply by saying it is not their place to decide what is racist and what isn't. If it was a black person, you still have to consider the possibility that this is this particular individuals pet peeve, and not a universally accepted attitude among Black people. It would be polite not to use that expression around the person who reacted that way. Heck, if they wanted you to never say the word "cucumber", it would be courteous to not say "cucumber". But we can't say that nobody should ever say anything that ever offends a single person.