If the negative reaction to the phrase becomes widely spread, of course I would stop using it. This is just not a hill worth dying on, and I try at all times to cause as little suffering as possible. But do we really want to lengthen the list of "offensive" words to include phrases that almost no one is actually bothered by? Especially when there is no possibility for any of these alleged historical connections being true. According to the link in my other comment, the first use of the phrase "low-hanging fruit" was in in 1909, and the song was written in 1937. I have never seen, in this article or anywhere else, any use of the phrase "low hanging fruit" that connects to lynching in any way. This article just asserts that the connection exists. It does give any examples of someone using the phrase that makes the connection.

I'm not saying I have a vote on this. But if the majority of Black people don't find this phrase offensive I think it's a bad idea to ban it "just to be safe". I don't think we should purge a phrase out of the language just because one or two people are bothered by it.