They matter. My students are studying diction these days, and whether it's "rain" or "bad" in A Farewell to Arms or colors and a rich vocabulary in The Great Gatsby, words matter. We think a lot about how writers use words to create particular effects, but we don't often think about the effect of some of the words that we throw around has on others.
I am particularly sensitive to some words because of my experiences in life. And, while we won't be able to predict everyone's personal sensitivity, some sensitivities are shared by so many that we cannot continue to ignore those effects. When used in a derogatory way, the meaning of a word changes. The poison spreads and then, there is no good, positive context in which the word can ever be used.
So listen to the people whom this harms the most. Stop the R-word, in any context, around any person. It is a poisoned word and should be allowed to die. I will never say it again.