If you work around downdown (Bishop & Hotel streets), you see some really wild and woolly ones out and about.
I don't avoid these people per se, but I do avoid making eye contact for any length of time. No sense in trying to initiate a reaction.
As for putting your phone away, it's their perception that creates their belief that they are the cause. How do they know you didn't simply finish your text conversation (i.e. What do you want me to pick up for dinner?).
If you're too insecure to handle a comment by a stranger in passing, then maybe spend more time at a soup kitchen, food bank, etc.
People are human, but some act less human than others. Everyone deserves respect, but not everyone deserves your trust -- especially strangers.