Not sure that the definitions posted here are all that correct, but I do agree we see/hear the terms "racist" and "racism" as catch-alls for anything remotely related to race, including the mythical "hate crimes."
Something can be racially offensive to someone, and not offensive at all to another. Offensiveness is up to the receiver's interpretation. It's subjective, unquantifiable, and a matter of opinion. The best example is calling a Black man the N-word. That word exists in all kinds of movies, music, Hip Hop (not to be confused with music

), and literature. Context and the person speaking or writing it matter, but it can still be offensive to some, not to others.
The basic rule I learned is to not use language that MAY offend someone **IF** you care about offending the people you live and work with.
Offending someone isn't a crime, nor is protection from being offended a right. If the comedian in question is offensive to you, you can leave -- even ask for your money back. But don't blame the comedian. They are working their art form. Censoring them so people who aren't offended can't hear/read/watch them is fundamentally unAmerican.
So, racist? Nope. Racially insensitive and potentially offensive? Absolutely. That's neither right nor wrong.
Now, if we were talking about a 4th grade teacher, that's a different story. I wouldn't want my kids learning from someone in authority that's how normal adults talk about people different from themselves.