Here is why:
"Why did the chicken cross the road? BECAUSE NEGROES CAN'T READ. HAR HAR HAR HAR!!!!!!! Pretty funny, huh? Oh, come on, lighten up."
^Typical context in which "lighten up" is used . You don't want to sound like that guy, do you? So don't say "lighten up', no matter what race or orientation you are.
If someone finds your joke uncomfortable (like I'm hoping we all found the joke above uncomfortable) trying to cajole them into comfort with it is wasted effort at best, continuation of harassment at worst, and a simple dick move most likely.