You made a dumb joke. You should consider that that joke is only funny to people who are comfortable in their surroundings and among their peers; to the lone latino or black person in the office, they come across much differently. It is a form of privilege to be able to chuckle as you toss around which rap group someone looks like they belong to; when that privilege is flexed so that it alienates coworkers, it is abusive.