A lot of Czech people use this phrase in a very subtle, almost undetectable jest, but it is still fundamentally in jest. Drawing a connection between having to work for a living and likening it to being forced to work. It's similar to saying "Back to the salt mines" meaning back to work.
I don't doubt that, you're probably from somewhere around Ostrava. There are of course regional differences. I'm from Brno and instead of "jít do práce" we usually used "jít do hokny", "jít do háčku", or just "jít makat". Whenever I hear word "robota", I connect it with my grand-grand-...-father who was forced to work ("robota") by "dráb" (overseer) and didn't like it. So he cut overseer's head with his scythe, became a village hero and founded a church there.
