A bunch of things. First and foremost: cultural representation. There are many things that are simple words in Chinese that have no English equivalent. The values and traditions of a culture are subtly communicated via its language, and even if we can instantaneously translate the literal meaning of what is being spoken, subtext will be lost.
This is why at a high level (beyond "where is the restroom") translation is a highly involved field.
Secondly is the usability of this system. Even if it is 100% accurate and immediate you'll just end up with the UN problem: communication becomes asynchronous because you need to wait for the translator. You'd say one thing, the other person would listen to your translator. He'll say something back, and you get to listen to his translator. It's a hell of a lot better than nothing, but there is still a tremendous advantage to being able to converse in real-time.