Browsers have a much harder time dealing with binary XHRs, huge typed arrays, huge strings, and a real fucking lot of arithmetic operations.
I think the most difficult part was the debugging. Jens Nockert, Matthias Georgi & I finally used node.js and carefully diff'd the output from minimad.c (the canonical libmad example) and jsmad.
However, it quickly became obvious that comparing exact values wasn't going to cut it. Listening was also almost useless (try to debug when all you have is SCRRRCHHHHHH vs CRRRRRRKKKKKK). So we plotted graphs. With gnuplot. It was a matrix-like experience but quickly allowed us to see what was wrong.
Really quite an experience!
And I agree, debugging it was a special experience.