The HTML5 canvas and audio API's are fairly simple and easy to coordinate. You probably do not need a library unless you require complex collision detection, or physics math.
I Agree. A library you use might not align 100% with the type of game that you are developing, slowing you down, especially if you target mobile.
The game I have built with web technology, Orbium (http://jsway.se/m/), I choose to use no libraries, the HTML5 stuff that already exist in the browser is already very powerful and easy to use.