Prototype.js seemed to be targeting the problem of providing a familiar class system, standardizing some functional features, and some utility functions. It was something else, and I loved experimenting with it and reading the source, but ultimately, I think fewer people were finding this to be their pain point with the language or that using it made them significantly more productive.
Of course, considering the fact that they seem to solve different problems, it may be an interesting question why they were fighting for mindshare in the same space.
Because Prototype does what the name says, i.e. pollutes the prototype chain. Also, Prototype tries to monkey 'classical inheritance', whereas jQuery encourages chaining, which obviously people like more.