Other languages have this sort of test service available for open-source modules. One example is cpantesters.org, which is integrated into Perl's package repository. Not only is it easy to check what passes in your environment, it also makes it easy for module authors to fix bugs in environments that the author has no (other) access to.
