* Puppet's error messages are next to useless. Sometimes it can only tell you which file had an error, and not which line number. Sometimes you'll be left wondering where in the many files that make up a complex project there was an error.
* The only way to actually test the correctness of your puppet scripts is to deploy on real servers. Your staging environment has to be identical to production if you don't want any nasty surprises.
* Despite being written in ruby, the design doesn't follow ruby's philosophies or best practices. The config language is not dynamic or ruby-ish. Compared to ruby, you get code blowout when you can't use DRY and have to unroll loops to fit inside the bounds of what the config language can do.
* Simple things are hard. You have to write your own modules or use 3rd party modules for what I consider simple things, such as editing an existing system config file to add two lines in the middle if they don't already exist.
* There's a steep learning curve because a large project is like spaghetti code where you aren't quite sure which file to find what you're looking for. It is very easy to create dependency loops, and hard to find them, and even harder to resolve them. I ended up generating the dependency graph to a PNG file and eyeballing it, several times per day.
I got the impression that puppet is ideal for getting a "PHP+MYSQL" server up and running that requires almost no deviation from the operating system defaults. If your setup is a more complex one I'd stay well away.
At first blush, future-proofing against lower-level AMI changes and maintaining instance role flexibility don't seem like a good tradeoff against increased startup time in most cases.
I often spend way more time than i expect to when configuring a new server, and have recently been very interested in ways to speed that up.
I don't know if there's a puppet equivalent to chef-solo but using it with vagrant and VirtualBox is my favorite way to develop chef recipes. I did short presentation about this a few months ago, see https://docs.google.com/present/edit?id=0AVIsoyl05MRmZGNwY2N...
BTW, the OP talks about needing a separate security group for the puppets but last I checked, the default security group on EC2 makes the machines you own visible to each other and nobody else's that you haven't explicitly permitted. So I think all of that was unnecessary.
Another light-weight alternative for configuring new servers is Babushka: http://babushka.me/
The other option is to have the instance sign its own key, much the same way I have the instance set itself up in Cacti.
If anything, I'm going to 'port' this post to an bcfg2 example.