
AWS OpsWorks - Flexible Application Management in the Cloud Using Chef - jeffbarr
http://aws.typepad.com/aws/2013/02/aws-opsworks-flexible-application-management-in-the-cloud.html
======
newhouseb
Wow, I have that sinking feeling that this OpsWorks stuff may have turned what
was a coin flip decision we made between Chef and Puppet (in which we chose
Puppet) to a one heavily weighted towards Chef, but hard to say without
reading more.

My immediate question is how easy it would be to keep your Chef recipes in
source control somewhere and have equally easy one click deployment to both
AWS and to a local Vagrant development box. We currently have a similar
environment set up with puppet and it makes testing sysadmin stuff pretty
awesome.

Edit: already finding holes - can't deploy to cluster compute instances (a
deal breaker) and deployment seems rather slow (15 minutes to deploy from
scratch versus about 8 with normal puppet) :(

~~~
harshreality
Why would OpsWorks' choice of configuration and deployment management software
affect your own choice, unless you want some sort of hybrid?

I think this will be a boon to AWS users who are currently deploying manually,
but I don't think very many people who already use deployment/config
management software will want to switch and rely on AWS.

I don't think it's a puppet/cfengine/ansible/salt killer,

~~~
newhouseb
I think the big advantage here is really just how much you just get out of the
box without sacrificing flexibility -- if you're deploying on AWS in the first
place. This seems like it would also save us from writing a bunch of
management scripts (around scaling, etc) which would be tied to AWS anyway
(read: no inherent disadvantage since any solution in-house or not would
require the AWS platform). I'm always looking for ways for us to reduce our
code/services footprint and this would seem to do so without a huge amount of
vendor lock-in.

------
mryan
This is the most interesting news I've heard from AWS in a long time. I don't
mean to suggest that they have not produced any exciting news in a while, but
that this is service is so exciting because it will fundamentally change how a
lot of people use AWS.

Of particular interest to me is how this will affect platforms like Heroku.
After the recent routing mesh debacle a number of people contacted me to
discuss migrating their application from Heroku to AWS.

My Heroku -> AWS migration procedure is already pretty straightfoward:

\- identify Heroku services

\- write Puppet manifests to manage EC2 instances

\- write CloudFormation stacks to control other aspects of the infrastructure,
e.g. autoscaling and S3 buckets

\- write deploy scripts so clients can continue using git-based deploys

OpsWorks is only going to make this migration easier, and will have the
additional benefit of letting my clients manage more of their own
infrastructure without my help. Training people to change autoscaling group
configuration via a web interface is one thing, training them to update
CloudFormation stacks is a little less user-friendly.

From the clients I have spoken to, the key benefits of Heroku are a) git-based
deploys and b) easy scaling configuration by a web interface. All of these
things are possible with AWS, albeit a little more difficult to set up -
especially in the case of scaling configuration - and OpsWorks is going to
close the gap even further.

If AWS can provide most of these services directly, will people still be
willing to pay the Heroku premium?

There is one downside to the OpsWorks announcement, at least from my personal
perspective. I'm currently writing a book on AWS System Administration, and
I've almost finished the chapter describing various Puppet workflows. Like
many other commenters I was undecided whether I should focus on Chef or
Puppet, given their relatively similar feature set and popularity. I'm not
sure if I've backed the wrong horse here and should rewrite the chapter using
Chef, or if having a Puppet-perspective would be useful. Any thoughts?

~~~
jrs235
I don't think you should throw out the stuff on Puppet. Perhaps expand and add
how to do it with Chef too.

~~~
mryan
Thanks for the responses folks - it seems like a combined approach is
preferable.

I'll be doing a ShowHN as soon as it is published :)

If anyone wants to make suggestions about other content you'd like to see in
the book, please let me know!

------
Argorak
The Amazon blog post is not very clear on this, but Werner Vogels post is:

[http://www.allthingsdistributed.com/2013/02/aws-
opsworks.htm...](http://www.allthingsdistributed.com/2013/02/aws-
opsworks.html)

This was previously known as "Scalarium", which was a third-party solution
developed by Peritor that added several features on top of AWS, including
management of instances using chef, infrastructure planning etc.

------
Smerity
AWS have owned the base cloud market with EC2 and S3. What I find impressive
is that they're continuously lowering prices competitively[1] and expanding
their services in a way that makes their base offering more attractive.

OpsWorks starts to battle one of the biggest reasons to go with PaaS[2]
providers such as Heroku: the difficulty of managing these services yourself.

[1]: This benefits both Amazon and the consumer. Lower prices and continuous
reductions improves customer confidence in and use of the AWS platform, whilst
consumers are just happy they're paying less. Heroku and Google App Engine
(even VPS providers) rarely decrease prices, making AWS something certain to
be considered.

[2]: Accidentally wrote IaaS -- thanks Argonaut.

~~~
argonaut
Nitpick: Heroku is a PaaS.

EDIT: Second nitpick: AWS OpsWorks is no Heroku. Amazon's already released AWS
Elastic Beanstalk, though, is supposed to replicate Heroku's functionality
(there's a reason Heroku is still around, though).

~~~
aalbertson
Yeah, I noticed some of the similarities to Beanstalk myself and am kind of
curious where they are going with this concept? It seems to me that instead of
building a fully integrated Chef management platform with fancy gui (which was
my initial thought), they just recreated Beanstalk with some major bonus
features and perhaps a lot more control.

------
DenisM
I wonder what is the easiest way to automatically snapshot EBS volumes? I
don't want to run a script - scheduling, managing access keys, etc are all too
much hassle. Besides runing the script on the machine itself is unreliable - I
want the setup to take one last snapshot after the instance dies.

Ideally I would right-click on an EBS volume, select "backup periodically" ->
"every 10 minutes" -> retain each for three days, and then dailies for 30
days" and be done with it.

~~~
j2d3
You can do this with ylastic - "scheduled tasks" - it's $25-50/mo (depending
on the plan) for managing your whole AWS infrastructure and is basically an
alternate WEB UI that autodiscovers your AWS footprint. Has some nice ways to
manage autoscaling configurations as well, and also migrating AMIs between
regions. I do not work for ylastic or anything but have used it for 2
different clients and used the "scheduled tasks" feature to do periodic
backups of EBS volumes just as you've described you'd like to do.

------
themgt
Amazon is still committed to solving the wrong problem, which is "how do I run
a lot of virtual machines", when the actual problem is "how do I
deploy/scale/maintain complicated applications"

Chef is an excellent fit.

~~~
Argorak
Amazon AWS is a low-level platform, not necessarily a full solution. They do
solve the "right" problem, but not necessarily yours. Thats where Heroku and
(the now obviously aquired) Scalarium come in.

------
obilgic
Bad news for heroku...

------
aserdp
What has always bothered me is AWS comes up with a service which is half
cooked and doesnt answer the true requirement of the application. I have used
almost all of AWS offering in different contexts and every time, when we
really needed the action, it failed miserably.

I believe, AWS should concentrate on its core offering and that of being a
awesome IaaS. It needs to fix it severe and long downtimes and many such core
issues, unpredictable network performances, etc

------
danielhunt
Word of warning - the step-by-step deployment of a PHP application is _full_
of errors, and the application itself doesn't work as expected from the
document.

[http://docs.aws.amazon.com/opsworks/latest/userguide/getting...](http://docs.aws.amazon.com/opsworks/latest/userguide/gettingstarted.walkthrough.phpapp.2.html)

It needs some massaging on the web servers in order to actually respond
without a 500 error

~~~
devopstom
I'm currently writing a blogpost about how to spawn up a simple Rails stack.
I've got a broken instance in a boot-stop-terminate-boot loop. It's very beta.

~~~
danielhunt
I was hoping to do the same (albeit much easier to follow than their own
walkthrough), but I'm going to sit back for a while and see what develops.
These could be early teething problems, so I'm not going to judge them so
soon.

That said, I'm surprised by what's _not_ included in the main interface (VPC,
ELB, to start with).

~~~
aalbertson
you hit the nail on the head (oppositely, I was surprised to see them deploy
out HA Proxy by default). The lack of VPC makes this far less appealing. I've
been working on a scaling capable VPC configuration with quite a few moving
parts and got so giddy with excitement over the potential of OpsWorks, then I
tested it out and realized it's nowhere near ready for prime time and is
missing some major players.

Still...Props to them and am excited to see it mature.

------
j2d3
<https://aws.amazon.com/opsworks/faqs/#amis> : "Q: Can I use my own AMIs? No,
however you can customize the AMIs OpsWorks supports using Chef scripts to
install agents and other software that you require."

This makes OpsWorks useless if you want to autoscale with any reasonable
"nimbleness" - meaning the amount of time it will take to wait for a non-
custom AMI to be bootstrapped with chef client and then load and run all my
standard and customized cookbooks is way too long... I need to be able to
specify custom AMIs that are already largely prepared so they can boot fast.
Also, the Chef bootstrap process is brittle. twice in as many weeks it has
been broken by dependency fails. (First, a net-ssh dependency, then a JSON gem
dependency).

------
bashtoni
Looks interesting, but it seems strange to me that it's something of an
island.

There's no integration with Elastic Load Balancer, no integration with RDS or
in fact any of the other AWS services except for deployment from S3.

~~~
nsoun
Yeah, I spotted that too, it's interesting that this service puts the focus on
coordination + orchestration with additional app servers versus leveraging
existing AWS services.

I believe the general intent of this is to enable users using non-AWS services
(yes, I know that's redundant) to more efficiently work with outside apps on
Amazon - the example in their Layers guide is specific to Redis for example.
([http://docs.aws.amazon.com/opsworks/latest/userguide/working...](http://docs.aws.amazon.com/opsworks/latest/userguide/workinglayers-
custom.html)).

So this service basically says - "Just because we don't have a Redis Amazon
Service for you to snap into doesn't mean you should not use AWS or look to
other PaaS, instead, easily integrate other apps with OpsWorks."

------
jondot
Something in my gut tells me this is a game changer, and we may yet to see a
great PaaS/IaaS consumer show in the next weeks/months.

Also, feels like a great move with Peritor/Scalarium and Chef.

I'm definitely grabbing the popcorn!

------
tbatchelli
I don't see any press release from OpsCode, which makes me think this is not a
joint venture with them, which in turn makes me wonder how dangerous is to be
part of the AWS ecosystem...

If you prove a market around AWS, what are the odds of getting this market
gobbled by AWS themselves? They have a level of access to their own
infrastructure that you don't have, let alone the access to their own (huge)
customer base, integrated billing and branding.

~~~
rgbrenner
Amazon has been repeatedly accused of exactly what you describe with Amazon
Marketplace. This is the first article I was able to find, but I've seen
similar articles over the past few years (One of my companies is an ecommerce
site that considered selling through Amazon marketplace)
[http://online.wsj.com/article/SB1000142405270230444140457748...](http://online.wsj.com/article/SB10001424052702304441404577482902055882264.html)

So it would not surprise me if they used this same tactic with AWS.

Edit: here's a google news link to the article:
[https://www.google.com/url?sa=t&rct=j&q=&esrc=s&...](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=newssearch&cd=1&ved=0CC0QFjAA&url=http%3A%2F%2Fonline.wsj.com%2Farticle%2FSB10001424052702304441404577482902055882264.html&ei=kcAjUf2MBObVyAHEzoGgBQ&usg=AFQjCNHh6eNon3AU-e6dKXJWVMv4ieDPEQ&sig2=P2QizYFUsceZCT0BghYQjg)

~~~
tbatchelli
I doubt OpsCode is getting much revenue from OpsWorks, or any, since AWS is
giving it out for free. It's hard to compete with free. This tweet from
OpsCode reads like they were blindsided and trying to catch up
<https://twitter.com/opscode/status/303941706888409089>

------
danielhunt
If you want to undo your setup and wait for a while for this system to mature
a bit, you need to delete the security groups that are auto-created in your
account in this order:

AWS-OpsWorks-Monitoring-Master-Server

AWS-OpsWorks-DB-Master-Server

AWS-OpsWorks-MemcacheD-Server

AWS-OpsWorks-Custom-Server

AWS-OpsWorks-Blank-Server

AWS-OpsWorks-PHP-App-Server

AWS-OpsWorks-Default-Server

AWS-OpsWorks-Rails-App-Server

AWS-OpsWorks-nodejs-App-Server

AWS-OpsWorks-Web-Server

AWS-OpsWorks-LB-Server

 _edit_ 1 or 2 of these were typed from memory - but it should be clear at a
glance which one is which in the EC2 console

~~~
danielhunt
For the record - don't do this. The system doesn't seem to like it at the
moment

------
narsil
The lack of VPC support and inability to configure custom security groups
would prevent me from using it for right now. It seems like a good alternative
to PaaS out there though, and I'm sure it will be rounded out with more
features in the coming months.

(Note: I wouldn't see us using it in the near future anyway, since we already
manage our machines on AWS with Puppet.)

~~~
cbbarclay
Thanks for the feedback on the need for VPC support. We're listening to
customer input to prioritize our roadmap. You can add custom security groups
to each layer in the layer configuration.

------
danielhunt
I find it _very_ odd that they've built in the HAProxy load balancer option,
but there's no mention of ELB at all

~~~
flog
I had to look into the docs to find the "ELB is not supported at this time"
statement.

~~~
danielhunt
Not only that, but there's no allowance made for micros either.

Also, SSL on a per-app basis is confusing me - does that mean that each box
individually is handling SSL termination, or is it done on the loadbalancer
side? I have a lot of reading to do before I can really understand what's
happening here I think

~~~
devopstom
The lack of micro instance is a real shame. I suspect that's because the time
taken to do a full chef install run on a micro might be vast.

~~~
pravka
I've used t1.micros extensively with chef -- it's not as bad as you might
think.

------
ajtaylor
OpsWorks looks quite impressive from my non-devops perspective. And most
interesting of all is the fact that it's free! I understand that it's build to
encourage more usage of AWS, but there are plenty of 3rd party services out
there which to similar things but are not free.

------
fmkamchatka
Do AWS Elastic Beanstalk and OpsWork integrate in some ways or is OpsWork just
more flexible?

~~~
cbbarclay
There is currently no integration between AWS Elastic Beanstalk and AWS
OpsWorks. You can see the differences between the services here
<https://aws.amazon.com/application-management/>

------
needcaffeine
This still has a long way to go before it is a threat to Scalr / Right Scale,
but as it matures, that is definitely going to happen. Where companies like
Scalr and Right Scale add value is the ability to deploy your cloud across
multiple providers.

------
hnwh
Confused... is this an alternative for chef server?

~~~
cparedes
Sort of, but not exactly. It's not API compatible with Chef server, and I
believe you don't get the nice search syntax for grabbing node information
across your fleet that match some kind of criteria. But you still do get info
on your other machines, and you do get nice orchestration across your fleet.

~~~
neonlex
you get a much better orchestration as you could with chef server you also
have much faster interactions, better life cycles and to the searching... you
get every information of evert machine under your fleet it is much much more
better the chef server

~~~
hnwh
sorry, do you mean chef server is better, or this looks better?

~~~
neonlex
OpsWorks gives you better features, try out to start a small stack and then
also have a look at the default cookbooks, there you can see what you can do
with OpsWorks: <https://github.com/aws/opsworks-cookbooks>

~~~
makerbreaker
Im not clear on why you get better features.

I have a webapp that I built, that utilizes rackspace's API and chef. I have a
chef server that I use for all my configuration management, I have a Rails app
that I use to talk to my chef server to manipulate machines, or rackspace's
api to spin up machines, and I am building "triggering" into it (low disk
space, high CPU do X). I am able to change IP addresses, spin up a machine
with a specific stack etc. Granted this isnt the default chef-server, but the
ability to do all this stuff, and not be locked into AWS is there.

~~~
aserdp
You are right about this, as it is now, it looks more of a vendor lock-in
gimmick

------
zzimbler
Does anyone know if this will support web sockets? Major issue (for me) in
regards to using heroku is the lack of web sockets.

~~~
Kudos
Given that this doesn't support ELBs yet and you'll need to roll your own load
balancer with HAProxy, there's nothing stopping websockets from working.

------
programminggeek
I would be more excited if it supported micro instances, but as it stands, I'm
still very tempted.

------
lipeno
Heroku, GAE and Appharbor, brace yourselves

