

Buildpacks: Heroku for Everything - lstoll
http://blog.heroku.com/archives/2012/7/17/buildpacks/

======
pnathan
I'd like to point out a Common Lisp buildpack. I've used a fork of it and it's
pretty spiffy. (Now if I only knew a simple app I could give people. :-) )

<https://github.com/mtravers/heroku-buildpack-cl>

~~~
chimeracoder
That's awesome - thanks for pointing it out.

I've only poked around Heroku a bit with basic applications and the standard
stacks (Python w/ Flask, etc.), but I think I'll look into what would be
required to create a buildpack for Racket.

That is, if someone hasn't already beaten me to it...

EDIT: As figures, it already exists[1]; I'll toy around with it and test it
out before suggesting they add it to the list.

<https://github.com/onixie/heroku-buildpack-racket>

~~~
soegaard
Hi.

OpenShift is also interesting. And if you need something to hack on, see

    
    
      https://github.com/RayRacine/rackos 
    

Almost there.

------
jeffh
ActiveState's Stackato private PaaS platform
(<http://www.activestate.com/stackato>) has support for Heroku buildpacks to
deploy (<http://docs.stackato.com/deploy/buildpack.html>), as well as the
usual options for polyglot app deployments (it also has a 100% compatible
Cloud Foundry API). This is useful when you want portability (deploy on a PaaS
that you have full control over).

~~~
philwhln
Stackato rocks!

~~~
emperorcezar
I really wish they would publish some prices though. I'm hesitant of the
"contact us" pricing model.

------
spicyj
I wonder if it's possible to hook this up to something like Vagrant
<http://vagrantup.com> to use a buildpack for local development and easily
install all the required dependencies.

~~~
mwmanning
If you want to use buildpacks locally, check out Mason:
<https://github.com/ddollar/mason>

~~~
spicyj
Ooh, nice -- thanks for the pointer.

------
gee_totes
I'm a bit surprised there isn't a buildpack for PHP. It seems like there would
be a demand, since PHP it so popular (I know I would definitely use it).

I wonder if it's because modern PHP apps (such as Symfony) have some external
dependencies (such as on the APC cache).

~~~
lstoll
There is, it's just not officially supported:
<https://github.com/heroku/heroku-buildpack-php> . Github lists it as having
82 forks, so people are definitely playing around with this.

~~~
sync
Yep. You can even run Wordpress with our fork:
<https://github.com/Americastestkitchen/heroku-wordpress-php/>

~~~
ridruejo
How do you deal with uploaded files/persistent storage?

~~~
sync
S3. Ignore the scary warning, this plugin works great for us:
<http://wordpress.org/extend/plugins/tantan-s3/>

------
leafo
I maintain a buildpack for lua if anyone is interested:
<https://github.com/leafo/heroku-buildpack-lua>

~~~
seclorum
Great! I was looking for exactly this .. thanks for doing that!

------
kfinley
Does anyone have any insight into why Google App Engine has not taken this
route? Are there technical limitations within Google's infrastructure that
would limit it?

~~~
slurgfest
I am not sure what you mean by "this route". App Engine supplies multiple
runtimes (Java, Python, Go). Heroku somewhat analogously supplies buildpacks
for using different languages, but also allows custom buildpacks.

Understand that Heroku and App Engine are fundamentally different kinds of
things.

The commonality is that they are both selling some kind of service to run code
on, and they are both (in different ways) trying to make it easier than just
administering everything at the low level of a VPS or dedicated server, while
also giving you similar levels of isolation and capacity as you get outside
the shared hosting world.

But other than that, they are aimed at quite different goals and the
architectures (necessarily) vary accordingly.

App Engine imposes heavy constraints on your app architecture with a primary
goal of making scaling easier from the outset. You are very much doing
everything inside a framework provided by Google. They are trying to make sure
your handlers do not run too long or do certain classes of insecure things.
And the infrastructure of production is very much opaque to you. You can't
just run any old combination of services, they provide you a set of very good
and very transparent but proprietary APIs for things like memcache and
datastore. You just use them and get billed by usage. While they offer
multiple runtimes and some multi-runtime tricks are possible, they really
aren't trying to cover the 'polyglot' use case or support every language under
the sun. It has good technical merits but your app is almost completely
married to Google. Running a low-traffic app is free but scaling can be a
little costly compared to approaches where you are doing more yourself.

Heroku is not a platform in the same sense as App Engine. That could be good
or bad depending on what you need. It provides high-level interfaces to reduce
most users' deploy and management overhead. But like a VPS, it isn't trying to
limit your flexibility or determine your architecture. It explicitly gives you
control over your mix of services and your own languages and use different
things together. However, in the course of simplifying it also does constrain
and hide details. It just isn't a fundamentally different environment from
normal Linux VPS, as App Engine is - it's only one which you manage at a
higher level using Heroku's tools. You are left with less lockin than App
Engine, but you are still dependent on Heroku insofar as it is supplying all
your deploy/management scripting and stuff like that.

App Engine says something like "do it our way with our tools and you will be
able to scale easily on high grade infrastructure" and Heroku says something
more like "do what you could do on a VPS in a slightly different way, with
slightly less choice of tools, and we can make it much easier. We can help you
cross the scaling bridge later." So they are designed to meet different kinds
of needs and have attracted somewhat different kinds of audiences.

~~~
kfinley
slugfest, thank you for the detailed response. By "this route" I just ment
creating independent isolated environments for the various runtime to operate
within. Based on comments made on the GAE mailing list, I suspect they have
moved in this direction. After reading your post I now understand that the
issue is not allowing indiscriminate code to work in isolation, but to allow
that code to interact with external services E.g. make http requests. Maybe in
the future GAE will provide a generic API for interacting with the various
service, but I don't know what that would look like.

~~~
slurgfest
App Engine just isn't intended or designed for that kind of interaction with
services. Take memcache as an example.

On a VPS, you install an OS package and manage memcached as any daemon. You
are editing the configs and compiling your own binaries if you want. Set it up
on a separate box if you want. Set up a bunch of memcached boxes if you want.
Do the wiring how you want. It's up to you and you handle all the details and
you are billed by time. Handle it correctly and it's crazy fast.

On Heroku, there is a layer of abstraction which makes this more like 'give me
3 instances of memcached'. You are billed by the service-instance,
essentially. You get help making things work well out of the box. But at a
lower level it isn't too different from running deploy scripts to set up EC2
boxes. They just provide the scripts and sell you the resources and support at
the same time.

On App Engine, Google runs a clustered memcache service for you. It is always
running and there is no service setup. It is already on a scalable distributed
configuration, not necessarily the fastest or cheapest but good; micro-
optimizing memcache servers is not even intended as a supported use case.
There is a ton of Google secret sauce involved, behind the scenes you are
connecting to some cluster with RPC, all you know is the simple API interface
they give you. You are billed based on how much you use it.

I think I can predict confidently that GAE is never going to expose an
interface where you allocate individual memcached instances. That kind of
interaction with services just isn't the App Engine way. Google has Compute
Engine for the cases where you want EC2 style control of what is going on at
the per-service level.

If you want to run your own App Engine platform you can try AppScale. (I never
met anyone who does.) It is a whole prescribed architecture and it is not
likely to be changed to become more like Heroku.

~~~
kfinley
Thanks again slurgfest. I appreciate the way that App Engine does thing. What
I ment to say, sorry it wasn't clear, is that by exposing a generic api to
various services memcache, datastore, etc. App Engine could make it easier to
add additional runtimes. And now that I think about it - I'm sure they are
doing this to an extent.

~~~
slurgfest
Sorry that I keep running off in the wrong direction.

They control the runtime in a very heavy-handed, opaque way to lock it down
and I doubt that would ever change. New runtimes will be 100% up to Google to
implement and they seem not to be a high priority in general - Java seems to
be favored for the enterprise people and Go and Python have had involvement
from the respective language creators... Google has only so many major
language creators ;)

Just part of how App Engine is a closed platform really (although it's not
like all of Heroku is open either)

------
btown
Oh, Heroku, how can you have such an excellent build system and yet you can't
support WebSockets? You've spoiled me, and it's so painful to have to use
other services now just because you think the standard is too much in flux.

------
themgt
The vulcan node build server seems to have a handful of dependencies that
aren't public (unless I'm missing something? - e.g. logger, spawner,
cloudant):
[https://github.com/heroku/vulcan/blob/master/server/web.coff...](https://github.com/heroku/vulcan/blob/master/server/web.coffee)

~~~
randometc
The logger, spawner and cloudant coffeescript files are in the lib folder:
<https://github.com/heroku/vulcan/tree/master/server/lib>

When the app is created it sets NODE_PATH=lib so the app can find those files
without specifying the full path:
[https://github.com/heroku/vulcan/blob/master/lib/vulcan/cli....](https://github.com/heroku/vulcan/blob/master/lib/vulcan/cli.rb#L155)

~~~
themgt
Ahh fantastic, thanks

------
gregwebs
what are the differences/advantages/drawbacks between a buildpack and an
automated deploy script that installs OS packages?

What happens in an environment where you have multiple applications with
different but overlapping dependencies?

~~~
zimbatm
buildpacks are designed to work when you have one system/container per
application. The goal is to produce an image that can easily be distributed to
the auto-scaler and associated with the reference OS to provide application
processes. The images are mounted read-only and no guarantee is made on the
process lifetime. This limits the class of applications to ones who don't keep
state internally or on the filesystem.

automated scripts like Chef / Puppet are designed to keep a system in a
predictable state according to some code+data specification. It's not a
guarantee though as things not specified by the scripts can have any state.
They're more useful to maintain an heterogenous park of machines, systems that
want to keep state on the disk like databases or even configure the underlying
system that will run the buildpack containers.

that's how i see it but notice that there are some similarities. it could be
possible to use chef to pre-configure a system image as a "buildpack", or to
add some disk storage to buildpacks. although both not on heroku obviously.

