
Flynn: open source PaaS - cheiVia0
https://flynn.io/
======
corobo
I've been using Flynn for a while now. Unfortunately there's always something
that entirely bricks the cluster eventually. Sometimes a node will reboot
unexpectedly (as happens with servers) and the cluster slowly implodes from
there, sometimes I'll just get a monitoring alert that all the sites hosted on
it have gone down for no apparent reason.

They announced during their 1.0 release that Flynn is now production ready but
I have to disagree at this time. I do honestly love the idea of running my own
Heroku but it's just not stable enough and doesn't recover well if a node goes
offline without manual intervention using the flynn-host fix command (and
sometimes that command just chokes up completely).

Next time it happens (when, not if, at this rate) I'll try to set aside some
time to gather up the logs to help debug the problem but I'm using Flynn to
avoid the sysadmin side of things and just build my product. In my opinion
Flynn is tinker-ready, not production-ready. It's infuriating that it has
these random problems because like I say I really _really_ love the product
and desperately want to be able to use it reliably.

~~~
merb
> Sometimes a node will reboot unexpectedly (as happens with servers)

if it's coreos it's probably the auto update mechanism.

~~~
corobo
Not in this case, Flynn runs on Ubuntu 14.04. I think the reboot was a
DigitalOcean droplet's host being restarted at the time though I can't be
sure.

------
duncanawoods
Question for the Flynn team:

For us to invest time/effort adopting a PAAS solution requires an amazing
amount of trust in your ability to execute now and in years down the line.
Unlike alternatives, you are a company and not an industry collaboration /
foundation, how open can you be to make us believe you are not the next
RethinkDb-like shutdown?

e.g. How many in your team, why can they do this, how much funding do you
have, what is your burn, when do you become default-alive?

~~~
danielsiders
Thanks for the great question. I'm the CEO & co-founder of Flynn.

tldr: we're open to making Flynn a foundation/community project if we could
find the right kinds of partners and once the core innovation is complete.

This is something we've thought a lot about over the past few years. Everyone
on our team is a hardcore open source believer and contributor and the long
term success and sustainability of open source projects and Flynn in
particular are something we care deeply about.

When we started working on Flynn (with just a crowdfunding page posted to HN)
we had no intentions of commercializing it. Before asking HN to fund the
project we reached out the founders of many growing startups (many of which
are now unicorns) who we saw as the core users. Our goal was to agree on an
architecture and set of APIs, then have each company "own" one component of
Flynn, with one engineer at each working on their component full time.

Everyone we talked to said they loved the idea of an open source Heroku that
could run anything (not just 12 factor apps) and run anywhere from AWS to
their own colos and datacenters, but none had the staff to spare on
infrastructure engineering (which is why everyone wanted a PaaS in the first
place). Several, like Shopify, contributed to Flynn financially so we could
hire a dedicated team to build Flynn for everyone.

So a community/industry collaboration was absolutely our goal from the start,
but several things made us question that route. Not only were our first
choices for collaborators and partners unable to participate directly, but we
spent a lot of time talking to companies in the then-thriving OpenStack
community as well as other foundation/industry run infrastructure projects.
They shared a number of horror stories of how the most promising new features
and capabilities were killed by cross-vendor compatibility concerns and how
specs were limited by legacy vendors on committees.

Committees and bureaucracies are great at keeping something alive and
maintaining the status quo, but not at innovation.

We also met executives from very large companies in regulated sectors like
banking and telecommunications. They were extremely excited about Flynn but
had a completely different set of requirements than the younger companies like
startups whose needs we knew best, and had experienced ourselves. It was clear
that we would have to build a very different project to serve the largest of
enterprise customers. Crucially, foundations/industry collaborations on
infrastructure attract the largest of vendors who have these mega-companies in
their sights as the real customers.

If you want to see what a PaaS built to their (imagined) needs are, look at
CloudFoundry built by Pivotal, which is currently owned in part by Dell/EMC,
VMWare, GE, Microsoft, Ford or at OpenShift, which was built as a startup but
acquired and retooled by RedHat.

While we think Flynn has a lot to offer today and in the future to these large
institutional users, they have not been our focus.

Flynn today has the _minimum_ feature set we consider necessary to be useful.
Looking towards the future we want it to encompass a lot more. We'll be
publishing a comprehensive roadmap/master plan soon, but most of that
innovation would be harder or impossible to accomplish if we were locked into
a foundation today. Development would be slower, changes would have to work
their way through approval to consensus, and many partners would object to new
features competing with their own established products.

We've been very careful both to limit external dependencies and to provide
end-to-end integration between all our components. As a result we're able to
move very quickly when changing APIs, implementation details, even major
pieces of architecture, all without impacting customers. This gives us the
freedom to stay nimble in a changing landscape, but partners might be more
attached to technologies in which they already had a large institutional
investment.

Infrastructure companies today are already a world apart from where things
were in 2008 when Heroku upended PaaS. The default is now open source, making
lock-in (historically the bigger concern) far rarer. There are few companies
among our peers battling to become the next Oracle.

Additionally there's a tremendous amount of portability both among unified
PaaS's and build-your-own infrastructure because of technologies like
buildpacks and containers. As a result most apps that run well on one platform
can be moved to another. There are of course features, like Flynn's database
appliances, that are not available on most others.

We've made other choices like a small codebase with few external dependencies
that make managing Flynn's development and maintenance easier, whether it's
our team or yours doing that maintenance.

That being said, we're entirely committed to Flynn's future personally. While
we'd love to see it become a huge company, that's not necessary for us
personally. We're a team of five, and speaking for at least my cofounder any
myself, we're not going anywhere, regardless of the VC market.

We work on Flynn (for several years and many long hours now) because we
believe it needs to exist in the world, not because we think it'll make us
rich. We're committed to keeping Flynn alive as long as that stays true
(basically until it becomes unnecessary or there's something else that does
everything we want but better). Since our 1.0 launch we've had huge growth and
adoption and are currently fundraising. We're single digit new customers away
from default alive with the current team. Whether we become a small,
sustainable company to selling to/partnering with a large enterprise vendor,
to creating a foundation, everyone on the team is 100% committed to a
permanent, sustainable future for the product, with or without venture
capital, unicorns, or IPOs.

~~~
duncanawoods
Thank you for the detailed response - much appreciated. You have tempted me to
look into Flynn further.

The headline feature that attracts me is HA postgresql - that has instant
relevance. My greatest concern is stability - I'm not seeing many success
stories but lots of problems. I want to hear that Flynn is "the one bit of my
stack I don't have to worry about". I hope you can publish some case-studies
(both successes and post-mortems) including discussion of the attempted
architectures.

A fundamental philosophical worry I have is that it is a "wrapper tech". I
increasingly stay away from these - the promise is that they make the
underlying tech easier/quicker to use but they make assumptions that don't
hold for you in the long-term, they serve too many unrelated use cases so
become bloated and poorly fitted to your own needs and they reduce the
features, flexibility and stability of the tech they wrap. The cost in fixing
& working-around the wrapper can quickly exceed the cost/value of building
your own glue based on the underlying tech because it also gives you a greater
depth of knowledge in the fundamentals and more power.

------
erowtom
Experienced Flynn last year, we spent about 2 weeks making it work properly
with our solution (We had a complex SaaS solution hosted on Heroku, with tons
of addons).

We had some very bad bug that happened at the time (like deploying a new
version made the dyno crash and we had to deploy a completly new environment).

Other than pretty bad bug, it was working properly, and did what they sell.

Good point is that the dev team is very responsive and we reported a bunch of
bug that got fixed in no time. They also have a smooth updater when a new
Flynn release came out, side bad note is at this time, it was kinda bugged
(because of our specific setup).

For all those reasons, we choose to stick with Heroku at that time (we also
tried Doku [https://github.com/dokku/dokku](https://github.com/dokku/dokku),
various AWS services such as BeanStalk, and DEIS
[http://deis.io/](http://deis.io/)), but for sure it was the best candidate
out there and I would love to give it a try again.

~~~
_lmars
We have pushed a ton of stability fixes recently, and we now have a stable
release channel which we release to monthly (see
[https://flynn.io/docs/stability](https://flynn.io/docs/stability)).

I highly suspect many of the bugs you encountered have been fixed, and we make
it a priority to investigate and fix reported bugs as quickly as possible, so
if you end up giving Flynn a try again let us know if you have issues and
we'll fix them.

As for updating, the in-place updater still has some kinks but the backup /
restore route is fully tested and is reliable (see
[https://flynn.io/docs/production#updating](https://flynn.io/docs/production#updating)).

------
rcarmo
I love this kind of thing so much that I built my own:
[https://github.com/rcarmo/piku](https://github.com/rcarmo/piku)

(Originally coded to deploy services on the Raspberry Pi, but now used on
"real" machines too)

------
harigov
This is really cool. I am not sure if there is a way to deploy this on public
cloud like AWS/Azure. If this were cloud agnostic way of building software, I
would be totally into it.

~~~
joshmn
That's kind of the whole purpose of it.

> The CLI includes a local browser-based installer that can boot and configure
> a Flynn cluster on Amazon Web Services, Digital Ocean, Azure, and your own
> servers via SSH. It automatically performs all of the steps required to
> install Flynn.

[https://flynn.io/docs/installation/cloud](https://flynn.io/docs/installation/cloud)

~~~
harigov
Thanks! I know this is slightly different from what Flynn is trying to do, but
what I was looking out was a way to use standard cloud provided PaaS rather
than using cloud as IaaS. This approach may work fine in the short-term but
cloud native services provide a lot of extra functionality (monitoring,
security, sandboxing, eventing, logging, encryption, etc.,) that would be hard
to replicate for something like this. An abstraction to those cloud native
service, OTOH, may be useful against lock in. I believe it should be possible
given that most of the cloud native services across different clouds provide
similar set of features.

------
exratione
I've worked with a group using Flynn to host a score of internal apps. It
required one full time developer to keep it running, and saved a bunch of time
and effort for the people maintaining those apps - no need for devops, pretty
much, as deployment boils down to a ten line bash script. So if that tradeoff
works for you, you should consider using Flynn.

In this age of it being next to impossible to hire devops folk, the idea that
you can have one of them servicing the devops requirements for a sizable
fraction of developed applications is somewhat attractive.

I wouldn't put Flynn in place for external-facing and high-traffic
applications at the present time based on the experience I've had with it,
however. Not ready for that, I think. You have to have a fair tolerance for
short outages.

------
epynonymous
having worked on cloud foundry platform in the past, paas was extremely
exciting at the time (2012), we even came up with something called warden
which was way before docker. but since then, i have to ask, is there really
that much benefit to having a paas? i understand the arguments about not
needing to have devops/sre's, etc, but i think iaas works just fine, sure
there's the initial plumbing for things like load balancers, installing an
operating system, upgrading/patching that operating system, securing all the
network connections, but it's not an everyday task. cloud foundry at the time
required a minimum of 20 vm's to get the environment running (cloud
controller, bosh, service nodes, dea's, nats, etc), and these weren't your t1
micro aws machines, these were memory and processor hungry instances; to
manage the entire environment required a team of sre's and devops. nice to see
that flynn only requires 3 nodes minimally, but as the environment scales,
you'll need more nodes and i wonder how the overhead scales with flynn.
obviously for small environments (let's say 6 vm's, an additional 3 vm's to
support flynn is not ideal). but if you get up into the 20+ vm arena, perhaps
the overhead starts to pale in comparison?

perhaps it's just me, but the devops part isn't that annoying, yet, but i've
got an ultra small environment running nginx, golang binaries, and some
services like rdb and k-v store.

~~~
epynonymous
not sure why this was down voted, but whatever, my questions remain and if
anyone is willing to answer, i'd appreciate that.

~~~
duncanawoods
Looks like a decent question, I upvoted.

I think the main motivation compared to IAAS is to run on your own choice of
machine e.g. on-premise or dedicated from OVH/Hetzner, or across different
cloud providers while not incurring the full admin overhead of baremetal. The
reasons can be portability, control, performance, privacy or the potential for
1-2 orders of magnitude cost-savings.

And yes, scale, high availability, self-healing, auto-scaling & load balancing
is where the admin pain comes in.

~~~
epynonymous
thanks duncan for the reply, i am still not sold on paas as a long term
solution, my reasoning being as such, companies like google that have their
own paas (gae), don't even use it to run large portions of their own
infrastructure. take for instance, facebook as well, they do not use paas as
far as i'm aware for major portions of their sites, and i would say that those
2 companies are representative of extreme scale. so why wouldn't a company of
this scale need/want paas? i guess part of it is that they can afford sres and
devops, but the counter-argument would be that if they could do without
sres/devops, why wouldn't they?

i guess what i don't like about most paas is the fact that you lose some level
of control over the environment for the sake of convenience and bliss, but at
the end of the day, someone has to fix all those bugs in your paas layer.

~~~
duncanawoods
I think you could describe the systems that google/fb use internally as paas
despite not being externally facing - they have internal teams supporting a
platform that then hosts apps and services from 100s-1000s of teams. I know
what you mean about control but things like kubernetes were developed for
Google's internal paas so they do kind of have control despite the level of
abstraction.

For us normal people at more human levels of scale - I agree its wise to be
skeptical - serve your users in the simplest manner and don't get sucked into
HN hype. Its easy to get an infrastructure inferiority complex :)

------
spamlord
How does this compare to Cloud Foundry?

~~~
EngineerBetter
Good question - I dare say it is nowhere near as battle-tested in production.

~~~
EngineerBetter
Caveat: I run a Cloud Foundry consultancy.

A quick glance shows a couple of things:

* Automatic cluster scaling, so your PaaS deployment is as small as possible. Cloud Foundry doesn't have this, and I wish it did

* Flynn has built-in service discovery, in Cloud Foundry you bring your own (if you want)

* Flynn comes bundled with some data services

* Flynn has no BOSH equivalent, so there's nothing there to bring back VMs when they fail, perform rolling upgrades or manage the IaaS

* Flynn doesn't seem to offer a metrics stream, only logs

* Flynn builds apps from source, so not sure how that works for compiled Java/Go, or anything you'd run with a binary buildpack

* I couldn't find anything about role-based access control in Flynn, something Cloud Foundry has (which can be backed by LDAP for enterprises)

* Flynn only appears to route HTTP traffic, so no TCP routing for IoT workloads

* Cloud Foundry is run at massive scale in production for mission-critical workloads by banks, payment processors, IoT data ingesters, manufacturing companies, research companies, governments and everything inbetween. I can't see any evidence of the same being said for Flynn.

* Flynn is owned by one company, Cloud Foundry is owned by a non-profit organisation affiliated with the Linux Foundation and can never be transferred back to a regular company. This prevents the sort of nonsense you've seen in the Docker community recently.

~~~
arekkas
* Flynn builds apps from source, so not sure how that works for compiled Java/Go, or anything you'd run with a binary buildpack

You can deploy docker images, at least that was the case one year ago

* Cloud Foundry is run at massive scale in production for mission-critical workloads by banks, payment processors, IoT data ingesters, manufacturing companies, research companies, governments and everything inbetween. I can't see any evidence of the same being said for Flynn.

I think that's the whole idea behind flynn. Instead of being heavy-weight,
complicated to deploy (no BOSH) and resource intensive, you can run a cluster
with 3 nodes and deploy a ton of services there. It's not targeted at banks,
it's targeted at people who don't want to use heroku or pivotal or GCE. And
this makes it a lot faster in many ways, at least last time I checked.

~~~
EngineerBetter
That's cool about Docker images. Cloud Foundry has a binary buildpack, and the
Java buildpack deals with .jars, which means slightly lower complexity for the
end user than needing to roll a Docker image. CF does Docker too now, which is
nice for compatibility, but I think people are missing the point by deploying
images instead of apps.

Fair points about BOSH, it's got a steep learning curve (cliff). I'd love to
see Cloud Foundry have an install experience that's along the lines of "Hey,
here's some IP addresses and an SSH key. Go install yourself there."

------
Mojah
Happy to see Flynn get more adoption, it got featured in the cron.weekly
newsletter a few months back, when it was in the earlier stages;
[https://www.cronweekly.com/issue-39/](https://www.cronweekly.com/issue-39/)

------
sztwiorok
what gives you more than docker-compose itself? with docker-compose and docker
swarm I can easily add services there, configure the application environment
and scale it.

seems interesting but please enumerate things I can do that are not delivered
by docker compose and docker swarm.

~~~
_lmars
A main advantage of Flynn over using docker-compose is that Flynn is more
aware of how to run applications out-of-the-box, rather than just running
containers which the user has to fully configure.

For example, Flynn comes with PostgreSQL, MariaDB and MongoDB appliances which
are deployed in a highly available, fault tolerant configuration with sane
defaults: [https://flynn.io/docs/databases](https://flynn.io/docs/databases).

Through the use of buildpacks, it is also straight forward to deploy
applications written in many popular languages and frameworks without being
intimately acquainted with how to containerise those types of applications
(e.g deploying a Ruby on Rails app on JRuby:
[https://flynn.io/docs/languages/ruby](https://flynn.io/docs/languages/ruby)).

You also get things like full cluster backups (including all your data),
simple git / docker / tarball based deployments, log aggregation, release
rollbacks, domain / path based routing, service discovery, and many more!

------
splitrocket
The bit that made the decision for me is managing self healing, highly
available postgres in the same infrastructure and tooling as the rest of my
deployment.

I have yet to see any other PAAS come close to offering HA postgres built in.

The second bit was dead simple service discovery.

------
stemuk
Looks pretty cool, do you have any plans to add RethinkDB support?

~~~
Titanous
No current plans, but feel free to request it in a GitHub issue. We consider
the number of votes on GitHub when adding to the roadmap.

------
jondubois
I'm working on a similar service, but focused around my own open source stack:
[https://baasil.io/](https://baasil.io/)

The main issue with deployment/orchestration platforms is that they cannot
handle ALL different kinds of stacks/apps (each one has unique scalability
requirements).

With Baasil.io, our strategy is to get it 100% working and auto-scalable; one
stack at a time.

------
weitzj
Does anybody have experience with Cisco Mantl (Mantl.io) ? It looks like
CloudFoundry or Flynn, but I never saw it mentioned on HN.

~~~
mongrelion
I worked on Mantl for a while last year. To understand the difference you can
see Mantl just as an opinionated coalition of tools for running microservices.
So, Mantl is not a thing but a collection of things (Mesos, Marathon, Consul,
ELK stack, etc) put together using Ansible playbooks.

Flynn is an actual product which, from what I can see, is also a collection of
tools but these are written by the Flynn team themselves.

I'm happy to answer any other question you might have about it.

~~~
smartbit
Thank, makes sense. Feels like Mantl can go with the flow, add products when
they become trendy.

And from the perspective of the developer, does Mantl offer a similar PaaS
experience as Heroku or Flynn?

------
ngrilly
Is it possible to store file in some kind of NFS volume?

Is it possible to store objects in something similar to S3 (for example using
Minio)?

~~~
_lmars
> Is it possible to store file in some kind of NFS volume?

NFS support is not currently implemented, but we have a clear plan and it is
on our upcoming roadmap:
[https://github.com/flynn/flynn/issues/1521](https://github.com/flynn/flynn/issues/1521)

> Is it possible to store objects in something similar to S3 (for example
> using Minio)?

Flynn has a built-in blobstore for storing files and supports multiple storage
backends: PostgreSQL Large Objects (default), Amazon S3, Google Cloud Storage
and Azure Storage (to be released into stable next week), see
[https://flynn.io/docs/production#blobstore-
backend](https://flynn.io/docs/production#blobstore-backend).

~~~
ngrilly
The documentation about the blobstore says PostgreSQL should be used only for
small deployments. What if you deploy Flynn to DigitalOcean, which has no
S3-equivalent?

~~~
Titanous
You can use S3, Google Cloud Storage, or Azure Storage even if you're running
on DigitalOcean.

~~~
ngrilly
Yes, but at the cost of increased latency, and paying the bandwidth?

~~~
DeadBabyOrgasm
There's also expandable block storage
([https://www.digitalocean.com/products/storage/](https://www.digitalocean.com/products/storage/)),
though I'm not sure if it fits these circumstances.

------
tomhallett
Does anyone have any experience with Rancher? I feel like it would fill a
comparable role as Flynn. But I might be off base here.

------
Kiro
Completely OT but was this posted because of the "Remediation Plan for WoSign
and StartCom" thread from yesterday?

~~~
cheiVia0
No, because of
[https://news.ycombinator.com/item?id=12703121](https://news.ycombinator.com/item?id=12703121)

------
omouse
How does this compare to OpenShift?

------
stanislavb
Note: Flynn seems to be the most popular package on "Awesome SysAdmin" under
the category "Cloud Computing"
[https://sysadmin.libhunt.com/categories/1727-cloud-
computing](https://sysadmin.libhunt.com/categories/1727-cloud-computing)

