Launch HN: Armory.io (YC W17) – We Make Deployments Boring and Self-Service

[imosquera]

Hi HN, I’m Isaac, co-founder and CTO of Armory in the YC W17 batch. Ever had a deployment fail only to find out it was impossible to rollback to the previous version? Or get paged at 2am when the user experience is broken?

Armory.io makes deployments boring (like ‘waiting for your code to compile’ boring), non-events that happen continuously, and always in the background. We do that by simplifying the installation and configuration of Spinnaker - an open source continuous delivery platform from Netflix.

We all worked together at our last company and experienced the pain of scary deployments with low engineering velocity. When I became the SVP Engineering there, I decided that needed to change. We broke a brittle monolith app up into microservices, started deploying with Kubernetes, and created a CI/CD pipeline that allowed us to go from deploying just a few painful times per month (taking weeks and multiple manager approvals to deploy) to 2,000x deployments per month, continuously. The cascading effects were that we started working on parts of the codebase that had been stagnant for years because we were afraid to touch them (the original developers had long left the company). Here’s a screenshot of a chart that shows the delta: http://drod.io/43452e3V0N28 — each color is a different microservice; you can see how those blossomed as we broke the monolith up and started deploying each microservice on the cadence that was best for it. And we had much happier engineers, too, because they could see their code running immediately in production, and they took ownership of it running successfully in prod (as a side effect we went from a traditional ops team to a no-ops approach).

We were so passionate about the transformative effect this had on the company that we started Armory to bring it to any company — which is especially needed in large, low performing organizations that typically deploy just 7x per year (compared to 4,000x per day at Netflix).

We’d love to hear your stories of deployments gone wrong, hear your questions about Armory, or anything else on your mind!

[georgeaf99]

Hey y'all, I recently started working on a CI/CD product at an organization where the number of deployments per month is in the same ballpark as Netflix. In my experience, adopting systems for rapid deployments also requires adopting new processes. For example, products that are deployed frequently need to have developers on call during deployment windows. These developers tend to have a specialized skill set that makes them effective at handling botched deployments and bugs in production.

How y'all are planning to approach the cultural changes companies will have to adopt in order to leverage faster deployments?

[imosquera → georgeaf99]

We believe that DevOps encompasses 3 areas that must change: people, processes, & technology. We’re seeing evidence that the people/culture is already changing organically because companies must change in order to compete in the software era, and the velocity at which we see software-first companies entering traditional spaces like banking/finance, real-estate, hotel is also accelerating.

That said, in order to make this rate of change to happen faster, instead of trying to convince people through debates, we have to show proof that moving towards a true DevOps model is successful, not just because you get to deploy more, but because it matters to the business. We want to show that CI/CD helps meet SLAs, new valuable features are added and allows the business to address new markets.

[drodio → imosquera]

To add on to that: The best way we've found to work with large enterprises (aka Global 2000) is to start w/ just one team & app, show success with that, and then expand from there.

We've found that big companies need to see one team achieving success before they'll adopt it throughout the organization.

We've employed this approach successfully with our first several customers. They're going from from one app --> 5 apps --> "GA" rollout internally.

[nodesocket → georgeaf99]

I just founded my third startup Elastic Byte (https://elasticbyte.net), which is a DevOps and infrastructure consulting company operating as SaaS (public monthly pricing plans).

Indeed culture is a big part of the DevOps puzzle besides software. This mentality shift is also a part of what we offer (training and advice). Individual ownership, moving fast, are all parts of the greater strategy. I've found the talk "You shipped it, you fix it" [1] by Ron Cohen (CTO of Opbeat) a great source of solid ideas.

https://opbeat.com/community/posts/you-shipped-it-you-fix-it...

The old model where developers hand off their code to the ops department is fundamentally a bad strategy because it leads to:

   "It worked fine in test... Ops problem now!"

There is no accountability. If your code causes problems when deployed to production, ops is the one getting a call/page in the middle of the night, and trying to fix the problem for which they have no domain experience or understanding. However, if you deploy your own code at your own pace, then there is accountability.

[drodio → nodesocket]

Yeah that mindset shift was the hardest part of transitioning from a traditional ops to a no-ops approach at our last company. The technology has no chance if the culture isn't ready.

We especially see this with F500 companies. They insist on multiple manager approvals for deployments. CI/CD is really scary to them because it feels too risky.

One of our prospective customers (at a F500 company) told us about how they have a cross-functional deployment 'swat team' that all gather in a room on a Friday night and literally press a button to deploy the update, then see what happens. They take their website offline in anticipation of errors (and put up the '90s construction worker logo) and then their ops people get paged over the weekend to deal w/ errors. Just ridiculous!

The irony is that the slower deployment velocity creates more risk, making deployments scary affairs. It's a terrible cycle. Deploying 7x per year means each deployment contains many more features/fixes, which introduces more risk. Vs. breaking a monolith up into microservices, and letting each component deploy at whatever cadence is best for it. And with Spinnaker each deployment can be traced back to the original git commit hash / jira ticket / etc., creating much more transparency in the deployment process.

[techcofounder → georgeaf99]

We’ve seen a few fundamental changes in culture across the board.

1. Product engineers are embracing ownership, and despite being on-call, they are excited at the prospect of being able to deploy their apps whenever they want.

2. Ops teams are transforming into engineering teams. When you start to automate everything (monitoring, deployments, testing, etc), it frees up ops folks to work on more interesting problems within an organization.

[devonkim]

I haven't gotten very deep into the guts of Spinnaker, but how are you expecting to handle the enormous number of organizations that prohibit developers from having production access and require someone in operations to perform a deployment? Even if you do all the necessary steps of API versioning, feature flags, blue / green deploys and such, most of these start to break down fast in velocity when there's a requirement for removing control away from developers to those that have less familiarity with the impact of changes. I absolutely agree something needs to address low-performing software organizations because oftentimes they're among the highest-impact organizations out there such as the Fortune 100 and public sector (as opposed to some SaaS company with a whole 1000 customers) but I haven't been able to crack the problem of many different reasons for dysfunctions leading to such low rates of deployments (it's invariably a Wicked Problem due to being almost entirely social in roots rather than technical I've observed so far).

[imosquera → devonkim]

We believe that engineers should have access to deploy, manage and ultimately own their own deployments. This fundamental change is more commonly known as DevOps. In our interviews with over 100 enterprise companies, before we even incorporated the company, we observed a strong desire for teams to move in this direction. While we are aware there are many laggards, we’re making a bet, with Armory, that companies must move in this direction or die.

That said, it’s an immense challenge working with the F100, which for some, don’t want to move in this direction yet. We’ve chosen not to focus those laggard companies. Even within our current focus, we’ve found that we actually can’t hire fast enough to support new customers who want to truly enable their deployment teams. And yes, it’s very much a “Wicked Problem” to solve but we’re hoping that with Spinnaker and market pressure it helps loosen those roots.

[moondev]

spinnaker as a service. pretty cool idea. what is your planned pricing look like? spinnaker is not cheap to run yourself.

Also I tried the "try spinnaker" link and it looks like it timed out at armory.formstack.com

[techcofounder → moondev]

I'm one of the cofounders of Armory. Thanks for letting us know of the timeout - we're looking into that now.

Today, our enterprise version of Spinnaker is hosted within your AWS account, not truly hosted Spinnaker as a service, although we def plan to offer that down the road. Our current customers have told us they prefer to run Spinnaker inside their own accounts b/c they fear data leakage.

[moondev → techcofounder]

I see, so it's a centralized control plane for spinnaker installation/upgrades. looking forward to trying it out! I've done a bunch of spinnaker + kubernetes integrations myself. Exciting space for sure.

[imosquera → moondev]

As of today, that is main value prop, however over time we hope to add other microservices to Spinnaker that will provide value on top of the control plane.

[techcofounder → techcofounder]

Btw, I realize our website still has "hosted spinnaker" as a product offering. It was the first thing we built, but as mentioned above, we pivoted towards an "on-prem" (in your AWS) version after conversations with lots of customers.

[imosquera → moondev]

While we're not currently hosting it now, we did start with a hosted solution which is why it exists on our website. Engineering leaders were hesitant to give access to their AWS accounts so they wanted us to bring up Spinnaker in their accounts. Sometime in the future we do intend to revisit hosted Spinnaker.

[drodio → moondev]

Ben is going to talk about the 'as a service piece' in a minute. Here's pricing info: http://go.Armory.io/pricing

[drodio]

Here's a deeper dive on Spinnaker, and on how Armory is commercializing it for enterprises: http://go.Armory.io/Evaluate

[IgorPartola → drodio]

I love that there is a service named Igor. Could never find my name on a keychain or a door sign, but now I have a service named after me :)

[lintoma → IgorPartola]

We called it Igor because it was a friend of Jenkins. It's named after https://en.wikipedia.org/wiki/Igor_(character) . There is another service internally called Woodhouse

[drodio → IgorPartola]

ahaha nice -- you can learn more about Igor here: https://github.com/spinnaker/igor/blob/master/README.md