
Microsoft launches new open-source projects around Kubernetes and microservices - bastichelaar
https://techcrunch.com/2019/10/16/microsoft-launches-new-open-source-projects-around-kubernetes-and-microservices/
======
ubercow
Maybe a better link:
[https://cloudblogs.microsoft.com/opensource/2019/10/16/annou...](https://cloudblogs.microsoft.com/opensource/2019/10/16/announcing-
open-application-model/)

and to the GitHub projects themselves
[https://openappmodel.io/](https://openappmodel.io/)

[https://github.com/oam-dev/spec/](https://github.com/oam-dev/spec/)
[https://github.com/oam-dev/rudr/](https://github.com/oam-dev/rudr/)

------
lnsp
Somehow these enterprise people have turned Kubernetes, which basically was an
advanced job scheduler for compute resources, into a unnecessary complex
monster (just look at the LOC of kube-api). And that's still pretty decent
compared to all the other projects in the ecosystem (looking at OpenShift,
Istio ...). I really don't believe a lot of people have a valid use case for
these projects like OAM when you look at the complexity, failure cases and
administrative tasks these systems introduce.

The one of the main design ideas behind the original Borg system (which
Kubernetes is inspired by) was utilisation of resources and simplicity. Now we
just stuff all the compute power we got with unnecessary proxy systems that
provide minimal value. I truly believe we have lost our way.

~~~
derefr
Fundamentally, k8s isn’t a compute job scheduler—it’s an IaaS state converger,
like Terraform, or AWS CloudFormation. As such, it needs to know how to
model—and manipulate—the state of pretty much any IaaS resource you have.
(And, unlike alternatives, it’s also extensible with custom convergible
resource types, too.)

That doesn’t mean that k8s itself is all that complex. It just needs a lot of
libraries for all the stuff you _might_ do with it, where any given library
will be dead code to 99% of people; and an architecture flexible enough to
allow it to drive and track the state of those libraries.

~~~
jacques_chester
I think Kubernetes-the-scheduler and Kubernetes-the-patterns-with-reified-
examples have diverged and are going to continue diverging.

The latter is more of a microkernel for distributed control systems.

My hunch is that this means OAM may be dead letter in the long run. Abstract
from Kubernetes-as-scheduler, fine and good. But Kubernetes-as-microkernel is
close to doing that already. Whatever my gripes about the details, the
mechanisms and affordances are consistent and predictable. That's very
valuable.

~~~
the_duke
Keeping different perspectives and roles in mind is important here.

I think the quotes in the article make very good points.

I've introduced Kubernetes at multiple companies , usually with good results.

But Kubernetes is a relatively low level runtime. It requires a lot of
knowledge if you want to use it correctly, even with hosted Kubernetes
offerings.

Application developers want to specify how the application works without
having to learn a whole lot about Kubernetes internals. They also want the
freedom to run something on different platforms without a lot of changes.

Think Elastic Beanstalk or Google App Engine, but provider independent. With
the freedom to run it locally , maybe on a cheap self hosted cluster for dev
deployments and on AWS for production. This is a lot of work right now, even
with the hosted Kubernetes offerings.

Also enterprises often want to offload the decision making and rely both on
common standard solutions and outside support if things happen. This is hard
with k8s due to the flexibility of the platform.

I agree that it is too low level and too complex to provide a good company
wide foundation on its own.

OAM could be a very valuable tool to have.

Also the points about Dapr resonant e. Building a complex event driven
architecture with many services is what a lot of enterprises want , but it is
really hard to do it well. There is a similar effort by IBM , but I don't
remember the name right now.

~~~
jacques_chester
I strongly agree with the vision of abstracting away the infrastructure
details, having worked on, around or adjacent to Cloud Foundry over the past 5
years. These days I lurk on the fringes of Knative.

Cloud Foundry is my reference model for the power of clear, safe boundaries
between roles. Good fences make good neighbours.

Right now I'm writing a book about Knative and one of my goals is to require
zero prior Kubernetes experience or knowledge. It's turning out to be trickier
than I'd first thought. I can't tell if that's genuinely because of the close
adoption of idioms and patterns or whether it's just that I've spent a lot of
time around Kubernetes for the past 2-ish years and can't faithfully recreate
my original ignorance.

------
mweibel

      OAM is essentially a YAML file. 
      It can be put in a service catalog 
      or marketplace and deployed from there. 
      But what’s maybe most important, says 
      Russinovich, is that the developer can hand 
      off the specification to the ops team and the 
      ops team can then deploy it without having to 
      talk to the developer.
    

This statement sounds very backwards to me. Isn't this increasing the
separation between devs and ops which we want to get rid of?

~~~
Dev_2019
As a developer I don't want to touch ops. So "we" does not exist.

~~~
ownagefool
I think the idea that a Ops needs to wake up in the middle of the night to
deal with shit outages because a developer is unwilling to stand behind their
code is probably outdated.

Build & Run is becoming very popular.

~~~
mrweasel
We're reaching a point where two groups of people, rather than one, is
required to wake up in the middle of the night. Previously just the operations
people needed to get up, now developers need to be on-call as well.

Operations teams are scaling down their monitoring to just infrastructure,
because the applications are more opaque than ever. Incidents at the
application level are no longer fixable by Ops, because they have no idea what
the developers deployed or how it's configured.

Developers now need to take responsibility for application monitoring, patch
management and incident management. Meaning that we're shifting more work to a
group of people that where already in short supply.

I don't think we're necessarily moving in the right direction. There's
certainly benefits to development and operations working in tandem, but
currently we're just moving operations to developers without much
consideration for the people that needs to do the actual work.

In my opinion your company/solution needs to be somewhat limited for "DevOps"
to make sense. For everyone else, it's two separate roles.

~~~
klodolph
I would say that the problem in this situation is hard to avoid. If you can
have an operations team who are experts on the application as well, and have
the team more closely integrated with the development team, then you generally
don’t need to wake up developers at night. I spent a few years on a team set
up like this and it worked very well. If your project isn’t big enough for its
own operation team you can share a team between a few different projects.

But I don’t think it’s necessarily that much harder to hire developers than it
is to hire devops. Some managers have told me that hiring for devops is
harder.

And developers should feel some of the pain—I’m not saying page them in the
middle of the night, but they should be doing daytime on-call rotations. This
helps align incentives and makes developers aware of reliability issues in the
systems that they create.

> In my opinion your company/solution needs to be somewhat limited for
> "DevOps" to make sense. For everyone else, it's two separate roles.

When I think DevOps, I already think of it as a role separate from
development. You have devs, and then you have devops. You can combine both
roles into one, and that makes sense for smaller / earlier stage projects, but
otherwise I think of devops as a separate role.

Kind of a mess because devops varies so much between companies and isn’t even
consistently named.

But my experience is that it’s not necessary to wake up two groups of
people—it’s either a developer that gets woken up because the project is too
small or too early to be supported by operations, or it’s someone in
devops/SRE/production engineer who gets woken up. There’s a lot of practices
that need to be put into place to make this work, but it’s doable.

~~~
SideburnsOfDoom
> You have devs, and then you have devops.

Better call the second role "Ops" then, and wait for someone to propose
merging it with "Dev" again.

There's so much mental confusion and meaningless use of language now, about
the term "DevOps" that was a fairly simple suggestion about bringing
DEVelopment and OPerationS together. That sentiment is literally in the word,
I don't know how it could be plainer.

------
jeethsuresh
Dapr looks like Microsoft's answer to Istio - anecdotally, I've never gotten
Istio to work (as recently as a few months ago) so I'll have to give this one
a try at some point to see if they've done a better job.

OAM though, perplexes me. Even the justification - that k8s is out of scope of
the developer, and is handled by ops - speaks to a misunderstanding of some of
the advantages clusters provide. Some research [1] tells me that developers
author "components" which encapsulate singular parts of an architecture, and
are connected by operators into an application. And then auto-scaling is
handled by "traits", which are entirely separate? It's interesting, but it
seems to largely replicate (and act as a translation layer for) existing k8s
features. Maybe that's the real value prop here, and the techcrunch article
buried the lede - by defining components and traits through OAM you can move
to any cloud-based model that supports the manifest spec (thus avoiding the
high learning curve of something like k8s). Even so, I have a sinking feeling
that most of this will be folded into the ever-changing definition of devops,
so small teams that manage an app's entire lifecycle (and are told to get on
board with the new-fangled dealio by management) will have another layer of
indirection to debug when something inevitably goes wrong. If Azure comes out
with an implementation of OAM that uses their VSIs, I'll get interested. Then
it'll at least provide real value/choice, and hopefully make it easier to
migrate existing workloads that don't need an entire machine for themselves
onto a k8s cluster.

[1]
[https://cloudblogs.microsoft.com/opensource/2019/10/16/annou...](https://cloudblogs.microsoft.com/opensource/2019/10/16/announcing-
open-application-model/)

~~~
vturecek
The overarching idea with OAM is to standardize the model by which
applications are composed and operated, regardless of the environment you end
up working in. So as you go from one platform to another, you have a
consistent experience and a transferable process. We fully expect the
implementing platform capabilities to differ, and the model is designed around
this assumption. I think some standardization here is valuable.

At the same time, we aim to improve application modeling on systems like
Kubernetes that currently focus more on container infrastructure.

disclosure: I'm one of the spec authors.

~~~
jeethsuresh
Interesting - if you don't mind indulging me a little, it was surprising to me
that you chose kubernetes as the first implementation for this standard. After
all, if a large part of OAM's functionality is provided by k8s out of the box
(auto-scaling, for example) it's not very useful to someone who knows the
existing tech. Kubernetes is, by and large, already portable with minimal dev
resources between the major cloud vendors and/or on-prem resources.

On the other hand, if you can pair already-written software with a collection
of VMs and place an OAM layer between the machines and business logic, the
portability of your code between OAM-compatible vendors becomes a selling
point for the standard. I know a large project like this is a team effort, but
can you shed any light on the reasons behind your decision-making and
prioritization towards kubernetes?

~~~
resouer
OAM is the app mgmt API for K8s, while not only for K8s. So implement it
firstly on K8s is a natural choice. You may argue hey why I need app mgmt API
for K8s. Well, have you ever tried to expose full Deployment api for
developers to describe their app? My personal experience is super messy ...

------
markbnj
I don't see a link to dig into the Open Application Model, and I don't have
time to search for one tonight. Maybe kubernetes can benefit from a higher
level of abstraction than helm charts provides, but I would need to see some
use cases. I did get a kick out of:

>> He also argues that Kubernetes itself is too complicated for enterprise
developers. “At this point, it’s really infrastructure-focused,” he said. “You
want a developer to focus on the app. What we saw when we talked to Kubernetes
shops, they don’t let developers near Kubernetes.”

Not sure what he means by "get near" but it's really not that complicated to
use kubectl to interrogate and modify the cluster. All of the back end
engineers on our small team are comfortable with it.

~~~
hajhatten
Yes. If you don't trust your developers to use controls like kubectl, you have
a bigger problem.

And I'm sure it's easier to throw money at a project like this than to
actually fix that problem.

~~~
sheeshkebab
I think they meant get near editing Kubernetes yaml files (or some abstracted
equivalent there of)

------
geggam
All of this is well and good. Now find folks who understand the entire stack
and can support it.

Tell me you are saving money after you hire them :)

~~~
dajohnson89
I really don't understand how this point isn't more widely acknowledged. k8
expertise is _expensive_. I'm seeing companies jump in head-first without
doing any cost-benefit analysis, and winding up in some really difficult
situations.

~~~
Bombthecat
First mover advantage. And when the recession hits you fire them and your
pipeline should be mostly automated by then... The rest is still paying full
monolithic big servers. And slow deployments.

~~~
geggam
That only works until your first outage :)

K8s isnt bulletproof and requires significant care. Not to mention constant
upgrades

~~~
markbnj
The bit about significant care may be true in on-prem installations, I don't
know, but GKE for example is pretty darn bulletproof. As for upgrades, they do
come along frequently and you don't want to get too far behind. Rolling
nodepools has made the process a lot less labor and time intensive for us.

~~~
geggam
From my perspective using GKE or any other k8s service isn't you running k8s.
It's you paying devops salaries to release managers.

------
sandGorgon
Basically this is a helm chart. Which is one of the problems of k8s - a
typical application consists of multiple services that need to be deployed
together.

In Docker Swarm, that's a Stack. K8s has no equivalent. So there's no answer
to "how do i deploy/update my flask api and celery workers together"

~~~
blairhudson
Actually, Docker can now deploy stacks as a k8s “stack” resource:
[https://www.docker.com/blog/simplifying-kubernetes-with-
dock...](https://www.docker.com/blog/simplifying-kubernetes-with-docker-
compose-and-friends/)

This makes it especially easy for anyone transitioning from Docker Swarm to
Kubernetes.

~~~
sandGorgon
True. However the fact remains that k8s doesn't have a primitive here - so the
toolsets (helm, docker stacks) have to manage it on a beat effort basis.

In Swarm, the stack is an atomic unit.

That's what Microsoft is trying to fix - the atomicity of an abstraction level
that is higher than K8s Services.

Not sure why they didnt do this as part of the k8s core committee process.

~~~
cpuguy83
Swarm does not have a concept of a stack. This is purely client-side.

------
andy_ppp
I’ve really thought that something like this was needed for a long time. Why
isn’t there standardised containers for user management and permissions,
authentication, scheduling things, processing events, caching say... there are
off the shelf containers for simple things like proxying but nothing that
understands your application. And it looks like they have taken it further to
include Functions and Actors which might be useful and allow easier scaling of
certain things. Let’s hope this goes some way to addressing that.

~~~
mwjfussell
MarkF from the Azure team. We thought so too. Having seen developers reinvent
the same capabilities time and time again, and seen the frustration when my
favorite framework X did not have a certain capability, we wanted to provide a
distributed system building block approach. And one that can be just dropped
in with local calls without having to recompile in many different libraries.
It is an approach that we have found to provide easier extensibility and also
support.

------
rumanator
Links to related discussions:

* Dapr: an open-source project to make it easier to build microservices: [https://news.ycombinator.com/item?id=21272098](https://news.ycombinator.com/item?id=21272098)

* Announcing the Open Application Model (OAM):

[https://news.ycombinator.com/item?id=21272082](https://news.ycombinator.com/item?id=21272082)

------
bjt
This is exciting!

When Docker came out, I was just finishing an in-house-Heroku style project at
my employer, based on LXC containers. I watched as Deis came along, promising
to give the same benefits in an open platform. I was sad to see Deis go away
after the Microsoft acquisition. The industry got all excited about
Kubernetes, but it seemed to me like we were backsliding from progress that
had been made toward a 12 Factor platform.
([https://www.12factor.net/](https://www.12factor.net/)) . It's very
encouraging to see that coming back.

~~~
gabrtv
Gabe from the Azure team here. I’m also the guy who founded Deis ;)

Glad you like what you see. The original Deis team worked on much of it. I’m
happy to say we have a lot more innovation coming in the PaaS space.

------
chuhnk
We've been working on something similar for a few years now. Micro is a
runtime for microservices
[https://github.com/micro/micro](https://github.com/micro/micro). We primarily
focused on Go and are moving towards multilanguage via a http api, proxy and
SDKs much like dapr.

------
ArtWomb
Worry about vendor lock-in is real.

RedHat OpenShift (yes yes IBM) is gaining momentum because of it. But the
field is still wide open. And the burden will fall on small ISVs to provide
open solutions around data portability, redundancy, security, and a host of
other concerns.

In the early days of Cloud technologies, the original vision was highly
commoditized. You'd pull a Docker container from a registry hub and perhaps
not even know where it ran. Perhaps even an exchange would handle pricing and
performance. Instead, we have the Big 4 vendors in a highly fragmented space
offering roughly the same services.

------
jmcomets
I haven't used k8s that much, so I can't see what features OAM (or the
implementation, rudr) adds to k8s. Could someone provide me with some
insights?

~~~
rumanator
On a previous HK discussion on OAM[¹], it was mentioned that the idea is to
provide a higher level abstraction on kubernetes that enable developers to
configure and deploy distributed applications in a platform-agnostic way.

One of the author's of the OAM spec mentioned that an OAM abstraction might
enable developers to deploy to k8s, docker swarm or Apache mesos with the same
OAM config, provided that all platforms (or vendors) implement/support the OAM
API.

[1]
[https://news.ycombinator.com/item?id=21272082](https://news.ycombinator.com/item?id=21272082)

------
naveen_
I think Microsoft is becoming an open-source company!

~~~
aairey
That's what they want you to think!

~~~
steego
They have to become an open source company because they can't risk putting all
their eggs in the software basket.

They have to work on projects like this to ensure developers can easily deploy
to Azure so they can hedge their revenue.

------
sheeshkebab
Trying to manage an “application” in microservices world is misunderstanding
the whole concept of microservices, IMO.

~~~
vturecek
Interestingly, what we found through our research was that nobody could agree
on a definition of "application" in a microservices world. The way OAM is
designed currently doesn't enforce a rigid "application" structure for
services. We have a concept of application scopes that can be used to place
application-like boundaries around groups of services (modeled as "components"
in OAM). For example, grouping services in a "health" scope where the health
of each service in the group is evaluated when any one of the services is
upgraded as a trigger for automated rollback is something application scopes
are designed for.

disclosure: am one of the OAM authors.

------
malkia
First I thought dapper, but that was google, and it was about distributed
tracing... but sounds very close (and have some relation - e.g. you need
sidecar to capture network traffic, and propagate tokens for distributed
tracing - that is unless you want to change your source code to do so)...

------
gabrtv
Gabe here from the Azure team. Happy to answer questions about Dapr and OAM.

------
yodon
How does Dapr (or the Actors in Dapr) relate to Microsoft Orleans?

~~~
mwjfussell
MarkF from Azure team. The actors in Dapr are based on the same virtual actor
concept that Orleans has, meaning that they are activated when called and
eventually garbage collected. If you are familiar with Orleans, Dapr actors
will be familiar. The difference with Dapr is that because it is programming
language agnostic with an http/gRPC API the actors can be called from any
language (although there are also friendly language SDKs on top).

Creating a new actor follows a local call like
[http://localhost:3500/v1.0/actors/<actorType>/<actorId>/meth...](http://localhost:3500/v1.0/actors/<actorType>/<actorId>/method/<method>)

for example
[http://localhost:3500/v1.0/actors/myactor/50/method/getData](http://localhost:3500/v1.0/actors/myactor/50/method/getData)
to call the getData method on myactor with id=50

------
hardwaresofton
Dapr looks like a slightly beefier sidecar-based service mesh.

~~~
mwjfussell
MarkF from the Azure team. Dapr is not a service mesh, however it will work
with service meshes such as itsio, linkerd etc. Dapr does provide direct
service-to-service invocation which you can use in place of a service mesh if
you want, however Dapr does not handle network policies or traffic management
that service meshes do. Dapr is a side-car that is language agnostic, and
using http or gRPC provides distributed system building blocks via open APIs
for asynchronous pub-sub, stateful services, service discovery and invocation,
actors and distributed tracing. All of this is extensible, so you can add new
building block capabilities and only choose the use the ones you care about.

------
devkulkarni
Referring to Microsoft's original announcement post of OAM,
([https://cloudblogs.microsoft.com/opensource/2019/10/16/annou...](https://cloudblogs.microsoft.com/opensource/2019/10/16/announcing-
open-application-model/)), the problem of defining application workflows and
their dependencies that OAM is addressing is genuine. We have also faced this
problem as part of providing Kubernetes-native platform stacks to our
customers.

It will be interesting to see how OAM evolves, especially since it is coming
from the same team that is leading the charge on Helm and CNAB
([https://cloudblogs.microsoft.com/opensource/2018/12/04/annou...](https://cloudblogs.microsoft.com/opensource/2018/12/04/announcing-
cnab-cloud-agnostic-format-packaging-running-distributed-applications/)).

Here is what we have learnt about this space in the last one and a half years.

Application portability - While the point about application portability is
true, if an organization has decided to adopt Kubernetes, the portability
requirement is solved by Kubernetes itself to a large extent. If your
application platform workflows are built as Kubernetes YAMLs, then they can be
run on any cluster. Kubernetes YAMLs of built-in resources (Pod, Secret, etc.)
and Custom Resources (MySQL, Postgres, etc.) can be leveraged to create such
Kubernetes-native platform workflows. Our learning has been that a solution
that focuses on solving the platform workflows problem on Kubernetes needs to
augment existing Kubernetes tools such as Helm, Kustomize, etc. We have been
developing such a tool ([https://github.com/cloud-
ark/kubeplus](https://github.com/cloud-ark/kubeplus)). Check out this blog
post which provides detailed comparison between existing tools.
[https://medium.com/@cloudark/discovery-and-binding-of-
kubern...](https://medium.com/@cloudark/discovery-and-binding-of-kubernetes-
custom-resources-in-multi-operator-environments-b5acf88fa639)

On separation of concern between Devs & Ops - Again with adoption of
Kubernetes, our observation has been that Kubernetes YAML and the tooling
around it such as Helm is becoming a common language of communication between
Devs & Ops. In our view the goal for anyone developing new tools/frameworks in
this space should be to help break the barriers that have existed between Dev
and Ops teams further. One way we are trying to do this is by extending the
vocabulary of ‘as-Code’ systems from the Infrastructure world to the
‘platform’ world of application development teams. Check out some of our work
in this space primarily focusing on Kubernetes Custom Resources here -
[https://cloudark.io/platform-as-code](https://cloudark.io/platform-as-code)

------
Havoc
Dapr sounds like it could be cool. Almost like a cloud function but with more
intelligence.

~~~
mwjfussell
MarkF from the Azure team here. The idea is to provide a function like
experience with any programming language and then have common capabilities
like saving state, sending events that your app can use with local host calls.
And make this all extensible with components

~~~
Havoc
>MarkF from the Azure team here.

Glad to see you engage. Thanks to enterprise sub & monthly credit azure is def
my favorite cloud right now :)

