What were your biggest hurdles when first adopting microservices? Was it hard for your team to determine service boundaries? Did you struggle with developing and managing them?
You have to start out with a monolith and only if you realise along the way that some components might work better as a service (micro or not) you should extract those. Until then, commonplace modularisation will serve you just fine.
Once you have more than 1 microservice running infrastructure becomes a huge problem. There's no real turnkey solution for deploying and running internal / on-premises microservices yet. You basically have to build monitoring, orchestration and logging infrastructure yourself.
The number of different services is close to the number of people on the team.
This is working very well for us, and it provides us with some welcome isolation when there are problems with one of the microservices. Maybe we can go into read-only mode or stop processing batch jobs for a while, depending on what services have problems.
But we also have good infrastructure support, which makes this a lot easier.
Just for every team that I see that has a good use case for micro services, and does the hard work of instrumentation and deployment, I see 8 teams that go with microservices because they think it's a magic bullet. Then they don't spend the time and effort necessary to get instrumentation and orchestration up and running. They don't aggregate logs, they don't spend the time to create defined contracts between the services, they don't make services robust to the failure of other services. They just complicate their debugging, deployment, uptime, and performance scenario without getting any of the benefits.
It's also a lot easier to prioritize process scheduling than it is to prioritize thread scheduling.
I had a system where the main running cost was MySQL. It turned out that I needed to provision a lot of MySQL capacity because there was one table that had a high rate of selects and updates.
The hot table did not use many MySQL features and could easily be handled by a key-value store with highly tuned data structures. That's a place where a "microservice" which has it's own address space, if not machine, makes it possible to scale parts of the system that need to be scaled without scaling the rest.
One table is small in terms of data but involves an interactive service that might generate 50 updates/sec at peak times.
With the "hot" service implemented on top of a key-value store, the database is the ultimate commodity, I have many choices such as in-memory with logging, off-heap storage, distributed key-value stores, etc.
The service is not "in front" of MySQL, but is on the side of it so far as the app is concerned.
I'm lucky that the project that I'm working on has support to use JHipster (https://jhipster.github.io/) with microservices deployed to OpenShift (https://www.openshift.com/). I used MiniShift to test my deployments, metrics, monitoring, orchestration, and logging locally. This was mostly for a proof-of-concept.
There was so much fragmentation between each autonomous team that we were forced to standardize on a common framework (think of build/integration pipelines and common modules) which could almost never be changed or refactored.
The upside was that there was way more automated testing and flexibility in deployment, so overall it was a net gain.
Still, the actual ergonomics of working with the build/release system was so painful it made me leave to work somewhere where I could do actual innovation.
There was so much waste that was already 'baked into the system' and the other teams finished their work so if you had struggles with it it was your problem
That definitely does not sound like an overall net gain.
They were doing even worse before. They upgraded to a CD style pipeline to AWS instead of monolithic java apps. It was better but nobody realized there were deep and very wrong tech debt. The complacency around this was a big factor in causing me to seek new challenges.
The biggest pain points were always not the OSS stuff that was adopted, it was all the “NIH” innovations that were developed because the OSS stuff wasn’t quite right, stuff that “worked” but had no support or allowances for improvement.
When we started switching over to microservices, originally we were going to have some sort of standardized message format, which would include all of the info that service would need to do its job.
However, there were too many people building it too quickly, and it's since devolved into a spaghetti pile where almost every microservice has about 3-4 others on which it is dependent (and a handful on which those are dependent) - to query for some bit of data, set off an async task, etc. This obviously complicates the development environment significantly.
When I want to add a feature to a service it can take hours trying to bring up the correct dependencies, update their configuration, fix the errors that they're returning due to their dependencies being down, and set up the necessary databases and ssh tunnels.
Testing of individual microservices in isolation is insufficient. Interaction between services has to be tested as well (because real APIs never work exactly as documented/mocked). Proper integration tests aren't any easier than testing a monolith.
It's tough to fit this in a small comment, but I would say avoid microservices until you know they are the right solution. They are often the right solution when you have multiple teams with independent development schedules. They are often the right solution if you have daemons that have fundamentally different characteristics (web server vs message consumer).
They are often not the right solution when you decide you need another module in your app.
Distributed systems are moving parts. Moving parts break. You shouldn't have more than necessary. Deciding what "necessary" is is a pretty tough task.
That statement assumes this picture:
ServiceA -> Network (https) -> ServiceB
This involved creating a mutually agreed upon definition of what we actually meant by "microservices". I literally repurposed paragraphs out of Sam Newman's book to do this.
As a small team, I also didn't have examples to support my argument that microservices would benefit us.
My sense is that for a small team a microservices architecture might be more trouble than it's worth, but I can definitely see how some kinds of applications - or even kinds of teams - it could be a good fit.
Without an abstraction layer and proper planning, different teams will start building microservices with lots of common functionality like security, authentication, logging, transformations that over time will increase the complexity and the fragmentation of the entire system. Ideally you would want services to simply receive a network request, serve a response, and delegate all the complimentary middleware execution somewhere else, like an API gateway.
Traditionally API gateways in a pre-container world were centralized in front of an API, but modern API gateways can also run in a decentralized way alongside your server process, to reduce latency.
Then once you have everything in place, you end up with a bunch of microservices that your developers (maybe belonging to different departments) will have to use. You will need to fix discoverability, documentation and on-boarding. Some of these services may be also opened up to partners, so there's that too. Traditionally developer portals were only being used for external developers, but they are becoming a requirement for internal teams as well.
Finally, you need to carefully monitor and scale your services. Without a proper analytics and monitoring solution in place it will be impossible to detect anomalies, scale horizontally each service independently, and having an idea of the state of your infrastructure.
Generally speaking running microservices can be split in two parts:
1. Building the actual microservice.
2. Centralizing common functionality, documentation/on-boarding and analytics/monitoring.
Most of the developers or project managers I regularly speak to tend to forget the second part.
Disclaimer: I am a core committer at Kong
Btw, we thank you and your team as we are Kong users as well. :)
1- You first mistake is wanna to (be a bomb defuser) use micro-services.
2- You second mistake is (getting close to the bomb) start building a micro-service infrastructure.
3- You third and worst mistake is believe (you can defuse the bomb safely) you can build a microservice solution correctly.
The only reason to (be a bomb-defuser) use microservices is (you are part of the police or army and somebody must do this) you ALREADY HAVE SOLVED THE MAIN THINGS FOR YOUR APP and now are in the hostile territory where micro-service truly make sense. You are facebook-alike.
Latency is another big one - the communication latency between any two points is miniscule, but throw together even 4 services, and suddenly the time required to encode, route, transmit, receive, and decode adds up quickly. If you want to keep round trips to under 100ms in your own code, each microservice you add could easily subtract 10ms from that budget.
Or is it better to deal with that complexity up front and then adding new services/functionality will be easy as time goes on?
The easiness to develop a single service makes many engineers loose focus on the big picture. Without an proper architecture and governance in place, larger microservice projects tend to fail pretty fast and in horrible ways.
* auth / authorization: easy to grasp, tough to implement,
opens a world of architectural wtfs
* service configuration files / variables
* understanding that more microservices can run on
the same host
* system entry point, credential based, jumpbox
Happy to chat more, I am definitely not an expert on the domain but for what is worth I was technical lead (whatever that means to you) on an AWS Case Study project.
What was the case-study? Can you link to it?
What has been your best resource to support/inform any of your arch design considerations?
For architecture debates ( yes there were some :) ) the resources used would vary depending on the level of the issue at hand.
For inter-service communication we had a standardized API interface that would need to be exposed and most of the discussions would circle around how does the service solve a problem and what API does it expose to the outside world.
For general system workflow issues we would resort to diagrams or good old fashioned whiteboarding whose artifacts would also get converted into diagrams.
A general rule-of-thumb that I pushed was that every single service would be documented starting with the design process and ending with support info.
Hope that makes any sense :)
* k8s is apparently ridiculously hard to set up if you want to use something that isn't AWS / GCE / GAE / non-"big-cloud"-provider-here.
* Many of these have questionable (IMO) default authentication choices. I can understand if it's running on an internal network or something, but if you can't set up an internal / private network to hide the management interface (ex Marathon, k8s dashboard, ...) inside, you're SOL it seems. Ex. Openshift Origin having a login UI, but the default allowing logging in with basically anything you want.
* Rancher seems nice, but between the internal DNS service randomly failing all DNS queries until a full restart and rancher-server's internal MySQL server writing to the disk at 500Mbps, it's... iffy. At least it's a nice UI/UX otherwise and supports lots of authentication options out of the box.
* Straight-up not working. For example, I tried running Openshift Origin on stock Debian and Ubuntu installations, and it couldn't even finish starting up before it crashed and burned. I did file an issue against openshift/origin about this, but so far it's been unresolved. Another example with Openshift, after figuring out how to work around it failing to even start up, it can't even create pods/containers. This is apparently a known bug that's been around for a while (~4 months) with seemingly no resolution.
* Weird docker version requirements. I can get that things like k8s would want to pin to LTS versions of docker. That's fine. But if I can't even import the signing key to install the LTS docker version because it doesn't even exist in the keyservers anymore, that sounds like an issue to me.
Maybe this would be different if I was using one of the Big Cloud (TM) providers. Who knows. But with my budget of "poor college student," OSS offerings are all I can do when my entire budget is consumed by server bills - and I can't cheap out here; a few TB of bandwidth and maxing up to 16 CPU cores isn't cheap :( - and it just seems like the "state of the art" is pretty terrible, at least from the UX perspective. I spend enough time staring at text and delving into the CLI and whatnot when I'm writing software. Why is it seemingly so hard to get even a simple UI to cover the basic functionality - create a master node, add slave nodes, deploy containers, and scale them up? Rancher seems to cover this use-case the best, but I've run into enough issues with it that I'm starting to seriously consider figuring out how to write my own orchestrator. I'd rather not, since it'd be a lot of work, but setting up these "standard" tools is a ridiculous Herculean task if you don't have a massive budget, it seems.
Ninja-edit: For a bit of perspective, I'm talking about a spare-time project that ended up gaining far more users than I ever expected. I'm the only person working on it, so trying to figure out all the development, ops, etc. on my own has been a huge struggle since so many of these tools just have a god-awful UX.
Handling service discovery, API gateway in ever-changing kubernetes ecosystem.