Basic summary: "a modern IDE should be optimised for my particular use-case". The author then proceeds to describe a very atypical use-case.
One example: "The IDE should ask for the Github org eg, github.com/astradot and it should create a single project that contains all the repo as modules. It should then manage lazy-loading/lazy-checkout or whatever is needed to give me a seamless experience browsing the code of my entire org."
Like (I strongly suspect) a majority of developers, I work in a large monorepo. There are 400+ other repositories in the same GitHub org. 99% of them are small projects I'll never touch. I wouldn't want my IDE to go anywhere near them.
That's the fundamental problem with tools like IDEs. They have to optimize for just a handful of use cases or the UX for all of them gets worse than just learning the individual tools. Jetbrains is pretty awesome but still not quite good enough to convince me it's worth adopting.
Can people with expertise in this situation speak about how typical it is? I guess if you have all those services then you need your IDE to help - but surely this isn’t super common? If it is, what sort of size of micro service are we talking here? I’ve worked in orgs with 3M line codebases, and I’d struggle to find a hundred ways to partition the code naturally.
I work in a large microservices environment. It’s very important that IDEs don’t make it easy to reach inside other services. The whole point is to create cognitive barriers at the public APIs. The pain of coordinating changes across services is a feature: it is telling you that your architecture is wrong.
And what qualifies as a micro service in your environment? Can you give a few examples? How many people work on each micro service? Etc etc. Would love to hear more details so I can build a better mental picture.
I'm not the OP, but I can give you some examples from when I worked at Airbnb a few years ago:
* Each team (let's call it eight engineers per team on average) typically managed a few services.
* Services were typically ~thousands of lines of code to ~tens of thousands. Rarely, there were very small services that were hundreds of lines of code. There was a giant service -- the original Rails app -- called "Monorail" that had millions of lines of code, but was actively being dismantled. My understanding is that there is now a code freeze on Monorail, and the only changes allowed are changes that remove code from it and move it to a smaller service (either an existing one or a new one).
* Services communicated with each other over HTTP or Kafka, typically using Thrift, although also sometimes using JSON. Airbnb had an in-house "service mesh" (developed before the term service mesh existed) that abstracted away individual machines and was capable of operating at the HTTP or TCP layer.
A few classic examples of mostly-single-responsibility services:
* Prometheus (which I wrote) abstracted away reverse geocoding and correctly caching and manipulating results from geocoder APIs.
* Medusa abstracted away ranking and filtering listings by date/amenities/etc.
* Micasa abstracted away basic CRUD operations on individual listings.
Airbnb intentionally used codenames for services rather than calling them by what they did, which I wasn't a huge fan of; I forget most of the codenames now (and hardly anyone knew all of the codenames offhand even at the time). Some other example without the cute names were the pricing recommendation service (used to recommend prices to new hosts), the calendar management service, the experience CRUD service, etc.
It's pretty common. There's the extreme end like Uber (>3000 microservices) and countless cargo culting startups (I see a few dozen microservices for a team of half a dozen or fewer engineers in the wild frequently, which is big enough to desire cross-repo refactoring) but the 800 pound gorillas like Microsoft, Amazon, Apple, Facebook, etc. are at that scale by necessity. I doubt however that anyone is doing any sort of refactoring that touches a significant fraction of those and if they do, it's an organized manual team effort.
Microservices didn't become popular because the code could naturally be refactored that way, but because they enforced an interface boundary that worked better for mid to large sized organizations with multiple teams of engineers and levels of management where the communication overhead is much higher.
It depends on how you define "typical". I'd bet it's a narrow minority of companies, but most of those companies are very large, so then what percentage of all engineers are working at companies like that? Maybe not a majority, but a significant number
In corp this is certainly true - 100s and 100s of services - but YOUR work is scoped to one domain and I wouldn't see you needing to check out the world.
Each team has developed it's own way to work with it's subset of services - I doubt every team across every org could figure out how to run everyone elses domains to even make checking out the world a useful exercise.
As a long term corporate exercise, maybe that would "be awesome", but not trivial to do and PAINFUL to support and stay on top of over time.
We had a hard enough time keeping test suites up to date - I can't imagine trying to keep our portfolio "runnable" for everyone else.
That exercise is monorepo migration. Every target has to work with nothing more than "buck build" or "bazel test" or whatever, or you can't check it in. And anything that runs in production must be checked in.
I'm not sure whether it was really worth it for us, but it did solve that problem.
While we're not in the hundreds (and I don't know our loc), what I've seen happen is that on top of having a large number of services you end up with "meta-repos" that contain infrastructure-as-code or you end up having a repo from which you build the company wide Jenkins image with its configuration and so on.
Microservices are helpful only if the shape your teams match them. Conway's Law. Machine performance is not the main reason for them, but rather division of labor.
I switched from IntelliJ to Emacs about a year ago, and honestly these things seem relatively trivial in that environment, because I can write small ELisp functions that automate things like this without much effort or ceremony. For example it's quite easy to automate checking out repos, applying AST refactorings using LSP Mode, and then committing and sending PRs using VC/Magit. Or configuring projects and "Run Buttons" for entire groups of packages.
Granted there is effort in learning ELisp in the first place, but now that I'm comfortable with it, I find myself able to adapt to these changes quite easily while many other developers have to wait on big feature releases from their IDEs, use hard-to-automated GUIs, or heavyweight IDE APIs to develop "plugins" to modify behavior.
I haven't used it but as far as I know, you can write refactoring scripts using Kotlin or other languages within Intellij, see Tools -> IDE Scripting Console.
Cross-repo refactoring (even worse: renaming) seems like an at best worthless and at worst quite costly thing to optimize your toolchain around. Having stable interfaces between systems is very important. The whole point of having separate systems is so you don't have to change one when you change the other. If that's not the case, you're better off with a monorepo.
I see what you're saying but I don't agree. Repo topology might not be a choice. Almost certainly new features will cut across multiple projects. Why make that hard?
Tools should help you do the right thing. They should not be a painful bludgeon against other devs. Enabling fast and easy end to end testing is only helpful.
In general a new feature should have its home in one service and make calls to the existing network APIs of other services. If you’re regularly having to change the innards of those services to support new use cases, you have serious architecture debt.
You're being overly zealous. Every service has callers. Sometimes you want to work on the caller concurrently with the callee. Sometimes even the same person can work on the feature end to end.
There is no reason that making that easy to do will cause tech debt. You already have other means to your services cleanly versioned and compatible, such as tests. Making things environments easy to run locally doesn't change that.
If I want to add a piece of data and a visualization of that data the UI and the data store should be in the same service, no exceptions?
You're saying that a desire for when something gets worked on should trump any other design decision? "Want at the same time" informs separation of concerns?
You make services in an vacuum without considering the stakeholder/consumer of your service? You've never planned for a service in your org you consume to gain functionality because doing so would mean you should implement that functionality yourself in a service you own?
Teams that call themselves "platforms" are very often tempted to operate in a mode where they add a lot of one-off endpoints tailored to the specific use cases of particular callers. This skips the whole economic value of "platform" and makes them consultancies instead. My company has this problem in a big way, there are CTO-level efforts towards reusable abstractions and generic extension points instead of all this deep cross-team cross-service coupling. It is frequently the way I can add the most value in a design review: help the team come up with abstractions based on their domain model that are sufficient for their callers, instead of building to the letter of current callers' requirements (and signing themselves up to keep doing that for each new caller).
"How often will we need to change this and what will that be like?" is usually my top design consideration after "will it work?" So yes on that one.
I agree that you should write tools to improve the workflows you actually have. But this post, as I read it, is not written from the point of view of narrow problems and solutions — or rather, it is, but it is trying to be something else. It is talking about The Evolution of the IDE. And from that point of view I think my criticism is appropriate.
I completely disagree with the article (but I realize that this is only my opinion and IDE choice is something that comes down to personal preference):
Here is what I think IDEs need to do different in the 2020s:
- configuration needs to be in human-readable configuration files, not obscure XML configs that can realistically only be edited through UIs.
- how to build the project needs to be completely decoupled from the IDE
- compilers and language services need to be separated from IDEs. LSP and automated refactoring tools using the same infrastructure as the code that's doing code completion makes sure that everything is consistent
- emphasis on input latency is needed for editors to feel snappy. Lots of full blown IDEs are not great for editing text.
This doesn't align with my expectations/needs. I would love to see metrics/survey data on what others need.
- We no longer want code running on developer machines. We're actively working to embrace remote execution for everything.
- Repository boundaries exist for a reason. Why are you modifying two repositories at the same time?
- We think collaboration is a good thing. Live coding allows for pairing and troubleshooting. It's great for knowledge sharing and for getting new developers up to speed.
It seems to be a set of opinions and self-quotes based on the frustration of a particular project management pattern, which includes micro-repos and micro-services.
Managing dependencies, at build and run times, internal and external, seems to be the prominent pain point. As far as I can remember, it has always been and it cannot be solved in a single simple way.
Things have gotten "better" (ie at least there are tools and practices) with the omnipresent git, cheap VMs, CI, docker, kubernetes, etc.
> Refactoring so far has really been a single repo feature. But what if the code you are refactoring is called by code in 100 other repos in your organization? What if those repos are not even checked out locally? The modern IDE needs to evolve beyond single repo operations.
I hesitate to either call to Hanlon razor or accept that tools can help, further than basic monorepos. I don't see any valid ways to manage this with external dependencies however.
A number of interesting points that are thought provoking, but also a bit of flame bait about cloud IDEs "dying off". As founder of Codenvy, happy to say that Eclipse Che is an active and growing open source community and after our happy acquisition by Red Hat they sell it as OpenShift CodeReady Workspaces.
Cloud IDEs provide value for specific portions of the market:
1. Training
2. Remote contractors in secure environments
3. Support
4. Classrooms
5. Vendors who want embedded dev envs in other products
6. Open source clones / snippet evaluations
Codenvy was strong in the embedded dev envs for other products. When you look at the various vendors that have emerged like Coder, Repl.IT, among others, they have a tendency to specialize in one of these areas.
That is quite different from the first generation cloud IDEs which tried to compete with classic desktop IDEs.
I think long before IDEs implement any of these things - which represent a rather niche dev setup IMO - we need some way for libraries and applications to expose higher level metadata to language servers. Cross-repo search and replace is nice and all but it does't solve the problem currently filled by adhoc scripts and build steps that pull schema information from Swagger/Graphql/JSON schemas to generate types. Especially in the case of microservices where API boundaries are most often URLs or codegened RPC cliets, symbol renaming just isn't as useful outside of a single repo or workspace. Having, say, a protobuf or GRPC or OpenAPI language server feed information to the language server doing autocomplete and refactoring would be much more useful.
Liveshare is pretty useful for pair programming. Not about invading privacy but making it easy to work together. Given how easy it is to collaborate on docs with google docs it's kind of sad that remote collaborative coding tools are still in their infancy.
One big opportunity for IDEs to explore is the code visualization. Especially in languages like Java the abstraction, decomposition, delegation are heavily used for (sometimes) good reasons but at the cost of simplicity, and it's more difficult to see what's really going on (compared to simplified spaghetti).
There have been some weak attempts in IDEs (e.g. auto generated UML), but it's not really useable. Closest seems SourceTrail but that has many issues ...
When I read the first paragraph of your comment, I thought to myself "SourceTrail seems to be designed for this problem". Reading the second paragraph, it seems you have tried SourceTrail and found it lacking. Can you please elaborate those short comings. I recently became aware of SourceTrail from an HN post few days ago, but have not used it.
Honestly, I'm seeing a lot of these kinds of articles lately. What they have in common is that they're relatively short and expound on some arbitrary point which is highly subjective.
I wonder, are some people writing (and sharing these things on HN), not to share some insight on some subject but to have something to point at during interviews?
> We never need to see each other live code. That would be annoying/intruding on the other engineer's privacy.
I do need to pair program remotely, and collaborate on design docs. Collaborative editing isn't like a 24/7 feature that you need to enable, and I don't know why the author thinks those are the reasons that they're valuable.
Well a modern editor is more akin to an exobrain than a simple text editor.
Anyway recent intellij is pretty fast, also it is more memory bound than cpu bound.
My first time using IntelliJ was on trying to open a dozen microservices and discovering that I needed to mess with modules. Got back to Eclipse, especially because it already have the Launch Group feature to start multiple projects sequentially.
Version control also needs to get better. Version control right now has very little insight into how the code is changing beyond literal textual changes. Whenever I rebase a messy conflict, it breaks the code so easily.
Context: I most recently worked at Facebook, which develops and maintains an in-house IDE, Nuclide, for working with its massive codebases; currently it uses VSCode as a base, although it previously used Atom. In a previous life I led the DevTools team at Airbnb (and before that did various other things at Airbnb).
This article hits most of the nails exactly on the head, at least from the perspective of large corporate environments; I doubt any of these apply to small startups (and if they do to yours, something has probably gone very wrong). Nuclide handles these issues in fairly interesting ways:
* FB built their own incremental, caching compilers for several languages that Nuclide is aware of, so that typechecking and rebuilds are effectively instantaneous even across massive codebases.
* Nuclide integrates with FB's massive cross-repo search service, BigGrep, so that you can find references to anything.
* Nuclide's version control UI is phenomenal, and is aware of multiple repositories existing, which ones you have checked out, and integrates with their in-house version of Phabricator (a code repository/review tool similar to Github). I literally never learned to use Mercurial while I was there. I just used Nuclide.
However, there's one major difference between Nuclide and the vision that this article lays out: remote vs local code checkouts. Nuclide ran locally, but it used remote code checkouts, and your code ran on the remote, not on your local machine. I think the reasons and benefits were compelling:
* Massive repositories take up massive amounts of disk space.
* Massive repositories take massive amounts of time to download to your laptop, and if you're working in a large corporate environment, they also take massive amounts of time to download recent changes since there are many engineers shipping commits.
* If your machine gets hosed, you can spin up a new one quickly; in FB's case, it took seconds to get one of the "OnDemand" servers. If your local machine gets hosed... A trip to IT is not going to be as easy.
* If you run into issues, tools engineers can SSH directly into your instance and see what the problem is, fix it, and also ship preemptive fixes to everyone else. That would feel quite privacy-invasive for someone's local machine.
* Many employees enjoy being able to take their laptop with them so that they can work remotely e.g. to extend a trip to another country without taking more PTO. Laptops aren't great at running large distributed systems.
* Even some individual services (ignoring the rest of the constellation of services) are impractical to run on constrained devices like laptops, and operate on either "big data" or require large amounts of RAM.
* Remote servers can more conveniently integrate securely with remote services than laptops or desktops.
* Having the entire codebase stored locally on a laptop or desktop is a relative security risk: if it's in a centrally managed server and a laptop gets stolen, revoke the key. If it's on disk, well, hope the thief didn't get the password? Or in FB's case, hope the thief isn't an APT/the NSA and has backdoored the laptop's full-disk encryption? And it's not just laptop theft: a disgruntled employee could intentionally leak the code -- or in Google and Anthony Levandowski's case, sell it to a competitor through a shell company. If everything is stored centrally and local checkouts are extremely uncommon, you have a lot more room to automate defense against that scenario.
Overall I'm a big fan of running your code on remote devservers rather than on local machines once you get to scale.
I would be a fan of it if I could get Jetbrains-level code intelligence, but so far I can only get VSCode level intelligence, and to me that more than erases the benefits of faster execution.
I disagree that VS Code is just a "downgrade" of the IDE level experience offered by IntelliJ.
Certainly, a lot of functionalities are missing in VS Code, but they are compensated with speed and depending on your needs this compromise is more than worth it.
If you are using web-technology (like TypeScript), IntelliJ does not even offer much more than VS Code.
Its not hard to set up a multi-project view in Intellij. There are a lot of choices for build systems that will track cross repo builds. You can use something like Docker Compose (or several other options) to run many services locally.
Huh. Not a single mention of the word AI.
Yes i believe AI refactoring/assistance will be awesome.
Also Rust. Please Jetbrains remake IDEA in Rust!
We need faster ide's and lower latency
One example: "The IDE should ask for the Github org eg, github.com/astradot and it should create a single project that contains all the repo as modules. It should then manage lazy-loading/lazy-checkout or whatever is needed to give me a seamless experience browsing the code of my entire org."
Like (I strongly suspect) a majority of developers, I work in a large monorepo. There are 400+ other repositories in the same GitHub org. 99% of them are small projects I'll never touch. I wouldn't want my IDE to go anywhere near them.