Clangd already exists. It already supports LSP. It is already part of the llvm project. It was written by !Apple, and used by a wide number of people.
(I want to make sure these people still get credit, i already see people in this thread thanking apple for supporting LSP, which is not actually what happened).
What apple has proposed contributing is: Making clangd work with xcode, and extensions to LSP.
This is great (as long as the second ends up standardized over time).
But the benefits people listed here so far already exist in clangd. The main benefit to work is "xcode support" and "better refactoring" which is awesome :)
The proposal is to split the LSP implementation into a transport layer and a logical layer. The current LSP implementation is directly tied to JSON-RPC.
Apple's end goal is, as you say, to get clangd to work with Xcode. However, having a replaceable transport layer for LSP could benefit lots of people other than Apple.
Sure, it's nice, but again, being pragmatic, nobody else has desired such a thing that much.
Ie so far the "lots of people" have not materialized, even outside clangd.
I can certainly think of use cases for such a thing, but most people seem happy enough with json-rpc at the moment.
Part of that is likely that trying to speak LSP over a non local connection may be a fools errand anyway. It's a chatty protocol no matter what the transport.
Just because Apple is the first doesn't mean there won't be others. Maybe others have not materialized because the upfront cost of updating clangd was judged too high.
JSON-RPC has several alternatives that very large software companies have vested interests in: gRPC and XPC.
You seem more knowledgeable of these technologies than I am, though. It just seems to me a good thing that 1) Apple, and consequently, all iOS and macOS developers will be using clangd, and 2) Apple is putting significant effort into improving clangd. I'm not sure why any of that news or optimism needs to be quenched.
My point was precisely that plenty of languages have LSP implementations at this point, an f even in those, nobody has cared much to try to find an alternative to json rpc.
As for 2, all I'll say is based on history in llvm projects, I'm not anywhere near as optimistic as you. The history here is simply not one of all great things. The PR is, but the reality is much more mixed (some really great things, some really not so great). So while I certainly am hopeful, it's definitely not the unbridled optimism everyone displays here.
While I agree that not everything is always great, I'd say the same for our own contributions ;) I've definitely let patches unreviewed for way too long due to internal priorities, and have gotten my share of shit at conferences for that...
In this case, the collaboration with Apple and Ericsson on the underlying architecture has been going on for a while, and while not everybody always agrees on priorities and design details, I'm thrilled to see Apple double down on their involvement here!
That's not what i said at all. I said some things have been really great, some really not so great. This isn't even a controversial view among long-time llvmers.
Heck, last time i asked him (a few months ago), even Chris Lattner agreed with my view, and he used to run his stuff at Apple, so i don't think it's unreasonable.
I think it's perfectly okay to be cautiously optimistic instead of fawningly happy.
Do you actually have data or a real argument you'd like to use to suggest otherwise (IE that it's been all overwhelmingly positive?)
Yeah, but that is likely to actually go away, not happen more.
It's not a desired thing in that case, it was just easiest at the time for various reasons. We didn't want the public project to have to support our special snowflakism, hence it was done a different way.
clangd already has a layered architecture, where you can either use the JSON-RPC mechanism or more directly control clangd and implement your own transport. There are no plans for this to go away, as generally large companies often have environments in which they want better control of features than a standard protocol like LSP can give you, and we don't discourage that.
We also just talked with Apple folks at Euro LLVM about this, and while the plan for the near future is for them to have a transport conversion to JSON-RPC, I expect them to want the more fine grained control in the long term.
I'm glad to see that Apple is putting work into making their tooling better and easier to work with, but I'm still not clear on the benefits of this change. Would someone care to enlighten me? Currently, the benefits of libclang that I see are that I can easily drop it into a project without requiring a lot of dependencies, and I don't need to figure out how to get permission to run different processes and manage interprocess communication between them–if anyone doubts that this will be an issue, on iOS neither of these is really supported at all. In addition, I don't require a daemon to constantly be running to analyze my code. (Why is this bad? Take a look at Swift's sourcekitd.)
Well, I mean, it's a little more than that unless clangd is straight up finished software. Apple will now be contributing to clangd as well, which benefits all clients of clangd.
To be able use it from any editor that is able to understand LSP, instead of having everyone writing their own bindings to libclang, specially if the editors are written in type safe languages.
Yes, exactly this. If you have 3 different editors and 3 different languages, you potentially end up with 9 different implementation. LSP means that the same can be achieved by one implementation per language and one plugin per editor.
With LSP, you end up with one subpar implementation, and you need to duplicate all refactoring functionality for each language.
While with common IDE plugins, you have one implementation of the refactoring functionality, and each language plugin just exposes an AST.
So LSP actually leads to more duplicated code and worse editors.
VS Code will never be able to match the functionality of IDEA unless each language server reimplements all the functionality slightly differently. What a mess.
That being said, as an implementor I'm finding cases where LSP's feature set is underdeveloped. For example, there is nothing in the spec to account for renaming files, so I can't put in my rename action without the behavior being incorrect (because if you rename a class in Java you have to rename the corresponding file, which IDEA handles). Also some things require custom LSP requests, like getting the list of build configurations to run. I think this is a product of the Java support in IDEA being so much more comprehensive than anything in LSP so far. But hopefully as the spec is improved these issues will be worked out
Is your complaint that they're not iterating fast enough? Because this is something I'd prefer they take their time and get it right, especially once we get past the low-hanging fruit and into complex things like debugging.
My complaint is that the entire concept on which LSPs are based today — each LSP reimplements all functionality itself, the editor is only a dumb textview — is wrong.
Language servers should expose an AST, not some actions. Language Servers shouldn't even know where the cursor is — or how many there are.
With a proper LSP, you can also write a linter just by using the protocol, you can call the same action on a thousand places at the same time, and the refactoring functionality is implemented only once, and reused everywhere.
You mean, with common IDE plugins, you end up with one implementation of refactoring functionality per IDE. That's what's being avoided here (because, for one thing, there are far more IDEs than you think, and they don't deserve to be second-class citizens.)
This would be an interesting idea, provided that editors go all-in on supporting such refactoring logic on the client side. The IDEA code defines "extension points" so language specific support for common actions can be implemented in a modular fashion. When the user decides to execute a refactoring, it is just the matter of iterating through the loaded extension points and choosing the one for the language being edited. Having worked with the IDEA code myself to build an LSP server, there end up being language-specific parts such that the support between e.g. Java and Kotlin have to be handled separately. However that was probably due to the way I wrote the server; these could be generalized to a single "refactor" entrypoint with some work. It's just that using the current LSP model with the server being the part with AST awareness, there isn't much gain in doing so.
I'm no expert, but one major distinction I can see is that libclang would be used in (editor) process, and a clangd server would operate outside the editor process and communicate over a protocol with the editor.
You're always free to spawn your own process, which links with libclang, if you really want to and communicate with it over XPC. But then again if this crashed Xcode all the time maybe Apple would have more incentive to fix it ;)
The main problem with libclang is actually that it puts another abstraction on top of the AST that nobody really had time to maintain.
That's not made better by the amount of work changing libclang requires just due to its backwards compatibility promises.
There's no reason to discount iOS as a development machine–it's perfectly fine for coding, as Swift Playgrounds shows. (Actually, there are some issues with executing code in third party apps, but I'm working on a way to bypass the restrictions around this. Check back again soon!)
There is an article from 2010 titled "Emacs is dead" that is being reposted on HN from time to time. The author args that the greatness of Emacs is to rely on external, editor agnostic tools where text act as an universal interface/medium but that the practice is fading out, and thus the dead of Emacs. Now, ten years later, Apple announces this. Emacs, the undead?
I think you might be right, but the greater context is highly ironic.
The company resurrecting the Emacs concepts is Microsoft. They're the ones that created OmniSharp, bringing the idea of language servers to the programming mainstream. They then created the Language Server Protocol for what now is Emacs's biggest long term rival in the flexible editor arena: Visual Studio Code.
Visual Studio Code is for the moment on a somewhat bloated and shaky foundation because of Electron. But otherwise its design is quite solid. And even that foundation will probably become a lot stronger in the next few years as WebAssembly gains wide adoption. Visual Studio Code itself is written in Typescript and it's not hard to imagine Microsoft adding a WASM backend to it.
I think Emacs's greatest strength comes from the ability to use a single scripting language throughout the whole editor. I can quickly write up some ELISP statement to do anything, if I like it I can put it in a file and load it. If I want to share it, I put it in an extension.
I've just looked up the official tutorial on making an extension in VS Code. It seemed cumbersome. You need a code generator to lay the foundations, you need to consult the API docs etc.
With Emacs you can learn as you go and the editor itself guides you if you ever get stuck. You can read the source of any part of Emacs you like, change it on the fly, evaluate and try it (except for some core modules written in C). And I think that's the beauty of Emacs. Emacs is easy to tinker with.
What do you mean by "arbitrary stream of text"? I fail to understand, to be honest. A text always need to be parsed, even a simple number, but the point is to deal with characters instead of raw binary data.
>What do you mean by "arbitrary stream of text"? I fail to understand, to be honest
That it's structured and specific.
UNIX pipes, for example, can deal with any kind of text -- it's up to the user to ensure that the commands (e.g. "cut -d," etc) work ok with them, whereas the JSON RPC communications of the language server are not only in a specific syntax (JSON), but they also have predetermined content schema.
There's a difference if I tell you: I will send you any kind of text at any time, it's your responsibility to deal with it and parse it accordingly to whatever it might be, than if I tell you, I'll send you JSON with this schema.
They mentioned incremental compilation in the first few seconds of the video. Didn’t watch the rest of the video yet.
I’ve always wished for incremental compilation. Imagine incremental compilation that was so fine grained it would recompile only the functions you changed. Of course this makes full-program optimization impossible but I think most of us don’t compile at the highest optimization levels during development anyway because doing so slows down builds without any benefit for development builds. For production builds we use the highest optimization levels of course.
Additionally I would like to have hot patching. I know some people have this, personally I’ve never had that.
If anyone knows about any systems for hot patching Rust code please let me know :)
The future is a lot easier to build when you're vertically integrated. The risk with vertical integration is that keeping all parts of the integrated stack at the cutting edge is too costly and you start to fall behind.
A marketplace approach with broad consensus around interfaces is ideal, but it's hard to bootstrap. LSP via VSCode seems to have done the job.
Woah I had never seen this before. That's really cool. So in theory you can make your own editor and provide a ton of IDE-like features by communicating with this server?
Are there any other implementations of LSP by other orgs?
On one hand it's really nice to see Apple jump on the LSP train and use standard tooling for their IDE to interact with language tools.
On the other hand, instead of having a consistent strategy of using proper LSP (which by specification is JSONRPC), they are shoehorning LSP on top of their in-house XPC transport. To do that they're planning to introduce additional complexity to clangd (a transport abstraction), while actually defeating one of the main purposes of LSP, which is reducing the m-times-n complexity problem of matching compilers with tools to an m-plus-n complexity problem. No other tools (unless they support LSP-over-XPC as well) will be able to talk to XCode. XCode won't be able to talk to other tools. I hope they rethink that decision, keep clangd simple and instead adopt proper (JSONRPC) LSP in all of their other tools for Swift, etc instead. That way they'd not only open up XCode to clangd, but also all other editors to their refactoring tools for Swift, etc.
Moving language parsing/refactoring/etc into a separate daemon is a step towards unifying tooling around specific protocols for interacting with compilers. Microsoft's effort towards a single language server protocol is worth mentioning: https://github.com/Microsoft/language-server-protocol
I'd love to see a future when new languages/compilers can implement this protocol for their external compiling daemons and just get support in all the various text editors/IDEs automatically.
In process vs out of process; and language-specific binding vs JSON-RPC.
Moving stuff out of process means, increased reliability and potentially ability to share data across multiple clients.
Moving everything to LSP with two supported transports (JSON-RPC and something more native) means that editors like Emacs, vim, VSCode can benefit from tooling without needing to link against a library and work with C bindings.
A quick cut-and-paste from the first URL explains why LSP was created:
"Adding features like auto complete, go to definition, or documentation on hover for a programming language takes significant effort. Traditionally this work had to be repeated for each development tool, as each tool provides different APIs for implementing the same feature.
A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication.
The idea behind the Language Server Protocol (LSP) is to standardize the protocol for how such servers and development tools communicate. This way, a single Language Server can be re-used in multiple development tools, which in turn can support multiple languages with minimal effort."
Does anyone have any details about Apples cross language indexer they mention? A good indexer is the key to and efficient language server and I think clangd’s is a little weak right now.
Clangd already exists. It already supports LSP. It is already part of the llvm project. It was written by !Apple, and used by a wide number of people.
(I want to make sure these people still get credit, i already see people in this thread thanking apple for supporting LSP, which is not actually what happened).
What apple has proposed contributing is: Making clangd work with xcode, and extensions to LSP.
This is great (as long as the second ends up standardized over time).
But the benefits people listed here so far already exist in clangd. The main benefit to work is "xcode support" and "better refactoring" which is awesome :)