
Rust Analyzer in 2018 and 2019 - jamesmunns
https://ferrous-systems.com/blog/rust-analyzer-2019/
======
jamesmunns
Hey! This is a post by matklad [1], one of the members of the Ferrous Systems
[2] team. The Rust Analyzer project [3] is an experimental compiler front-end,
designed to power things like code completion, syntax error reporting, and
other things that are necessary when building IDE support. Let us know if you
have any questions!

[1]: [https://github.com/matklad](https://github.com/matklad)

[2]: [https://ferrous-systems.com](https://ferrous-systems.com)

[3]: [https://github.com/rust-analyzer/rust-analyzer](https://github.com/rust-
analyzer/rust-analyzer)

~~~
jmarinez
Sure. Great work! Would rust-analyzer provide enough of a foundation to build
something like this?: [https://gtoolkit.com](https://gtoolkit.com)

Mind you, in the IDE above reflection builtin the language allows for the
incredible amount of power. In Rust's case, the only way I can I see it
replicated is at a layer before compile time, hence my interest in your
analyzer.

~~~
matklad
Unlikely: rust-analyzer focuses on static analysis. So it aims to power
something akin to IntelliJ IDEA, but probably would be a bad fit for something
like lightable/elisp/smalltalk ide.

------
ceronman
To better understand why this project is trying to re-implement the compiler
frontend, I really recommend this video from Anders Hejlsberg (creator of C#
and TypeScript) explaining how modern compiler construction oriented towards
IDE works:

[https://www.youtube.com/watch?v=wSdV1M7n4gQ](https://www.youtube.com/watch?v=wSdV1M7n4gQ)

------
georgewfraser
There is another approach to bootstrap an incremental compiler frontend on top
of a batch compiler, which I used to create a Java language server [1]. Batch
compilers usually support incremental compilation at the file level, so you
are already close to a solution: re-compile the active file every time the
user edits. This works but is too slow. You can dramatically improve the speed
by simply erasing all the code that isn’t relevant to the current edits, which
usually means erasing all the method bodies. This simple trick allows you to
build a fast incremental compiler on an existing batch compiler.

[1] [https://github.com/georgewfraser/java-language-
server](https://github.com/georgewfraser/java-language-server)

~~~
jamesmunns
This sounds like a use for the everybody-loops compiler flag:
[https://github.com/rust-lang/rust/pull/19964](https://github.com/rust-
lang/rust/pull/19964)

------
sqs
We (Sourcegraph) have also found the “dump all the data from the compiler”
approach limiting because it doesn’t work for the live editing use case (as
the article says), and then that means you have 2 analyzers: one used with
editors and one used for offline tools like Sourcegraph, DXR, etc. That
bifurcation means each one gets less community involvement. The ideal IMO is
to always target the live editing use case and add caching and optimizations
that use language-specific knowledge, so that in both cases you’re querying a
live server but it can compute the answers quickly. I haven’t worked with Rust
specifically, but it’s interesting to see that they came to a similar
conclusion.

BTW Sourcegraph is interested in funding this effort and building better Rust
support for code browsing and search into our product using
[https://docs.sourcegraph.com/extensions](https://docs.sourcegraph.com/extensions).
I just reached out to the author.

~~~
hinkley
There's a whole class of problems around this that I'd like to see get some
more attention. It shows up for all kinds of static analysis in code editors,
including things as simple as autocomplete.

For instance, if I have a file that compiles and I start typing in the space
between two function bodies, I'm probably implementing a new function. In fact
that probability is so high that you should just assume that to be the case,
even if the code I've written so far doesn't have balanced nesting yet. All
decisions and advice should be based on the speculation that the code will
eventually represent a new set of blocks, not a reorganization of the
remaining ones.

You can't make the same assumption about if statements inside of a function,
or about incomplete multiline comments, but you _can_ assume that anywhere
from zero to all of the sibling blocks are about to have their scope changed
(but not ancestors).

In short, the gap between how I'd describe a commit and how the diff tool
represents it also prevents live code analysis from being able to work as well
as it might.

~~~
matklad
That's not a particularly hard problem, all tools which try to solve it (for
example, IntelliJ), solve it easily. For example, here's the syntax tree that
rust-analyzer builds for this example:

    
    
        fn foo() { }
    
        fn
    
        fn bar() { }
    

Syntax tree (whitespace nodes elided for clarity):

    
    
        SOURCE_FILE@[0; 31)
          FN_DEF@[0; 12)
            FN_KW@[0; 2)
            NAME@[3; 6)
              IDENT@[3; 6) "foo"
            ...
          FN_DEF@[14; 16)
            FN_KW@[14; 16)
            err: `expected a name`
          FN_DEF@[18; 30)
            FN_KW@[18; 20)
            NAME@[21; 24)
              IDENT@[21; 24) "bar"
            ...
    

The problem is, a lot of tools are build with the assumption that the code is
correct&complete, and don't try to tackle this problem at all.

~~~
hinkley
When llvm was young, one of its design goals was to be useful for live code
analysis, to prevent that problem of having two parsers that might not agree.

Sounds like we lost some of that along the way.

------
zellyn
It would be interesting to develop Xi and rust-analyzer in tandem, and see how
much of an IDE Xi could become while leaving it still basically a text editor.

~~~
cmyr
Language features are definitely in the near-term plans for xi, and depending
on what stage Rust Analyzer is at it could definitely be a good initial target
for some of that work.

~~~
kzrdude
Does xi have any Vim/modal like features?

~~~
raphlinus
We're still trying to figure out the best way to implement those. There's a
prototype, but it hasn't landed in master. It's definitely on the roadmap
though.

~~~
zellyn
I thought Kakoune was an interesting experiment: an attempt to embrace vi-like
modal editing, but keep it consistent, and go all-in on it.

~~~
cmyr
The major innovation of kakoune, imo, is to combine modal editing with
multiple selections. These are two very powerful concepts that haven't been
combined well before, to the best of my knowledge. In any case yes, I think
kak is a great editor and a definite source of inspiration.

------
phlm
I find building solid foundation and tooling really important, and it seems
where are right there at this point with Rust. But IDEs have always been
controversial beasts. Some might say Visual Studio at one point was one of the
best representations of good set of trade-offs that made great code-writing
environment. Now it's different story. But what I am far more interested in in
terms of Rust and it's IDE ergonomics is what can be achieved differently?
Rust by itself is different beast. Unique language that pushed different way
of thinking and structuring code. So can that be used as an inspiration for
something different in terms of IDEs and developer UIs?

------
thorel
I am not sure I understand the criticism of RLS. It feels to me that the
Language Server Protocol does allow for incremental parsing etc. In
particular, the approach taken by rust-analyzer

> Analyzer maintains a "database" of input facts (source code + information
> about project structure) and derived facts (syntax trees, name resolution
> information, types). The client then can change input facts, and query
> derived facts at the current state of the world.

seems that it could also be adopted by a language sever for Rust.

~~~
matklad
The criticism are not about the protocol itself, but about a particular
implementation. rust-analyzer also uses LSP as an interface.

------
coldtea
> _The only fundamental problem with IntelliJ Rust is that it is not written
> in Rust, and I would really love to have a pure-rust solution for IDEs. Why?
> The main reason is that Rust is the perfect language for building these
> sorts of tools_

Citation needed.

~~~
matklad
I assume you are talking about the last bit? This is of course a personal
opinion anyone is free to disagree with. However in practice many popular
programming languages are implemented in C or C++ (clang, gcc, swift, CPython,
V8, SpiderMonkey, HotSpot) and in theory the best language for compilers is
[Oca]ml. Rust is a nice synthesis of theory and practice.

~~~
pjmlp
That is mostly by convenience than anything else, as many devs don't want to
bother with the whole process of bootstrapping a programming language.

Hotspot long term roadmap is to be rewritten in Java for example, aka Project
Metropolis. And not all JVMs are implemented in C or C++.

------
n-gatedotcom
As an end user, one thing I care a lot about is if the dev env would be truly
integrated. I want a one-click/command install of everything that I need for
my vim to be a rust IDE. I tried in vscode and had to install two different
plugins (one of which had a similar name to another and I had to go look up
which one to install) python and then lldb using admin permissions which
failed on my school computers .

Side: Has emacs been losing traction slowly? It's a third of vim!

~~~
theredbox
AFAIK it was always behind VIM. People learned vim because of its ergonomics
and install base.

Also many people did not know TRAMP is a thing so yeah...

~~~
n-gatedotcom
Wasn't emacs farther ahead for full-time programmers though? I'd love to see
if there were some historical stats.

