Hacker News new | past | comments | ask | show | jobs | submit login
Giving up on wlroots-rs (way-cooler.org)
283 points by Bl4ckb0ne 51 days ago | hide | past | web | favorite | 124 comments



A good post, thanks for sharing.

> I want to make one (mildly controversial) thing clear: rewriting a library for the sake of only using Rust is not good engineering.

Strong agree.

> A literal rewrite of a project to Rust is not interesting, it’s not useful, it just causes churn and splits ecosystems. Time would be better spent either working with existing solutions that already have the effort put in to make them correct or to come up with new green-field projects.

This... I'm not so sure about. It really depends on what your objective is. For example, if your goal is to learn, you're not going to cause churn, and you're not going to split ecosystems. Working on project you already know well is a good way to learn, because you can focus on the language, not the project.

This also isn't exactly a re-write, in my mind. I mean it is, and it isn't. This is because...

> The biggest problem when wrapping wlroots was defining the ownership model of the objects that wlroots exposes.

The pain here isn't a re-write, it's an integration with an existing system. That's a good reason to not use something different! I also think this is interesting because it demonstrates something that's often said in discussions about Rust, but mostly in the abstract, and that's that Rust's rules influence the design of your system. It will guide you away from designs where the ownership of components is unclear. To many people, this is a benefit, but it can often cause struggles when learning. And, it can often cause struggles in situations like this: where you can't really re-write some external component to fit in the rules.

That being said, there should be a way to do the ownership part that makes sense, but I don't know wlroots well enough to comment.

That being said

> Currently there is 11 THOUSAND lines of Rust in wlroots-rs. All of this code is just wrapper code, it doesn’t do anything but memory management.

is also super legit. Managing this kind of thing is a pain. I can certainly understand not wanting to do it.


I agree and was going to say the same. That there was a huge challenge in writing code to manage memory ownership, is not surprising because memory ownership is so fluid and adhoc in C and its derivatives. And when you are asked to make that ownership explicit, if the original designers hadn't been thinking about it, you get a lot of cases which they would not have considered.

I would like to see a compositor written in Rust, I think it would make for a robust window system. I also know that starting from the position of "this will be written in Rust" would force the issue of memory ownership to the fore and result in a different architecture from the start.


Not a wayland, wlroots or rust expert, but dollars to donuts says that it's not "memory" management[1] at issue, but resource allocation on the other side of the graphics driver. Vertex arrays, textures, framebuffers, shaders et. al. all need some kind of allocation strategy, can in the modern world often be shared between process contexts, and they really don't fit well into the metaphors of either C or Rust.

But where in C you can just do it anyway, Rust is likely to flip out over your wrapper object abstractions.

[1] I mean sure, technically it is memory management, but you know what I mean.


I have a very simple GPU abstraction in Pathfinder that has to manage lots of these objects, and it was easy and natural to get them to work. Sometimes I have to throw in a reference count here and there to make sure things are destroyed in the right order.

If the model doesn't fit perfectly into Rust's ownership system, I usually just do the checks dynamically and panic, or accept logic bugs (but not memory safety bugs), if things go wrong. Is that admitting failure? Perhaps. But it's also just choosing my battles. I'm trying to keep my code clean and ship, not prove it correct. Memory safety problems are pernicious enough that it's worth significant effort to rule them out, but almost everything else is a judgment call.


This is also something that causes some pain when using C++ with RAII. Logically it makes some amount of sense that you have e.g. a C++ wrapper for a texture in OpenGL, and it knows to call glDeleteTextures in its destructor.

HOWEVER, the destructor can now only be called if the OpenGL context is active, and the texture may be deleted if the context is lost (which can happen). At some point anybody who's tried to wrap an OpenGL texture with a C++ class either decides to accept that RAII doesn't completely work here, or refactors it so that you manage something like handles to textures, which is a bit silly because you are really at that point managing handles to handles to textures.


re: the handles to handles stuff ... I don't think that's so silly really, when you are talking about a resource that in some fundamental sense belongs to a different system (in this case OpenGL). Sure, that system gave you a handle, but if the semantics of managing that handle aren't 1:1 matched with your languages model of things, another level of indirection is a clean way to handle it.


And just think about what most file and I/O wrappers, that are part of the standard library in most higher level languages, e.g. file objects in python, streams in C++ etc., naturally do: They also just wrap (for example) POSIX file descriptors, which are handles, with their own handles, i.e. the file object/Stream/whatever.


Maybe I didn't explain what's happening in OpenGL properly, because that comparison doesn't work at all.

With e.g. std::ofstream, you are wrapping a handle, such as a POSIX file descriptor. So that part of your description is accurate.

However, this is different from what happens with OpenGL. The problem is that if you wrap an OpenGL texture (which is a handle), you end up with some problems because you can only free it with the correct context active, and it might become free for other reasons besides the wrapper being destroyed. So you are no longer wrapping a handle (like std::ofstream), but you are actually creating a new handle-to-handle, and wrapping that, and explicitly managing the lifetime OpenGL resources with a separate object somewhere else.

Of course, you could just wrap an OpenGL texture, explicitly decide not to handle context invalidation, and then be careful to ensure that your objects get destroyed with the OpenGL context active. You lose some flexibility and you have to babysit RAII to be sure it "does the right thing".

So what I'm saying here is that OpenGL textures are not like file handles.


They don't behave like file handles - but at a certain distance if you squint at it looks like the same problem. The real issue isn't that OpenGL textures don't behave like file handles, but that neither of them behave quite like objects in your language, which can lead to problems. The specifics of how the behavior differs is I think less important that the fact that it differs. Of course, file handles are not the best example because the language designers often had to put some thought into it early on; less true of something like OpenGL textures.


> …but at a certain distance if you squint at it looks like the same problem…

The whole discussion here is about how these things differ in subtle ways. The problem with “squinting” at the problem is you end up e.g. using RAII and then having to redesign your system because RAII doesn’t match, and the problems weren’t obvious when you started out.

Not to make too fine a point about it here, but file handles are perfect matches for RAII / Rust lifetimes, unlike OpenGL textures. You just close the handle at end of object lifetime. You can get errors when you close but current recommendation is to ignore errors when cleaning up and have a separate path for closing/committing data on close when writing, so on most paths the destructor will nop. This works well.

An example of a mismatch with the language semantics and file handles is with GC. If you close a file handle in a C++ destructor or Rust drop(), it’s fine, those are run deterministically. If you use a JVM finalizer to close your file handle that might not happen soon enough (cue EMFILE).

Again, the reason why textures don’t work this way is because you have to release them in a certain context and they might be released outside your control.


Ok, file handles are often good matches, unless you start allowing for them being changed outside of the file object, e.g. being closed or their position pointer modified (by e.g. seeking or reading).

So my example might not have been the best, because usually, they have a closely matching lifecycle, but the greater point that I was trying to make is that managing handles of handles (of handles) is not weird if you have to do some impedance matching. Maybe file handles are often straightforward, and therefore only require one "additional layer of handle indirection", but there's probably other good examples like say, threads in thread pools, or a whole bunch of stuff that's going on inside an OS kernel, especially a multiprocessing one.


> Not to make too fine a point about it here, but file handles are perfect matches for RAII / Rust lifetimes, unlike OpenGL textures.

Agreed, something like an FD matches RAII very well, but you can open one file multiple times, dup fds and multiple processes can do all this. So the FDs end up mapping in some non-trivial, probably refcounted way to other objects in the kernel.

So it seems with OpenGL, this kind of mapping / bookkeeping layer gets hoisted up into the application framework -- which might be a reasonable choice.


OpenGL textures point to some refcounted thing, but the handle table is not per-process but per-context, and the contexts are not thread-safe, and the contexts can become invalidated. So it is similar in many ways but there are too many differences in general.


Yes, I am talking about this at a higher abstraction level than the specifics only of c++ & OpenGL textures. I guess that wasn't clear enough.

I was only attempting to make that point that wrapping handles with other handles, rather than being silly, is a pretty good pattern to address this problem, and it shows up all over.

The file handle thing I already pointed out; there are better examples.


> ... you end up with some problems because you can only free it with the correct context active, ...

I wonder if this is where Rust can actually help. I.e. could the lifetime tracking in the language allow you to construct a library where the unref was guaranteed to happen only when the right context is active?

And if such a library could be constructed, what would the loss of flexibility cost?


Incidentally, some of the people working on safe OpenGL wrappers ran into similar issues with the blog post, thanks to similar problems.


Right, it's the same basic problem, and a good pattern to solve it.


The problematic pattern I see here is whenever you manage remote state, your local guarantees about how that state changes just don't work.

You would love to have the abstraction of a standalone texture and be able to pass it around as a value and change it's ownership, but you never really owned it. The only thing you ever owned is the context (or the connection in the case of the wayland protocol). The texture you borrowed. Freeing your texture is equivalent to returning it to the context you borrowed it from. Of course not having an explicit context in the OpenGL API makes this much harder :)


Yes, a similar issue exists in C++ too when managing some types of resources in database kernels. Some types of resource management are a poor fit for idiomatic ("safe") resource management in C++. In that language, you always have the option of custom, non-idiomatic resource management -- everything you might want to do is always possible in principle. You avoid it to the extent practical but there are a few times when this kind of resource management is unambiguously the correct software design and engineering choice. It is a bit like "goto". Rarely used for good reason but there are times when any other implementation will be decidedly much worse.

I've hashed this out extensively when looking at implementing a database kernel in Rust (I’ve had Rust experts among my C++ programmers). Technically it is possible to implement these things in a safe way in both C++ and Rust. However, it comes at a cost of an unreasonably complex and inefficient software architecture in either language, so you would not want to actually do it that way in a sane code base.


Wayland uses "objects" that can appear and disappear and graphic driver is unrelated here. For example, wayland has a "registry" and it sends notifications to clients when new objects appear there or disappear from it. I assume this unpredictability could become the problem for the author.


Author here.

>> A literal rewrite of a project to Rust is not interesting, it’s not useful, it just causes churn and splits ecosystems. Time would be better spent either working with existing solutions that already have the effort put in to make them correct or to come up with new green-field projects.

> This... I'm not so sure about. It really depends on what your objective is. For example, if your goal is to learn, you're not going to cause churn, and you're not going to split ecosystems. Working on project you already know well is a good way to learn, because you can focus on the language, not the project.

I was a little extreme with this comment, and I knew there were a bunch of footnotes like the ones you mentioned (pedagogically reinventing the wheel is great for learning!) I explicitly didn't mention those so I didn't water down my point.

Ultimately people can do what they want, and it won't really bother me. My comment was more of a critique on what is and isn't worth other's time. That is for them to decide, at the end of the day, it's just my opinion that I think they should question if all the effort they are going through is worth it. If they think it is, I wish them luck.

> The pain here isn't a re-write, it's an integration with an existing system.

I was conflating two ideas here that in retrospect I should have been clearer about.

wlroots-rs was definitely _not_ a RiiR, it was bindings. However as I came to butt against these problems it became obvious why other projects (like rlua and wayland-rs) are RiiR: because writing bindings is so difficult it is easier to start from scratch. At that point you are now rewriting a library for the sake of using Rust, which seems like a problem to me.

I don't know what Rust can do to make it easier to write these bindings. I think it's very important, but not something that has been focused on in the community because most of this work is very niche and there are other problems that are probably more interesting (async, web assembly, etc.)


btw, wayland-rs is both RiiR and bindings currently (native_lib cargo feature), and I've used it with libweston for custom protocols!

My project https://github.com/myfreeweb/weston-rs was also abandoned though. For a much simpler reason — I realized that I didn't need to reinvent the wheel — https://wayfire.org is everything I could possibly want from a compositor :)


I'm surprised to see you agree strongly. A Rust re-write for the political/ideological reason of "only using Rust" seems not good engineering by definition - it is not an engineering decision.

However, most of the time that someone would rewrite for the sake of using Rust, it would naturally be to get the advertised benefits of the language. If you believe you will have safer, more manageable code in pure Rust, you are making a valid choice.

The fact that efforts will be split is unfortunate, and should factor into such decisions. But that doesn't seem a strong argument on its own - if you believe such a change is for the better then there's no way to avoid breaking some eggs.


I think we're differing on semantics.

> it would naturally be to get the advertised benefits of the language

I think "naturally" is doing a lot of work here; this is not "for the sake of using Rust", it's to gain the benefits of the language.


I agree, but naturally I think that’s the natural interpretation :)

Speculating: I think many Rust rewrite projects are criticised under “for the sake of it” when in fact there is a clear decision that Rust is just a better choice in 2019 (i.e. the decision was not political, but the criticism assumed it was). Many of those will nonetheless peter out because forks are hard. Such a result is only a partial judgement on the original decision.


> I think many Rust rewrite projects are criticised under “for the sake of it” when in fact there is a clear decision that Rust is just a better choice in 2019

Yes, I think that's true as well.


I worked on projects that required complicated FFI bindings to C/C++ libraries. Creating, testing and maintaining such bindings always ends up a huge undertaking. Now I'd think twice. I imagine something like Wayland will have hundreds of functions and if one doesn't want to write C in Rust (which isn't that bad actually) then wrapping up such API is a Herculean task!


I'm surprised nobody mentioned this yet- a rust rewrite means you are more likely to invite contributions from a larger community.


> rust rewrite means you are more likely to invite contributions from a larger community

I'd be really interested to know why you think that's the case. Somewhere around 3% of programmers know Rust, while over 20% know C. https://insights.stackoverflow.com/survey/2019#technology

Naively, I would have thought that a C project would invite contributions from a much larger community. Sure (as I've personally experienced!) the Rust community is friendly and helpful, but we're still small, so I'm curious why you think a rewrite would get more contributions.


Not parent, but I'd think twice before contributing to a C code base I'm not completely familiar with in fear of causing some bug/CVE. Rust projects are much nicer for drive-by contributions to projects one uses but isn't generally interested in working on. If you are not working on a security critical part of a system you can be pretty sure not to cause severe bugs like RCEs.

Also for pure rust projects the development environment is easier to set up: cargo+rustc do everything vs. autotools+make+installing system wide libraries+ccache+gcc (just what I've seen in the wild).


It is nice to think that 3% of programmers know Rust, but we all know that number is at least two orders of magnitude too large. It would be impossible to defend a claim that 3% of programmers know that Rust is a language.


That's not how OSS contributions work. C# is wildly popular in enterprise, and most of New york runs on F#. How often do you see .NET repos on Github?


> How often do you see .NET repos on Github?

Well, ok, if we're going by GitHub repos, C has 117k repos with at least one star and Rust has 11k. That doesn't seem to be that different than then number of programmers as a whole.

(FWIW, there are 104k starred C# repos and 2k F# repos, which doesn't seem all that out-of-line with the popularity of those languages as a whole.)


I know very little about Rust or Wayland (I suppose I'm not the target audience) but I got lost very quickly here:

> A Wayland “output” is the resource that represents a display device. Commonly this means it handles a computer monitor. This resource could disappear at any time in the life cycle of the application. This is easy enough to imagine: all it takes is a yank of the display’s power cord and the monitor goes away.

Surely even if a physical monitor is connected from a computer, the object representing it doesn't instantly go away? If it literally got freed as soon as the user disconnected the monitor then any access to such an object would be dangerous as you could be accessing an object after it's freed, or even another valid display object that got allocated into that space in the mean time.

Instead, I would expect an object in that situation to go into some error state, and even that might only be picked up when you perform certain operations. If that were the case, I don't really see how Rust lifetimes are a problem.

Since this was the main summary of the problem for laypeople like me, it made the rest of the article quite hard to follow.


I think you missed this part:

> [the resource handle] can only be dropped between event callbacks, wlroots/Wayland is callback based

So "at any time" means between any two user callbacks, but not during them.

Error states really aren't a solution IMO, as they tend to pollute the entire interface in that methods that would be expected to work unconditionally can now fail or panic. Either you have to check each method invocation (entirely unnecessarily, since if a resource isn't in an error state at the start of your callback it won't go into one during it) or you make the absence of error state a precondition, in which case you're guaranteed to miss one or two cases and panic when the user yanks out the cord.


> > [the resource handle] can only be dropped between event callbacks, wlroots/Wayland is callback based

Thanks, that certainly makes a hell of a lot more sense.


Picture a pointer to video memory. Or, simpler, picture a pointer to an SHM section that either side of the SHM IPC conversation can deallocate. From both sides’ perspective, that pointer is probably implemented as both an SHM section, but also an SHM pointer to the section, such that either side can set the SHM pointer to NULL, and then (if they managed to do that) proceed to tell the SHM infrastructure to unmap all mappings of the SHM section.

When this sort of structure is used, there are certainly going to be guarantees in place (probably by using some sort of session/transaction functions that compare-and-swap an atomic SHM spinlock controlling the SHM pointer) such that the SHM pointer won’t go NULL [and the thing it points to won’t be deallocated] in the middle of either side writing to it. But those functions aren’t actually consuming the resource and spitting out a new temporary one (i.e. a “handle” to the resource, as the article’s author implemented in their Rust wrapper library originally.) Instead, they’re just functions that block either side from writing to the pointer as long as you’re in them—sort of like disabling interrupts in a critical section.

How do you model the ownership of such a C-FFI-runtime-guarded IPC SHM volatile pointer-to-pointer, in Rust? Is there an idiomatic translation for it?

Because this sort of thing comes up all the time in the context of kernel handles to buffers, and I would be surprised if the folks writing OSes in Rust haven’t hit on it before.

IIRC, there’s an IPC abstraction called a ‘blackboard’ (sort of related to a tuple space) that is the generalization of this SHM model, so it might also help to ask how you’d model an IPC ‘blackboard’ in Rust.


Quick answer, without understanding all the details: a weak-pointer-like structure that performed all the necessary locking and checking before giving out access to the underlying SHM.

Rust's borrow checking is more or less a formalization of C++'s RAII style; in my experience, solutions for that translate relatively simply.


Rust's borrow checking does not correspond to RAII. A simplified explanation is that borrow checking gives you the guarantee that mutable references are unique and immutable references do not change.

Rust's lifetime system is vaguely like RAII, but the C++ type system gives you no way to create an object with lifetimes that don't correspond to some scope. In Rust this is done by moving values, but in C++ this is not possible, you have to fake it by using std::move(), which really just creates an rvalue reference.

One of the big things that causes problems in C++ is iterator invalidation. This is not solved with RAII, but it is solved in Rust with the borrow checker. The price you pay is that iterators in Rust are strictly less powerful than iterators in C++, because the way C++ iterators work cannot really be expressed in the Rust type system. (In short, C++ lets you have as many iterators as you like into the same container, and defines ranges as pairs of iterators. Some algorithms are more naturally expressed this way.)


> C++ lets you have as many iterators as you like into the same container

While it's true that Rust doesn't let you have multiple mutable iterators, it's worth pointing out that you can certainly have multiple immutable iterators.

    let items = &[42, 101];

    // All products (a*a, a*b, b*a, b*b)
    for x in items {
      for y in items {
        println!("{}", x * y);
      }
    }

    println!("");

    // Pairwise products (a*a, b*b)
    let pairs = Iterator::zip(items.iter(), items.iter());
    let pointwise_products = pairs.map(|(x, y)| x * y);
    for product in pointwise_products {
      println!("{}", product);
    }


That's really not what I'm talking about. In your example you are using multiple iterators to iterate over the same structure multiple times, but in C++ you can use pairs of iterators to represent ranges. For example, I can have three iterators i, j, and k, which represent two ranges: i..j and j..k.

Even if i, j, k are const iterators, it's not pleasant to translate this to Rust. So it's not an issue of whether Rust lets you have multiple iterators or not, the issue is that Rust iterators are strictly less expressive than C++ iterators.

And that's okay. It's a tradeoff.

For more information about how iterators are used in C++, I would refer to the <algorithms> portion of the standard library. https://en.cppreference.com/w/cpp/algorithm


Ah, I understand. You're referring to how iterators in C++ are generalized indices into their collections. Certainly, Rust iterators have a more constrained purpose.

I personally prefer having separate constructs for iteration and indexing, so I think it's a matter of taste.


Is that different from something like this?

  fn main() {
      let list = &[0, 1, 2, 3, 4, 5, 6, 7, 8];
      let i = 0;
      let j = 4;
      let k = 8;
      let iter_ij = list[i ..= j].iter();
      let iter_jk = list[j ..= k].iter();
      let pairs = Iterator::zip(iter_ij, iter_jk);
      for (a, b) in pairs {
          println!("{}, {}", a, b);
      }
  }


Yes, that is different, because in C++ you can move j forward and backwards.


Ahh, I see, thanks for clarifying. Rust's Iterators are similar to C++ ForwardIterators. It seems C++'s default is BidirectionalIterator.


Even C++ ForwardIterators are more powerful, because you can move them independently. Rust lets you move the beginning of the range forwards, and maybe the end of the range backwards. With ForwardIterator you would be able to move the end of the range forwards, making the range longer.


> but the C++ type system gives you no way to create an object with lifetimes that don't correspond to some scope.

Counterexample: a data member of a class has a lifetime tied to the lifetime of the containing object instance, rather than to a lexical scope.


You are just using the same storage duration as the containing object instance, and the C++ type system only gives you any guarantees for automatic, static, and thread storage duration. These all correspond directly to lexical scopes. The type system does not have a way to express dynamic storage duration, but Rust does because Rust has move semantics in the type system. C++ has move semantics, but they are runtime semantics built into the implementation of the library and not part of the type system, so you get no compile-time checking that you used move semantics correctly.

Phrased another way, I'm just saying that the C++ type system only expresses two types of storage durations: those tied to lexical scopes and dynamic storage durations, and provides no checks for dynamic storage durations.


Yes that's the real solution to the problem.

If you have to access something in lots of code, don't make it go away asynchronously. There are many techniques to archive that. Keeping objects around in an error state is one easy way to do that.

You never want memory management to be too fine grained. Even if he could implement the fine grained memory management it would likely be impossible to test all the corner cases.


I can't speak to every issue which the author might have encountered, but there is a better solution to the lifetime management problem than the two mentioned in the article.

Instead of this:

    fn some_wlroots_callback(output_handle: OutputHandle,
                             surface_handle: SurfaceHandle) {
        output_handle.run(|output| {
            surface_handle.run(|surface| {
                // maybe some more nested layers...
            }).unwrap()
        }).unwrap()
    }
One can do this:

    fn some_wlroots_callback(
        ctx: CallbackContext,
        output_handle: OutputHandle,
        surface_handle: SurfaceHandle
    ) {
        let output = ctx.get(output_handle);
        let surface = ctx.get(surface_handle);
    }
This is safe, because the lifetime of "output" and "surface" can be bound to the lifetime of the "ctx" (whose lifetime is controlled by the library: the library simply has to make sure that "ctx" is not accessible outside of a callback).

edit: Realised you can't tell that there's an implicit lifetime in `CallbackContext` here:

    struct CallbackContext<'a> { ... };
OR

    type CallbackContext<'a> = &'a CallbackContextImpl;


Author here.

The problem with that design (which is a great design given what I presented in the article by the way!) is that it doesn't allow you to share handles across callbacks, which is mandatory to do anything interesting. I'm assuming here that you can't use the handles except for that callback context. If you can, then that presents a different problem.

https://play.rust-lang.org/?version=stable&mode=debug&editio...

If you can own a context, even with a lifetime parameter it's possible to leak it using the Box api. That allows you to have access to a &'static CallbackContext. This will break that assumption that it only lives as long as the callback itself.


The intent is that handles can be used across contexts (as long as both contexts are from the same instance of wlroots) - the context checks that the handle is valid before unwrapping it. Could you clarify what you mean by it presenting a different problem?

With regards to the leaking issue - that's not actually correct. `Box::leak` does not give you access to the context with the `'static` lifetime, assuming the lifetime of the context is not 'static (that would fundamentally break rust's safety guarantees).

The only potential problem is that the destructor for the context may not run (cf. leakpocalypse). If the implementation requires the destructor to run, then you are right, you should never give ownership of the context object, it should be passed into the callback by reference. If you need access to a context outside of a callback, then you should use the callback-style:

    with_context(|ctx| {
        ... do stuff ...
    }
The nice thing is that `with_context` can enforce any threading constraints you might have (like contexts being accessed from a single thread), can ensure that the context is safely dropped at the end, and it's only required once - once you have a context you can use it as many times as you need.


Your edit to your original comment was important information I was missing - you are correct that that would probably work.

With how wlroots-rs is setup it does require the upgraded resources to not be leaked because their drop impl decrements the reference count.

Regardless, this was obviously only one part of the problem I was running into. As well, as can be seen, it's very complicated trying to ensure this is all sound and it's no longer worth the headache.


>If you can own a context, even with a lifetime parameter it's possible to leak it using the Box api.

The `std::pin::Pin` API prevents you from doing this (callbacks would receive `Pin<&'_ mut CallbackContext>`).

>The problem with that design (which is a great design given what I presented in the article by the way!) is that it doesn't allow you to share handles across callbacks

But how does sharing handles between callbacks look like? Is it something like "callback FooCreated provided me a Foo handle" and "callback BarCreated requires me to have the Foo from the FooCreated callback" ? So your FooCreated callback needs to save the Foo handle somewhere that it can be reused later?

If so, why is it not enough to have a `fn CallbackContext::set_foo_handle(&mut self, Foo)` (or `Pin<&mut self>` based on the above suggestion) ?


If you want to see some examples, Neon uses this pattern to solve a similar problem - preventing rust code from holding onto js objects that might get gced between callbacks.

https://github.com/neon-bindings/neon


I just want to say, having spent a bunch of time writing a context-management framework for embedded targets that I jettisoned because it sucked, this is a great approach I'm excited to experiment with. Thank you for the comment.


Interesting to see someone's reasons for moving back from rust to C. Normally it's the other way round.

Speaking as a C developer who has dabbled in Rust, It's always good to see the thoughts of someone who has spent a fair amount of time with the language who can give it a fair assessment.


> Way Cooler is a Wayland compositor that was written in Rust using wlc

I know it is not easy, but I wish the author could have started with a paragraph that could help someone like me know whether or not the rest of the article would be something I would like to read.

How about something like this (and of course I may some of the facts wrong, but I want to be as constructive as I can):

"Wayland is a Windows manager developed as a better alternative to X-Windows. Way Cooler is a tiling Wayland manager, written in Rust, and designed to be easily extendible. In this article I describe my experience in trying to refactor it, for reasons that will be described below. I will also explain why, in certain instances, C was a better choice for this project than Rust."


You weren't the audience of this article. Since he wrote it as a blog post on way-cooler.com he surely was expecting his audience to be people familiar with Way Cooler, Wayland and Rust. He doesn't have a responsibility to dumb it down for you. And anyway it only takes a few minutes for you to get the context. Which you did, good job.


So it's way-cooler.com's fault for not explaining what their library/website is about.

Which unfortunately is par for the course on 99% of company blogs, which never explain what the company does without going onto the homepage (which even then is often confusing).


It's not a company blog. It's a some guy's open source side project. Ya' know... free... open source... side project... that's he's voluntarily sharing technical information on.

Really, it's not the guy's responsibility to over-inflate the article because some people need their hand held. It's a blog about about Rust, Wayland, and window managers. If you don't know what those things are then, like, Wikipedia is there for you, man.

It's like going to a programming languages website and going "well, gee, these guys suck for not explaining what a programming language is on their homepage what is with everyone being so discourteous!?"

And it's not like a paragraph of context would have mattered anyways; if you don't know what Rust is, what Wayland is, or what tiling window managers are then you'll need a lot more background knowledge before the article's content begins to be assailable.


Definitely. So often, whole articles and even much of the ensuing discussion will be using project names and acronyms that I don't know about. A sentence or two of context, or even just links going off to the project/acronym the first time it is used, are wonderfully useful for the wider audience.


There are links in the first sentence!


So tempting to put a link to jwz.org here.

But seriously, this sounds more like the responsibility of the link submitter and/or this website then every person on the internet that wants to write a blog post for a limited audience.


I believe there are two parts to this argument:

1. articles should start with a brief summary of their context, arguments and conclusions.

2. articles should explain their terminology.

The first point I agree with (and believe most do). But the second point is arbitrary, and in my personal opinion, the linked article is fine in this regard (anyone familiar with X-Windows should be familiar with Wayland as a concept, and every single article related to Wayland cannot be expected to explain core abstractions like compositors).


I may be missing something, but what's wrong with the first example? The fact that you can leak an object shouldn't really matter for memory safety if it's just a handle to an object that the server manages. Yeah, it could go wrong, but it won't be unsafe. Object handles are essentially file descriptors, right?

In general I think there's a tendency to overcomplicate safety features in Rust. The solution to an overly-complicated system isn't to throw the whole notion of safety out the window: it's to look at exactly what the complexity is buying you. If intricate combinations of Rust features are one extreme of the safety spectrum and C is the other extreme, there's frequently a happy design medium somewhere in the middle.

Edit: Looks like oconnor663 over on Reddit had a similar but more specific proposal, which probably works: https://www.reddit.com/r/rust/comments/biq864/comment/em2kip...


Yes, the same concept is discussed in https://news.ycombinator.com/item?id=19779243


"A Wayland “output” is the resource that represents a display device. Commonly this means it handles a computer monitor. This resource could disappear at any time in the life cycle of the application. This is easy enough to imagine: all it takes is a yank of the display’s power cord and the monitor goes away. [Except it can't; it can only disappear between callbacks?] This is basically the exact opposite of the Rust memory model. Rust likes to own things and give compile-time defined borrows of that memory. This is runtime lifetime management that must be managed in some way."

Something leads me to believe that there is something very wrong with the architecture of wlroots-rs. The output should be attached to a callback parameter or something, maybe? I don't know enough about Wayland to say, but something ain't right.


Author here.

This output is attached to a callback parameter, but you can takes this resource from the callback in C (because it's just a pointer you copy around) and use it in other callbacks. Eventually a "special" callback that will trigger to indicate that the data the pointer refer to will be cleaned up and you need to remove all of your references to that resource because otherwise they will be dangling.


This code

    fn some_wlroots_callback(output_handle: OutputHandle,
                             surface_handle: SurfaceHandle) {
        output_handle.run(|output| {
            surface_handle.run(|surface| {
                // maybe some more nested layers...
            }).unwrap()
        }).unwrap()
    }
is just screaming continuation monad. You see the same code pattern in early Node.js code (sometimes leading to callback hell). In the JavaScript world, the problem was solved using Promises, and then async/await syntax. But more fundamentally, this is an instance of the continuation monad at work.

The continuation monad transformer is defined as

    newtype ContT r m a = ContT { runContT :: (a -> m r) -> m r }
and is nothing more than just a function that takes a callback. If this were Haskell, one could just write

    stuff = runContT $ do
      output <- ContT (run outputHandle)
      surface <- ContT (run surfaceHandle)
      -- and then maybe some more nested layers
      -- etc
Granted, continuation code can easily be misused to produce an incomprehensible mess in Haskell (its full generality can be compared with goto), but with Rust's FnOnce trait, the scope for misuse is considerably reduced.


JavaScript and Haskell get to enjoy the productivity of using a tracing GC though.


This doesn't really have much to do with the runtime. Certainly with a GC, programming would be easier, but I'm really talking about abstractions within the language. Rust already has language-integrated support for the Result monad. Rust doesn't have a general `do` syntax, but it has `?` which has made programming with the Result monad much easier. Can we think about the continuation monad and arrive at a new syntax that can ease this style of programming?


i don't really know Rust apart from some curious onlooking, but afaik expressing monads (i.e. the Monad typeclass) in Rust is tricky – something to do with the lifetimes of closures and the objects they close over, i think. ([Idiomatic Monads in Rust] probably touches on that). it might be possible to just build it into the language a la Result or async/await though.

[Idiomatic Monads in Rust] https://varkor.github.io/blog/2019/03/28/idiomatic-monads-in...


It has, because for some of those workflows the borrow checker gets in the way.


Question (for any more knowledgable readers here) from someone with a somewhat shallow understanding of the topics discussed:

Does this end up being primarily a negative reflection on the general structure of:

(a) Rust

(b) wl-roots

(c) Wayland

(d) all three

(e) none of the above, it's merely the incidental reality of trying to write code that's compatible/usable across multiple language ecosystems and none of the 3 projects can do much to improve this situation.


> Does this end up being primarily a negative reflection on the general structure of

A core problem in this case is wlroots using a memory management paradigm that isn't easily modeled in Rust. This isn't unexpected per se, since C leaves MM entirely to the developer, while Rust is opinionated.


I would say it's a bit more subtle than that. Rust can express this, but not safely. This is what the end bit is about.

The question then becomes, is it worth it if it's largely unsafe? That's a complex question. Unsafe Rust still does give you a lot of advantages, namely that the checked constructs are still safe, even in an unsafe block. Unsafe Rust is slightly more annoying to write than safe Rust, and so that's a downside.

It's also possible, and again, this is more in theory since I know nothing about wayland internals, that the safe abstraction was chosen to be a bit too low-level. That is, rather than trying to make the primitive operations safer, designing an external API you'd want users to use, rather than one defined in terms of some of the primitives, may make sense. This has a lot of pros and cons, as you'd imagine. And that's also more work to do.


Author here.

Ignoring the social impetus in the Rust community to not use unsafe, I also don't feel like unsafe Rust is something I want to program in all the time.

When I program in safe Rust I can be happy once it compiles because I can ignore all of the safety problems that come from C and C++.

However in unsafe Rust not only is it much more difficult to express what I want syntatically (the lack of auto deref is very annoying, having to write (*base).value all the time gets very old) and semantically (there is no standard for the unsafe parts of the language - not so much a problem if only smallish parts of this usage is used (because once a standard comes out just that can be updated) but a problem if a whole program is written in it).

Unsafe Rust is "good enough" to try to encode these abstractions but I would not use it over C or C++.


Speaking as someone who dabbled in Wayland in Rust before (but a few years ago so my knowledge might be outdated), has not tried wl-roots, and uses Rust a lot

(a) Only when talking about interacting with C libraries written in a certain style. So not in general, and not with most C libraries, but yes for implementing wayland compositors.

(b) Don't know

(c) Absolutely a negative reflection, there are two ugly things going on here. The official wayland libraries use a ownership style that really only makes sense in C imho, and despite the claim that wayland is just a protocol you can't actually not use the official libraries, because the drivers only work with the official libraries.


>(c) Absolutely a negative reflection, there are two ugly things going on here. The official wayland libraries use a ownership style that really only makes sense in C imho, and despite the claim that wayland is just a protocol you can't actually not use the official libraries, because the drivers only work with the official libraries.

Not really sure what you're talking about here. Wayland is just a protocol, and a pretty simple one to boot. There also exists a pure Rust implementation of the Wayland protocol:

https://github.com/Smithay/smithay

The problem is that Rust and libwayland, the C implementation of Wayland, don't get along well, and that wlroots is designed to work with libwayland and inherits a lot of those design decisions that make it difficult to deal with in Rust. And to be honest, Rust is special in this regard. Other wlroots bindings exist for Go, Haskell, Common Lisp, OCaml, and Chicken Scheme - and all seem to do fine.

I think this points more to a failing in Rust, in that it's not designed to cope with this particular model well. Since this is a fairly common model, that seems like a big flaw.


> Wayland is just a protocol, and a pretty simple one to boot.

For a Wayland client to render using hardware acceleration, it has to link to an OpenGL library (typically mesa, specifically EGL to set up the OpenGL context). Mesa (a library written in C) links to libwayland-client (also written in C). The problem is that mesa is expecting a pointer to a C struct that is defined in libwayland-client. It will cast that pointer and call libwayland-client functions.

Mesa could have been written to use a different level of abstraction (e.g. take a struct of function pointers) it would be possible to wire it up with your own wayland library.

It doesn't help that libwayland-client was designed to be stateful (since the protocol itself it stateful).

So, while technically you can write a wayland client without using libwayland-client, you cannot create an OpenGL context without rewriting mesa. You also cannot link directly or indirectly with ANY libraries that use libwayland-client.

So people give in, use C, and move their abstraction layers higher up in the stack.

Edit: grammar.


Thank you for spelling out the details, I knew the issue existed but forgot the details.


At a glance, smithay depends on wayland-server depends on the official wayland libraries. More specifically the issue that makes this necessary is that it's the only way to get an OpenGL context (AIUI - it's been quite awhile since I worked on this).

Rust is special in that's it's even worse at representing it, but the C controlled event-loop/fd-based-dispatching/ownership model wayland uses isn't idiomatic in any language other than C. Maybe wlroots makes this better, like I said I've never used it and can't speak to it.


Smithay has both a pure Rust and libwayland-based implementation, you can pick iiuc.

And the OpenGL problem isn't fair, OpenGL is such a flaming heap of poor design that binding to libwayland to use it is the least of your problems. Better to use Vulkan instead for this purpose imo.

Event loops are common in many languages.


> Event loops are common in many languages.

Including rust - I'd have to dig in again to remember exactly why the wayland one was so bad at interacting with rust to be honest. I know it had something to do with how ownership of callbacks and objects interacted with the event loop. I know I didn't think I could represent it much better in python.

OpenGL may be a shit show, but it's necessary (to be able to expose it to clients) for a modern desktop. I think I was doing this either before or in the really early days of Vulkan so I don't really know how that changed things.


Rust can handle event loops just fine; our entire asynchronous IO story is based on them!


I read it as friction between how Rust sees its application model and the real world. Total ownership of a resource can't be maintained if some jackass goes and pulls the physical hardware out from under you. So now the Rust implementation gets crufted up with a zillion checks for unexpected changes to the underlying system that are clunky in Rust because that's supposed to be an exotic condition, not something you're doing on practically every line.


This can also indicate a problem in architecture: if the monitor can go away any time, you shouldn't store long time references to it and instead, for example, call a function to get current default monitor every time you want to do something. Or at least validate those references before usage.


I suspect querying the monitor for its parameters every time you want to blit pixels to it is wildly impractical. EDID runs over a slow serial link, and you have 124.4 million pixels to blit out every second.

The sensible way to handle this is for the system to throw an interrupt when the state of the connection changes, but interrupt handling is something Rust's ownership model struggles to accommodate.


It reflects on the suitability of Rust as a systems programming language.


Interesting post that show some current limitations of Rust, especially when not being able to control the design entirely.

Rust's ownership system makes it harder to deal with the kind of dynamics wlroots exposes... It seems to me that the issue here is that there is no consensus (e.g library) on how Rust should deal with those issues, so if you want to just to provide bindings to a dynamic system you end up writing middle-ware that deals with that kind of dynamics in a Rust idiomatic fashion (taking advantage of ownership) instead of focusing on the task at hand. It somehow destroys Rust's promise of productivity.

A Rust project of mine[1], in an early stage and currently paused, deals with those issues (probably in a way similar to the Handles described in the post) by providing Service smart pointers (i.e, Svc<T>) and a component-framework that helps deal with the dynamics of a service being unregistered (e.g, a monitor unplugged, or a dynamic library unloaded) ; that component framework helps you be sure that if your struct contains a Svc<Foo>, that service is still available when you're called, or the component reaches an invalid state.

In general, it seems to be an area where Rust's ecosystem is still very early and would benefit from more input (such as this post) and consensus.

Regarding the callback-hell issue, that "dehandle" macro looks very much like async/await, and it looks like it could be implemented either in terms of async/await (still unstable) or generators (even more unstable).

Hopefully similar projects will be more likely to succeed as Rust and its ecosystem mature.

[1] https://github.com/magnet/socrates-rs


How does Rust handle these problems in the context of file I/O? I'm sure there is an axiomatically idiomatic (official) implementation of it as part of the language distribution, and at the same time file handles seem conceptually very similar to display handles and should have similar failure patterns (space can run out, a disk can fail, a plug-and-play disk can be yanked out at any time...).


That is all here: https://doc.rust-lang.org/stable/std/fs/struct.File.html

These failure patterns are handled by each method, for example,

  pub fn open<P: AsRef<Path>>(path: P) -> Result<File>
that Result will return an error if the file can't be opened, etc. Let's say you want to write some bytes, that's

  fn write(&mut self, buf: &[u8]) -> Result<usize>
This also returns Result, so if you've opened the file, but the disk is now out of space, this will return an error, etc.


I only skimmed the article, but the description of outputs and their asynchronous lifecycle making things complicated just made me think of what's become known as the ECS pattern in game development.

Your outputs sound just like entities in a game. They come and go as they please, and you often have multiple references to them from myriad places since entities may interact with many parts of the game.

I suspect if you investigated the established techniques for implementing ECS-style games in rust, you might find some simpler and established solutions for your troubles.


Personally I think Rust is a great language. That said it may not be great for everyone nor a great fit for every problem.

Sometimes trying and admitting failure is a perfectly rational and valid option.


Everything the author and the commentors here have said is completely legit. I'm not looking to start a war. And I completely acknowledge that what I'm about to say might be wrong as I haven't seen the author's code or the code of wlroots.

But I have to wonder if the design decisions of wlroots itself may be dubious. If it's this hard to manage memory safely when you're trying to wrap the API in a language that demands you're kept accountable for memory safety....


Does wlroots itself present good code quality or it is basically a horrible wrapper to support every Xorg use case?


wlroots is generally highly regarded in the Wayland community, both in terms of technical design and code quality. There's a reason that every Wayland project which started since wlroots has used wlroots, and those who didn't at first eventually rewrote their code to use wlroots.


I get the sense that most adopted wlroots because Wayland is a complex protocol. There's no alternative to wlroots so part of the reason everybody uses it is because there's no choice if you don't want to start from nothing.


Wayland is a simple protocol. The complicated part is everything else, like graphics and input and X11 compatibility.


Right, but that still means that getting to a point most end-users would consider 'usable' is quite complex. Especially because the documentation around these things can be lacking.

P.S. I like Wayland and use it everywhere for the record.


Is there any indication that mutter and kwin will rewrite to use wlroots? That would be glorious.


There's been some discussion regarding mutter, but kwin seems to be happy with their own implementation for now. Both projects pre-date wlroots and had already invested a lot in it when wlroots came on the scene.


I have had the view that safety is typically not the most important goal of a project (user satisfaction is). I have often said that safety can get in the way of writing software that is useful. It's great to see a real-life case with Rust from someone who obviously tried very hard but ultimately had safety get in the way of writing the software they wanted. Hopefully, Rust will evolve to interface with the "unsafe" world in a more ergonomic way.


good ol C :). Thanks for sharing these experiences. Happy to see some example of the places where rust still falls a little short / is a bit too restrictive for system programming. a lot of people deny this, but there's plenty of situations where you just want plain old C for it's straightforwardness to implement your own design. if you want to do it the 'rust' way, you find yourself restricted in all kinds of ways. if you can live with those restrictions it's great, but sometimes you just get stuck. Good luck on the rebuild in C!


A pity wlroots itself isn't written in Rust.


C is the lingra franca of programming. By writing wlroots in C, we make it easy for a half dozen projects to make bindings to other programming languages. Rust is the only one that seems to have failed, and it's not surprising given the constraints of the language.

wlroots brings together over a dozen different libraries and interfaces which are implemented in C. You'd have to repeat this process a dozen times over, running into the same problems which caused Timidger to abandon wlroots-rs, only more so. And for what? wlroots works great and didn't require shaving a thosuand yaks.


I know you know this, but for the benefit of others, you're talking about the C ABI, not about the language itself. Rust can expose a C ABI as well, and you'd get all those same bindings benefits.

> wlroots brings together over a dozen different libraries and interfaces which are implemented in C.

This is a great argument that it should be written in C, though.


> By writing wlroots in C, we make it easy for a half dozen projects to make bindings to other programming languages.

Rust is the same in this regard, you can bind it to any other language. And it doesn't have the horrible downsides of C. So why not use it?

> wlroots brings together over a dozen different libraries and interfaces which are implemented in C.

That's an issue as well, and this of course runs deep. But starting somewhere should be still possible.


>So why not use it?

Because I don't like it, and I do like C. Rust is not the second coming of Christ, it's a programming language and has many shortcomings and tradeoffs.


Which is fine, everyone has their own taste and every language has its shortcomings, but then why didn't you say that from the beginning? Instead you made up a stupid argument that with wlroots being written in Rust other projects would have had a much harder time to make bindings to other programming languages. Which is of course bullshit since Rust allows you to provide a C ABI, hence creating language bindings is as simple as with C.


>Which is of course bullshit since Rust allows you to provide a C ABI, hence creating language bindings is as simple as with C.

No, because C already uses the C ABI. You have to have a separate interface to your Rust code which uses the C ABI, and an uncomfortable transition point between C-land and Rust-land. This is definitely not as simple as it is with C.


That wasn't your argument. Your argument was that other projects have a much easier time to create language bindings for wlroots if it's written in C and know your argument changed to: I have a much easier time to provide a C ABI if wlroots is written in C instead of Rust.


In my view, tradeoffs of C are objectively much worse overall. I.e. it's not about taste, it's about the language limitations. Tastes are a different matter and they of course can differ.


C is the lingua franca of UNIX based platforms.

Thankfully not all OSes bow to it.


I think it has less to do with OS then you think. It has more to do with compilers/programming languages.

On Windows and Android if I'm interfacing to a Rust/Python/Java (on Windows)/C++ (on Android) library I still do it by going through C.


On Android you can do it via the NDK sure, but then again 90% of the OS is exposed via Java APIs, even a basic thing like opening a file requires Java, if you want to do it in a portable, certified Play Store way.

And on Windows, .NET MSIL or COM are much better ways to expose libraries.

Naturally one can still do it in an old fashion way.


>Thankfully not all OSes bow to it.

Right, just the successful ones.


That's hardly an argument for using C though. Successful OSes today were started years ago when there was barely anything else to use. It takes a long time for OS to gain success. Rust is so young, that to evaluate success of an OS written in it, you'd need to wait comparatively. So time will tell.


We could say the same related to JavaScript and Web, and it doesn't make it better.

Don't mistake success due to quality and being widepread thanks to historical accidents, free access to source code and the consequence market forces of POSIX adoption.


Not Windows.


Right, because Win32 is a shining beacon of good design. Thank god it's not C, right?


Meanwhwile Linux is still trying to catchup with the security level of mainframes.

68% of out of bound kernel expoloits in 2018, nice.


> When the benefit at the end of the day is just so I don’t have to write C, that doesn’t really make it worth it.

This line not only summarizes the content of the article, but I'm afraid it may also describe the actual situation of porting C/C++ code to Rust.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: