
Rust at OneSignal - gdeglin
https://onesignal.com/blog/rust-at-onesignal/
======
chamakits
First, I have to comment on that diagram:

That diagram is the most interesting representation of a diagram I have ever
seen. Do you have a tool to generate these for you, or was it hand crafted for
this post? If it's a tool, is it available anywhere? If this is a built out
tool, I would use this tool all the time.

On the article itself:

This is the most interesting type of posts about choosing Rust. Very balanced
and aware that Rust gives a lot of upsides, but compared to other alternatives
there are measurable downsides as well.

I think the biggest trade-off that they identified (and I've also gotten the
impression from writing Rust on my own time, not for work) is the relative
immaturity of the ecosystem, in as far as libraries available, and amount of
developers supporting some of these libraries. What makes me hopeful, is that
as a stable language (post 1.0) Rust is still relatively young. And you can
definitely see things improving, with the community growing, more libraries
propping up, and also gathering around specific libraries.

~~~
jwilm
> That diagram is the most interesting representation of a diagram I have ever
> seen.

Thank you! We worked really hard on it.

> Do you have a tool to generate these for you, or was it hand crafted for
> this post? If it's a tool, is it available anywhere?

It was hand-made for this blog post. The diagram itself was drawn in a vector
graphics editor and exported as SVG. Then, using web inspector tools to find
which visual box corresponded to some line of svg markup, classes/ids were
added to the items we cared about. Next came writing prose for each component.
Finally, a bit of JavaScript and a CSS animation makes it come alive.

> If this is a built out tool, I would use this tool all the time.

Me too!

~~~
ngrilly
> The diagram itself was drawn in a vector graphics editor and exported as
> SVG.

Which vector graphics editor did you use?

~~~
Lev1a
Not sure what they used but on Windows I would recommend Inkscape (perhaps an
obvious choice).

~~~
pjmlp
My obvious choice would be Visio or PowerPoint.

~~~
cpach
Powerpoint can’t export to SVG though, can it?

~~~
pjmlp
Not the version that I own, but then there is Visio or the other vector
formats that can be converted to SVG.

I seldom use SVG, because it is failed vector format. You still cannot
guarantee an image will work properly across all browsers (desktop, mobile
devices) or vector design tools.

PS or PDF have more chance of working across tools.

------
Manishearth
I'm always very happy to hear of companies using Rust.

I'm _especially_ happy to see that they're using clippy. It's not a core piece
of their infra like hyper, but it's still being used, which is nice.

How has it helped? Does it just keep the code clean or has it found mistakes
and stuff like that too?

Loved the diagram, btw.

~~~
jwilm
> How has it helped? Does it just keep the code clean or has it found mistakes
> and stuff like that too?

Clippy hasn't found any bugs in our code that I'm aware of, but it has been
invaluable even so.

Clippy is really good at preventing bit-rot. When refactoring, it typically
points out constructs that are no longer needed after other changes.

It's great for teaching how to write Rust effectively and succinctly. I've
certainly learned a lot from it. One example lint in this area is using `for
item in list` vs `for item in list.into_iter()`.

OnePush sets `#![cfg_attr(not(test), warn(result_unwrap_used))]` to make sure
no `unwrap()`s are used in production code; all errors must be handled!
`result_unwrap_used` has easily been the most valuable lint for us.

~~~
Manishearth
Nice to know, thanks!

Yeah, clippy isn't solely for finding bugs -- a lot of the lints are about
better style and cleaning up code, which often become useful after
refactorings.

~~~
ehsanu1
I'd really like to use clippy, but for production use I don't want to use
anything but the stable Rust channel, whereas clippy apparently needs nightly
Rust. Maybe I'm being overly cautious (OneSignal seems to do fine with just
version pinning), but there's a reason nightly is separate from stable.

It would be cool if I could configure cargo to only use nightly Rust when
invoking clippy, and use stable for all dev/test/prod builds. Do you happen to
know if that's possible?

~~~
Manishearth
Yeah, clippy is a bit of an outlier because it hooks into the compiler itself
to extract information during compilation. These are APIs that will never be
"stable", nor are they really part of the "stdlib"; it's just that they're
around and accessible in a nightly compiler.

Given that clippy is a tool, not a dependency, folks aren't overly concerned
about this -- it cannot infect dependency trees and only requires nightly for
local development. Most people do `rustup run nightly cargo install clippy`
followed by `cargo +nightly clippy` to run it locally without needing to
change the default toolchain.

There are plans to distribute clippy as part of the rust "extended"
distribution, so the stability issue will go away.

~~~
yazaddaruvala
:) There is one way to have clippy work in the stable compiler: The
integration of useful tools!

But that is clearly an awful idea. No one would like useful tools at the
expense of "separation of concerns" and marginally increased compiler
maintenance :P

~~~
Manishearth
Not sure what you're trying to say here (you seem to be using sarcasm but I'm
not sure), though I will note that clippy is wedded to a lot of compiler
internals, so being a part of the compiler build/CI is a good thing for
clippy. Often APIs get removed or changed from the compiler internals and we
have to work around or sometimes revert it. Usually it works out, but it would
be nice if the concern of having to support clippy's use case was part of the
decision of how an API is to be changed. In the opposite direction, rustc does
lints already and clippy is an extension of that. So I'm not really sure if
"separation of concerns" is an issue here -- they're really the same concerns.
While a lot of the clippy contributors are not well versed with compiler
internals, the clippy maintainers all basically have to be, and we often have
to dig deep into compiler changes to make stuff work with nightlies.

~~~
pjmlp
I don't see any reason why a set of stable APIs couldn't be defined, after all
other languages do have them for the same purpose as clippy, e.g. Go vet, .NET
Roslyn....

~~~
Manishearth
A set of stable APIs could be defined, but it would probably remove some of
the more powerful lints in clippy.

You have the same issue with clang -- libclang is stable, but a lot of the
plugins use the unstable clang plugin API.

------
staticassertion
This is an excellent blog post - thanks a lot for writing it up. Solid use
cases defined, I particularly enjoyed how you dealt with the shortcomings of
joining a relatively young language.

The 'belligerent refactoring' resonated with me - I have worked on untyped and
statically typed codebases of varying sizes and when making huge changes there
is no better tool for me personally than an expressive type system.

------
im_down_w_otp
Given the use case described for OneSignal and the enumerated benefits gained
from Rust contrasted against the difficulties... I'm curious why Erlang/Elixir
+ Dialyzer wasn't considered or used?

We're starting to use Rust for static binary quasi-safety-critical
applications, but communication/control-plane stuff like what's described at
OneSignal we've found is much easier w/ Erlang.

~~~
jwilm
We actually did consider them briefly. The problem in this case was having
zero in-house knowledge about the technologies nor anybody particularly
experienced with FP. We did not want to take on that much risk.

~~~
im_down_w_otp
You started with experienced Rust people?

Where'd you find them? I'd _love_ to find a vein of them to mine over time.
:-)

~~~
adrianN
Post in HN's "Who's hiring". Rust being constantly on the front page seems to
be a good indication that there are many people here who are interested in the
language.

~~~
thenewwazoo
Indeed - I write things in Rust on my own time and enjoy doing it so much that
I'd consider "come write Rust" to be a big, big positive differentiator
between jobs.

------
squiguy7
I think OneSignal will benefit immensely once Hyper lands async support in an
upcoming release. I know this is an immature side of Rust but there is a lot
of inertia behind async libraries right now.

~~~
jwilm
We're definitely looking forward to the tokio-based version of Hyper async. In
case it wasn't clear from the post, we are actually using the unreleased async
version of Hyper (powered by mio/rotor) today.

~~~
steveklabnik
You could switch to the tokio branch of hyper, my understanding is that it's
pretty much stable, just waiting on the final tweaks before the impending
release of tokio 0.1. We're so close...

~~~
jwilm
I'm still a bit hesitant to pick up tokio and tokio-hyper so soon. We've got a
lot of production miles on the current async client - on the order of billions
of HTTP requests. Qualifying the new hyper code for production use will take
some time. There's also still some discussion on the tokio issue tracker about
problems with panicking; we'll probably wait until that category of issue is
resolved before diving in.

~~~
steveklabnik
That is super, super fair. :) I'm just incredibly excited for this release,
almost as much as I am for Rust 1.15 generally. It's a good year so far...

------
filleokus
Nice read. Apart from the excellent writing I also think the interactive
diagram was really cool!

~~~
jwilm
Thank you for the compliments! I'm especially glad you like the diagram.

------
rebelidealist
How does OneSignal make money? Since the pricing is free, i'm not sure how it
will sustain itself after the vc chest dries.

~~~
haimez
Wet blanket! The rust will sustain them!

~~~
newsat13
What does the expression 'Wet blanket!' mean?

~~~
neandrake
It generally refers to something that ruins a happy/good situation -- like
throwing a cold wet blanket on someone out on a summer day.

~~~
newsat13
Thanks.

------
EugeneOZ
Every time I read "Rust tooling is weak" I'm sure in following lines I will
not find words about IDEA plugin (powerful and with regular updates) and Cargo
(with a lot of builtin features, such as testing). Rust tooling promotion is
weak, not tooling itself.

~~~
pjmlp
Still has a bit of catch up to do, compared with what I enjoy daily in Java
and .NET projects.

Right now my biggest pain, are the compilation times, specially seeing cargo
compiling multiple times the same crates instead of reusing the already binary
compiled libraries.

Unless there is some magic cargo incantation I am missing.

~~~
EugeneOZ
Cargo doesn't recompile all crates on each iteration, you are missing
something definitely. It only recompiles changed code. Or if rustc/cargo
version has been changed.

"a bit of catch up" is one thing, not mentioning most powerful IDE plugin is
another.

~~~
pjmlp
This is not what I see, I imagine you are talking inside one single project,
while I mean across all Rust applications I might have available.

1 - Get Project A

2 - Crate X gets compiled as one of the dependencies

3 - Get Project B

4 - Crate X gets compiled _again_ as one of the dependencies

I usually see this when compiling all the VSCode related projects during new
Rust releases.

Could the reason be that Crate X on Project A and Project B isn't the same
version?

Where are the binary rlib visible to all cargo projects stored? I could only
find text versions zipped together.

I haven't checked this deeply yet, just seeing "Compiling crate X" multiple
times isn't fun.

~~~
EugeneOZ
All projects are independent and binaries are stored in "target" folder. It's
a good thing. Even libs nominally same versions can have different source code
and dependencies (because of "replace" or vendoring).

~~~
pjmlp
No, it is not a good thing. It is a waste of my development time that I could
use for productive work.

On other AOT compiled languages that I use at work, I can reuse binaries
across projects, including the languages Rust intends to replace.

It is a hard sell, if using Rust means having to spend more time compiling
than even with C++ (specially now that modules are finally coming, at least
for us with VC++).

~~~
EugeneOZ
For me is a good thing, because, as I explained, version number is not a
guarantee.

~~~
pjmlp
So, than cargo needs to be improved so that there is a set of dependency
values (name, version, rustc version, etc) that can guarantee the uniqueness
of a binary library.

To me it seems FOSS undervalues the value of binary libraries in the
enterprise, or how they get used in many corporations or companies selling
components.

Rust improved safety is a good thing™, but not at the expense of continuously
seeing the same libraries being compiled over and over, vs reusing binary
artifacts across projects.

~~~
EugeneOZ
Cargo needs to be improved, here I agree :)

------
nemo1618
>Languages like Go make it too easy to ignore errors.

What? I've almost exclusively heard the opposite: Go makes error handling
_too_ in-your-face. People don't like writing "if err != nil" constantly.

>Finally, the developer leading the effort had experience and a strong
preference for Rust. There are plenty of technologies that would have met our
requirements, but deferring to a personal preference made a lot of sense.
Having engineers excited about what they are working on is incredibly
valuable.

I wish it were more acceptable to admit this instead of having to half-assedly
argue that your favorite language just so happens to be a perfect fit for the
problem. There's no shame in simply using the language you're most proficient
in. Practically speaking, language choice is only a problem for large teams or
for projects with language-specific requirements.

~~~
jwilm
> What? I've almost exclusively heard the opposite: Go makes error handling
> too in-your-face. People don't like writing "if err != nil" constantly.

Do you actually have to look at the value you err, or can you just pretend
like it's not there? (genuinely asking, I thought it was the latter).

> I wish it were more acceptable to admit this instead of having to half-
> assedly argue that your favorite language just so happens to be a perfect
> fit for the problem.

This is one of the reasons we felt it important to mention. Rather than demand
purely technical motivations, we choose to acknowledge the human aspects of
software engineering as well.

~~~
rcaught
> > > Languages like Go make it too easy to ignore errors.

> > What? I've almost exclusively heard the opposite: Go makes error handling
> too in-your-face. People don't like writing "if err != nil" constantly.

> Do you actually have to look at the value you err, or can you just pretend
> like it's not there? (genuinely asking, I thought it was the latter).

While Go may not have all the capabilities Rust has at forcing error
inspection, it really isn't in the class of languages that "make it too easy
to ignore errors". I agree that a lot of it is convention, but multi-value
returns, usage of named variables and good practices in the base libs (flowing
upwards) can't place it in the realm of easily ignoring errors.

------
michaelmior
> To get the same effect in a dynamic language, one would need to load this
> dictionary out of Redis and write a bunch of boilerplate to validate that
> the returned fields are correct.

How is this not what serde does internally? I could certainly imagine a nice
library for a dynamic language that has a roughly equivalent API.

~~~
jwilm
It's essentially what serde is doing internally. As an author, you don't have
to do any of the work, though. I struggle to imagine how a dynamic language
could provide equivalent functionality without the call side becoming more
complex.

~~~
michaelmior
See the sibling comment for an example. I really don't see the challenge here.

------
echelon
This is an amazing postmortem--thanks for posting it! I'm really happy to hear
about your success with Rust!

------
wchrisn
I'm always excite to hear about companies who are using rust in production.
OneSignal team has done an excellent job to bring out the strengths and
weaknesses. Would like to know more about the Unit Testing practices that
OneSignal and others follow.

BTW - I have all the motivation now to plunge into the rust-world

------
p0nce
> This wasn't something we anticipated up front, but build times have become
> onerous for us. A build from scratch now falls into the category of "go make
> coffee and play some ping-pong." Recompiling a couple of changes isn't
> exactly quick either.

Rust should really address the top problem with C++ which is slow build times.

~~~
humanrebar
Slow build times are a problem for C++, but there are bigger ones yet. Like
the reliance on textual inclusion and the preprocessor instead of an actual
module system. It has massive second and third order effects on tooling, code
sharing, and community building.

Incidentally, a module system will help with build times, but that's only one
of the benefits.

------
the_duke
Since someone from the company seems to post here:

I'm curios: this looks like a perfect use case for Apache Kafka.

Did you evaluate it?

------
sandGorgon
why are you using a language specific connection pooler (r2d2).

why not something like pgbouncer?

this is similar to the java approach of building connection pooling into the
application itself using jdbc.

~~~
jwilm
We use PgBouncer as well. It's still useful to have a connection pooler within
the application for ensuring connections are still good, reconnecting as
needed, and making it easy to check out a connection.

~~~
sandGorgon
that has not worked out well for me in the java world - one connection pooler
connecting to another one. unless one of them has ultimate control on timeouts
and eviction.

I'm actually very intrigued on if you are seeing actual benefits using r2d2
... or would using only pgbouncer make it just as efficient.

------
EugeneOZ
Overall impression after reading this article: using Rust is painful and
risky, and author is thinking often about Go.

There is huge advantage, not mentioned in article: when code has a lot of work
with multithreaded tasks, race-conditions bugs will bite you sooner or later,
and they will bite very very painful. They are hard to detect, very hard to
reproduce and it takes long time to fix them. Rust is immune to race-
conditions, and it's really awesome and enough reason to use it in apps like
this.

~~~
ZoFreX
> Rust is immune to race-conditions

Safe Rust guarantees no _data races_, but not race conditions in general.[1]

[1] [https://doc.rust-lang.org/nomicon/races.html](https://doc.rust-
lang.org/nomicon/races.html)

~~~
EugeneOZ
thanks for correction.

