
The Sandstorm Team Is Joining Cloudflare - jgrahamc
https://sandstorm.io/news/2017-03-13-joining-cloudflare
======
kentonv
Hi all, I'm not sure how responsive I'll be able to be on this thread since
I'm in orientation today. :)

But, here's the things I expect to be repeating a lot:

\- Sandstorm is still an independent company, under control of Jade and
myself.

\- Sandstorm is "my baby" and I'm not going to stop working on it just because
I have a day job. Yes, it will slow down a bit -- but on the bright side, we
were previously spending the vast majority of our time trying to figure out
enterprise sales (rather than actually building cool features), and we don't
have to do that anymore. So, the difference may not be as big as you'd think.
See: [https://sandstorm.io/news/2017-02-06-sandstorm-returning-
to-...](https://sandstorm.io/news/2017-02-06-sandstorm-returning-to-community-
roots)

\- Nevertheless I'm pretty excited about the work I'll be doing at Cloudflare
as well -- which includes Cap'n Proto and other fascinating infrastructure
tech.

~~~
cstrahan
I'm pretty stoked about future cap'n proto developments. In the scenarios
where I've brought up Cap'n Proto, all but the most senior/prolific developers
have a near impossible time of ascertaining _why_ one might want to use capnp
over competing approaches (e.g. Protobuf). I think, despite a pretty decent
description on the website and my own skills in explaining/teaching, such
protocols are nearly impossible for the majority of developers to intuit -- so
they're all perceived as being nearly identical, save for financial backing
and other forms of social proof.

With some of the rough corners filed smooth (e.g. Windows support), a bit more
marketing and promotion, and eventually more big names advertising their use
of capnp, my job getting buy in from my peers would be _much_ easier.

~~~
hueving
Can you provide a 'top 3' list of reasons to use Cap'n Proto over Protobufs?

~~~
doh
1) Cap'n Proto doesn't encode/decode messages thus it's nuch cheaper for
processing and memory management

2) protobuf in the proto3 design doesn't cary default values. So if you have a
bool field and want to explicitly send false, well you have to change it to
some other type or use the default values all the time

3) protobuf generates incredibly large serialization/deserialization support
coce for each template. For some languages like Python in can be in hundreds
of kilobytes. Cap'n proto messages are significantly smaller

There is more for CnP but Protobuf has much better support and is by default
used in projects like gRPC. Also new CnP is lacking speed in new development
in comparison to Protobuf.

But I'm using in one of my side projects and I'm very happy with it

~~~
euyyn
(2) is explicitly one of the reasons to use proto3 over proto2, though :) It's
weird seeing it listed as a disadvantage instead of the other way around.

~~~
doh
Depends on the use case. For instance if you're mapping the data to database,
you can't send null variables. Or you store the defaults (waste of space) or
you send it as another type.

In our case we've now over 1T of rows. Storing default variables would be
incredibly inefficient and expensive.

~~~
euyyn
I'm not understanding your case much. Proto3 does not serialize zeros/false,
precisely because it'd be a waste without a benefit. After deserialization,
the value is as equal to zero as if the zero had been written on the wire or
on disk. Why would you want to store that?

Is nullability of primitive types what you're missing? As in, having a boolean
value that can be true, false, or null. If you need that, defining your field
as google.protobuf.BoolValue instead of bool gives you that (same with the
other primitive types:
[https://github.com/google/protobuf/blob/master/src/google/pr...](https://github.com/google/protobuf/blob/master/src/google/protobuf/wrappers.proto)
). But making primitive types never null aligns proto3 with most programming
languages, making the generated code more idiomatic and performant.

------
devnull42
>Although Cloudflare recently suffered from a widely-reported security
incident, their response was impressively fast and transparent.

Really....seemed like they massively downplayed to me.

~~~
kentonv
FWIW I blogged more about it a couple weeks ago (editorial at the bottom):
[https://sandstorm.io/news/2017-02-28-cloudbleed](https://sandstorm.io/news/2017-02-28-cloudbleed)

~~~
ryanlol
Your blog post doesn't seem to discuss their response very much.

What leads you to believe that CloudFlare was impressively fast and
transparent in this case? Especially since statements from Project Zero seem
to imply that they were anything but.

~~~
kentonv
CF disabled the problematic feature within hours, on a Friday evening. After
that, figuring out what private data was stuck in search engine caches was
obviously going to take some time. It seems clear enough that they were
working as fast as they could. Tavis is awesome but I think he was being
unfairly hard on them in the project zero thread.

(Note: All of the above is based on my _external_ observations, as I was not
yet an employee nor did I have any internal access at the time.)

------
tannhaeuser
Sad to here sandstorm.io didn't make it into a viable business for you. I can
only imagine just how much energy went into it (and hopefully continues to
go).

When time allows, I'd very much appreciate a reflection on the economic
viability for innovative infrastructure software in general (hosted vs
enterprise sales, also taking RethinkDB's demise and that of other "new stack"
startups into account, etc.), and I'm sure others here will as well.

~~~
kentonv
I still think we had a reasonable business model. The problem is, it was
enterprise-targeted, and none of us actually knew how to sell to (or even
_talk_ to) enterprise customers. It turns out this is a skill that is not easy
to learn. We really should have hired for it, rather than trying to figure it
out from first principles. But by the time we realized that, it was too late
-- enterprise sales cycles are long so we needed a lot of lead time.

With more investment I think we could have made it work, but it turns out that
once you run into any kind of "trouble" as a startup, getting further
investment becomes incredibly hard -- and we were pretty bad at the
fundraising game to start with (it is, essentially, sales, after all).

I don't think we have any particular evidence that Sandstorm wasn't viable as
a technology or a business -- we just personally did not have the right
skillset for the business model we chose. :/

~~~
nickpsecurity
"The problem is, it was enterprise-targeted, and none of us actually knew how
to sell to (or even talk to) enterprise customers. It turns out this is a
skill that is not easy to learn. We really should have hired for it, rather
than trying to figure it out from first principles."

That's how I predicted it would fail, too. You were at least able to figure
out the problem. Many people will blame everything or everyone else.
Enterprise sales is it's own beast that's much different from how things
operate in the Valley or smaller shops. That's only half the problem. The
other half is you usually have to integrate with systems designed not to
integrate with 3rd parties easily. Lock-in loving systems. The healthcare
startups are particularly being hit hard by this.

I keep thinking there's potential to get a business going that specializes in
the sales and/or enterprise features at reasonable price. A public benefit or
nonprofit company that forces the charges to be limited to something
reasonable. They can help all these new companies bootstrap the enterprise-
specific aspects of their goods for a price. What you think?

------
wyldfire
Nice to see a soft landing for this worthwhile project.

EDIT: disregard the above, that this was a careless misunderstanding of the
title. TFA says "Sandstorm, for its part, remains an independent entity, and I
won’t be working on it during my day job at Cloudflare. However, I will be
working on Cap’n Proto."

------
diggan
The people behind Sandstorm are joining Cloudflare and will now have less time
to spend on Sandstorm. That's sad, because I was excited in general behind the
idea of Sandstorm, and I hope it continues to be worked on, even if it's just
during the weekend.

~~~
yebyen
The really big announcement came about six weeks ago, when Kenton announced
that Sandstorm for Work was open sourced and not anymore a product that is for
sale.

If you didn't follow that link in the first paragraph and didn't hear about it
already from some other source,
[https://sandstorm.io/news/2017-02-06-sandstorm-returning-
to-...](https://sandstorm.io/news/2017-02-06-sandstorm-returning-to-community-
roots)

IMHO this is all really good news, because people who were on the fence about
Sandstorm before can feel more free to commit resources to developing it,
without worrying that Sandstorm the company is going to come along and make
decisions against their interest, or try to make their work proprietary, or
leverage it for the company to make more profit at their expense. (Not that we
thought those things would happen...)

They would certainly have less time to work on Sandstorm if they were not able
to go right out and get full-time jobs, once it was determined Sandstorm the
company was out of money! IMHO there is nothing but good news in these
announcements.

------
the_common_man
Best of luck, kenton!

If anyone is looking for alternatives:
[http://alternativeto.net/software/sandstorm-
io/](http://alternativeto.net/software/sandstorm-io/) . I think Cozy,
Yunohost, Cloudron, arkOS are the closest alternatives (Bitnami is about
1-click images and DO/linode are about cloud servers). Would be great to hear
about other alternatives...

~~~
BuuQu9hu
Unless I missed something, none of these are capability-safe designs, whereas
Sandstorm is capability-safe on the wire up to the edge of Capn Proto.

~~~
the_common_man
From the alternatives link "Sandstorm makes it easy to run your own server"
and from their website "Sandstorm is a self-hostable web productivity suite".
With that in mind, the alternatives seem reasonable.

------
bogomipz
Can someone say how Cloudflare uses the Cap'n Proto rpc framework? I would be
interested in hearing how this fits into a CDN architecture. Cheers.

~~~
kentonv
At present they (uh, we, I suppose!) use the serialization only, for a variety
of uses including (publicly disclosed) the log analysis system. I have some
ideas where the RPC would be useful, though.

~~~
bogomipz
Thanks for the replies and the link(interesting), cheers.

------
kev009
As someone in the CDN industry, that was a smart aquihire.

------
jamesinsf
This is one of the better CloudFlare acquihires. Congrats!

------
Pica_soO
In other news, silicon valley naming conspiracy discovered who's whole intend
it is to make the stock-market report sound like a weather-forecast.

------
nilved
Oh boy. I guess that's it for Sandstorm.

~~~
kentonv
No. Sandstorm is "my baby" and I'm not going to stop working on it just
because I have a day job.

~~~
jameskegel
I hope as your work life evolves this statement holds true. Congrats and good
luck, and thanks for the code.

------
hnbroseph
cool, but what does sandstorm have to do with cloudflare? then again i don't
quite get what it is... some kind of file/app sharing thing.

~~~
yebyen
Sandstorm takes apps that normally would need someone to run a server, breaks
them down into the smallest shareable unit (a "grain"), and lets you host them
all centrally, on the Sandstorm server. Whether that's the Oasis service, your
own machines, or something in between.

So instead of running a whole Gitlab server, you'd run a Sandstorm server that
hosts Gitlab grains, each consisting of a single repository. Gitlab itself is
no longer responsible for sharing and authentication, the Gitlab grain lets
Sandstorm handle that. Apps need to be rewired to fit into this paradigm, but
once they do, it feels like one coherent platform rather than a bunch of
separate individually hosted apps that maybe you glue together somehow.

For Etherpad, instead of running an Etherpad server that hosts many notepads,
you create an Etherpad grain that has just one shared pad. For a drawing app,
the grain would be just one drawing. For a photo gallery app, maybe it would
be just one gallery. (These are all real examples, these grains all exist and
work as I'm describing.)

~~~
derefr
> Whether that's the Oasis service, your own machines, or something in
> between.

To clarify: does a "grain" just run in one place, or is each "grain" a cluster
of containers that can be spread-scheduled between disparate regions (i.e.
"the Oasis service" and "your own machines" at the same time)? And, if the
latter, do the cluster nodes have awareness of their locations, and can take
advantage of that?

~~~
yebyen
I have honestly no idea, other than I know that the grains are not permanent
resident processes. IOW when they are not in use, they spin down and do not
actively consume resources other than disk.

I think the grain runs in one place, I think to use the container analog it
would be one container, but as a point of clarification I believe that it is
not using one of the container runtimes like docker or rkt that you will be
familiar with, but uses its own sandboxing instead.

I have not tried running the newly opened up Oasis backend so I can't really
speak toward how it works. My experience with a single server running
Sandstorm has been good, painless automatic upgrades, I would expect from what
I know about this team, that since it's their dogfood they made it taste good.

