
Sandstorm – An open source operating system for personal and private clouds - dack
https://sandstorm.io/
======
anilgulecha
For massive take-off, Sandstorm is _almost-there_ with making everyone's
online world de-centralized.

It's solved half of the problem beautifully -- it's very easy, once setup, to
launch any new app on your server.

The other half of the problem is moving of server/data. Right now, either you
use their self hosted instance of sandstorm, or setup your own server. There's
the hassle of keeping track of data/server. There's pseudo-centralization
happening here, given the difficulty of changing compute or storage by a
layperson.

However, if sandstorm had built in federated storage (either based on
bittorrent filesystem or any other p2p storage), then I no longer worry about
losing my data. I simply move my domain to a new "server-provider" or use the
main project's own service, and then at a later date can spin up my own
compute, or use another provider, and point them to my data on the
p2p/federated storage.

~~~
kentonv
FWIW, it's straightforward to see how this could be accomplished with
Sandstorm's model, since user data is very clearly divided into grains (app
instances / documents) with clear owners, and these grains can easily be
transferred between servers. Today, you can click "download backup" to get a
zip, then upload it to some other server, and end up with exactly the same
grain reproduced there. We plan to streamline this further, to allow mass
transfers and synchronizing grains between servers. It's a major goal for us
that it should be trivial to move between hosts, so that you're never stuck
with a host that you don't like.

~~~
anilgulecha
Good to know. Just to re-iterate one point, the ideal is that the server does
not have a hold over the data (by say withholding download of backup data), or
that the data is held tied to a certain server.

Server and data (encrypted) should just be commodities, and the domain DNS is
what should describe ownership. That's why I would like the focus to be on
federation than easy import/export.

~~~
kentonv
Realistically, if you want to run code on the server, the server needs some
control over the data. There are some distributed systems lately that really
only distribute _storage_ while all computation happens client-side. That
doesn't work well if you want real-time collaboration (the latency is too
high) or multi-user permission constraints. Not to mention, it requires
rewriting the world. So, I'm skeptical of those approaches. I'd rather make it
easy to get your data out when you need/want to.

But it's certainly an opinion that is open to debate.

~~~
anilgulecha
Copying from a sister comment on this thread:

> To ensure low latency, I'm OK with the protocol allowing a server to cache
> and render recent data, while a background service syncs to the de-
> centralized store.

This would be a help for the UI from the protocol, but the ideal IMO should be
something that normal SaaS companies/products hold you on for -- they have
your data hostage.

I understand that for sandstorm above many not be possible as you'd have to
re-write a lot, and this feature does not hold sufficient priority for the
relevant code re-write to happen.

That's too bad though -- I was expecting this need of mine to be provided by
the project. Still I wish you the best of luck as you're head and shoulders
the better option above any of the current SaaS apps for a personal clouds.

------
StevePerkins
This is an "operating system" only in a marketing hype sense.

In technical reality, it seems to be a Google Apps alternative that you can
self-host.

Still very cool (frankly even MORE cool than yet another hobby OS!)... but
"platform" would be a less groan-worthy phrase than "operating system".

~~~
kentonv
Actually, I agree. We originally used the term "platform". But, we found that
people understood more easily when we switched to "operating system" \-- and
it's not too far off the mark, honestly. Sandstorm is an OS in the sense that
Android is an OS. Both use the Linux kernel, but largely control the
userspace. (Sandstorm can run along-side your existing Linux distro without
interfering with anything, but it doesn't really use any of your distro's
infrastructure so this is somewhat of a technicality.)

The problem with "platform", which surprised me, is that a lot of people now
think "platform" is somehow synonymous with "cloud service". For example, if
you watch HBO's Silicon Valley, you may remember the whole "platform vs. box"
debate from this season. It's a frustrating perversion of the word, but it is
what it is.

~~~
seagreen
It's actually restricting our thought to an annoying extent that we've
overloaded the term "operating system" so much. This isn't Sandstorm's fault
-- Windows already has billions of people thinking "operating system" ==
"desktop environment" so the ship has already sailed.

I would really like a word that describes this set:

    
    
      + Unix CLI
      + WIMP desktops
      + Browsers
      + iOS touch interface
      + Android touch interface
      + Emacs (when used for general computing)
    

Perhaps "user environment" would work?

EDIT: For me at least "platform" doesn't quite fit. I think whatever word is
best should imply "single user interface" which platform doesn't necessarily.

~~~
mcbits
How about "clam" or "mussel" (as a variant of "shell" that has some meat to
it)?

~~~
seagreen
Hilarious, but not nearly boring enough to use in actual business writing:)

------
goodplay
I'm excited. Today was the first time I tried Sandstorm via their demo
serivce, and I must say that I am genuinely surprised and impressed with the
level of polish. One click gave me a private gitlab, a collaborative document
editor, and even a running instance of browserquest!

This level of polish might just be what it takes to get the masses to host
their own digital services. I'm strongly considering setting a sandstorm
server up for my family.

One question promptly and prominently pops to mind: Given the focus on non-
technical users, How does sandstorm mitigate issues that are caused by NAT and
NAT traversal?

~~~
kentonv
> Given the focus on non-technical users, How does sandstorm mitigate issues
> that are caused by NAT and NAT traversal?

At present... not very well, to be honest. If you use sandcats.io (our dynamic
DNS), you do have to open port 443, which means you need to understand router
config. It would be cool to automate that with UPNP but we haven't gotten
there yet. The prevalence of "hairpin routing" bugs in home routers is also a
downer. :/

One idea I'd like to pursue someday is the one taken by IndieBox (no longer
Googlable -- the one by Johannes Ernst): have a device which sits between your
modem and your internal network, so that it can intercept DNS and make the
server available at [http://sandstorm](http://sandstorm) or whatever. Then do
UPNP to make it externally accessible (if desired).

~~~
goodplay
I wasn't aware of Sandcats.io. Thank you for mentioning it.

While it doesn't address all issues, it does go a long way towards mitigating
them. I must also take this opportunity to commend you guys on how transport
security is handled; the fact that traffic remains secured end-to-end using a
host certificate rather than terminating security at the proxy puts sandstorm
well ahead of anything else in this space.

Secure, easy, and open source. I'm sold as both a developer and an end-user.

------
jegoodwin3
I saw this last night and signed up for oasis on my chromebook. I created a
Jupyter notebook and entered an expression in it. Nice. Then I closed the lid
to my chromebook for the evening.

Next morning I opened up the lid, and it helpfully told me it wasn't
connected. I waited a bit longer and updated the expression. The cell text
updated but after pressing 'play' the result value did not update, leaving my
screen in an inconsistent, incorrect state (wrong value for displayed input to
function). There was a little sign telling me I wasn't connected.

I had signed in originally with gmail. People are going to expect a similar
interaction model to google docs. There was no obvious way to get back online,
short of killing the tab... and getting back how? Remember a link? A bookmark?

This doesn't work yet -- not for most people.

Beautiful, beautiful job. Get the rest of the way there please...

~~~
kentonv
Thanks for the feedback.

The connection problem sounds like it may be a bug in Jupyter, or the
Sandstorm Jupyter package, rather than Sandstorm itself. It would be great if
you could file a bug on the package:
[https://github.com/jparyani/ipython/issues](https://github.com/jparyani/ipython/issues)

Usually these kinds of issues are solved by a refresh, though that's obviously
not great UX.

> and getting back how? Remember a link? A bookmark?

Go back to: [https://oasis.sandstorm.io](https://oasis.sandstorm.io)

All your stuff is listed when you log in.

~~~
jegoodwin3
Almost certainly -- and therein lies a problem for you. There is an ambiguity
between who owns the tab -- you or Jupyter app? The user of course will think
of you and the app as the same thing. If you let Jupyter throw up a message
saying 'you're disconnected' the user will infer that they are disconnected
from _you_.

It's not clear the user is wrong btw -- why does a mis-behaving, buggy app,
throw me into offline mode? Is it just Jupyter that is offline, or the whole
of sandstorm? The message says the tab is. And I can still see components of
sandstorm. Of course, I bet if switch to another app it will heal itself -- or
will it? Most users won't think of that (or a refresh, or a CTRL+F5, or
looking at the debug console for a hint etc.)

So my challenge to you is to explain how the offline/online model works, in a
way the user can understand. I suppose big organizations like Google just
solve this problem by not hosting buggy apps in the first place. :) :)

It is probably worth your while, from an adoption standpoint, to 'take
ownership' of a few key apps and treat their bugs as your bugs -- I would be
showing this to coworkers tomorrow if this had worked better, and it probably
isn't your fault. They are keen on Jupyter and this is the best delivery
platform I've seen for that app. Sharing jupyter notebooks in an enterprise
setting could be your killer app -- they contain private data, and a public
cloud won't work in many use cases, but most enterprises won't have enough of
a constituency of scientists and engineers and statisticians to want to host a
notebook server with 'front line' support. We're in the carpet cluster days
here...

~~~
kentonv
> 'take ownership' of a few key apps and treat their bugs as your bugs

Actually, we plan to do something like that! Though, Jupyter isn't part of the
initial set -- our most popular apps are Wekan, Etherpad, Rocket.Chat, and
Davros, so we'll be focusing on those.

~~~
jegoodwin3
That's good to hear and I see why those would be popular. I'm a bit surprised
that a kanban board is so important.

I'll put in my vote for replacing Wekan with Jupyter -- it's more strategic to
win mindshare outside the closed circle of developers. The average
unsophisticated user who just wants apps, your target market, doesn't want
their very own kanban board. A photo album yes. I speculate your early
adopters are predominantly developers themsleves and that's why it's popular.
It does fit my comment about private data being an important use case for you,
of course.

Sharing homework assignments or spreadsheets of data would be more strategic
for you, imho. It could help you break into the 'educational shared docs'
submarket -- every school I know uses google docs right now for convenience.
And don't forget homeschoolers, and teachers setting up courseware sites. I
would think the educational market would be _key_ for you to gain mindshare
and grow beyond the community of early-adopter developers.

Anyway, best of luck. This is some really good work you are doing.

------
studentrob
Love seeing this on top of HN, in light of recent discussion about securing
your own email. I've tried Sandstorm a couple of times and loved it. They're
always improving.

Kenton himself seems ready to respond in depth both whenever Sandstorm is
discussed on HN, and at times when I email Sandstorm for support.

I'm sure he's got his limits. He is setting a high bar for customer support
within the company. I think that's smart.

Innovation does not happen in a vacuum. Talking to customers with an open mind
is a great way to learn what they want and spur some creative process within
yourself.

~~~
kentonv
:D

We believe in releasing early and often and using feedback to direct
development. (Truth be told, this is why the "powerbox" features that we like
to talk about a lot aren't totally done yet: user feedback identified many
other, more pressing things to improve!)

That said, my dirty secret is that I have a little app that queries HN search
every five minutes and gives me a desktop notification whenever Sandstorm is
mentioned. :)

~~~
studentrob
> my dirty secret is that I have a little app that queries HN search every
> five minutes and gives me a desktop notification whenever Sandstorm is
> mentioned

I guessed that. You were too good at it. Good job making the name unique haha.
Just hope there is no anti-real world sandstorm tech that's developed in the
future ;-)

I don't think it's dirty. That's a service itself that other companies would
do well to follow by implementing in-house or through investment.

I think many companies do track mentions. Few follow through with real
engagement. There is a difference between observing and engaging. In these
fast times, I imagine it makes a difference.

Perhaps some users would find this intrusive, and I guess it could be done
poorly. I think you set a good example (so far.. ;-) )

~~~
eitland
Google alerts and/or some kind of analytics service can also (to a certain
degree) give you an opportunity to answer discussions while they are still
active.

------
dack
I just thought this was relevant in light of the recent discussion about how
to protect yourself in the event your google account is shut down. Sandstorm
has an email client as well - and you can 100% control your data.

~~~
xiaomai
I've been hoping someone packages nylas for it. I've been meaning to look into
how to do this.

~~~
kentonv
We have been looking into it too... ;)

------
IanCal
So I've been thinking a bit more about sandstorm recently as I'm planning to
build something and need to work out how I want to host/distribute it.

It'll process data for people and I'd absolutely love to be able to write the
app "ignoring" multiple uploads, user management, team-like sharing, etc.

I'd be expecting this to potentially be a paid app though. In app billing is
mentioned as a "coming soon", is there a roadmap or expected timescale for
this? Is it a "we kinda plan this sometime" or "currently in development,
releasing soon" kind of phase?

If not, I don't know how well it'd fit on their shared hosting, as the
processing might be fairly intensive. Is it reasonable for me to install
sandstorm on a server with my app and let people use that? Sort of like the
shared hosting but with only my app available? Does that make sense or am I
completely misusing the intended overall structuring of sandstorm?

~~~
kentonv
We decide what to do next based on demand, so having someone like you asking
us for in-app billing would make it likely to happen sooner. Feel free to
email us and tell us about your plans!

With that said, I would expect in-app billing to be at least 3-6 months out,
possibly longer if we see more interest for other features first.

However, note that you don't necessarily need in-app billing to sell a
Sandstorm app. You could always transact the sale yourself, and then ship an
.spk file (Sandstorm package) to the user.

Anyway, please email me (kenton at sandstorm.io) if you'd like to discuss
this!

------
jalami
Sandstorm is simple an intuitive which is what a lot of our decentralized
projects really need a good helping of. I watched the recent Decentralized Web
Summit, and there was so much talk about decentralization, yet so little talk
about self-hosting, federation or the projects that are out there doing the
dirty work today. It made me sad. People are looking for alternatives to
silos, not every one of course, but a lot of people just get overwhelmed with
the thought of self-hosting. Sandstorm is helping with that it seems. I wrote
more about this here[0] if you're curious.

Keep up the good work!

[0] [https://www.alami.io/post/decentralized-web-
summit-2016/](https://www.alami.io/post/decentralized-web-summit-2016/)

------
amelius
Does this OS provide building blocks for collaborative applications? Or does
every app have to reinvent collaborative editing?

Update: Okay, so I created a flowchart in one instance, then shared the link,
and opened it in another tab. Edited the flowchart in one tab, but it didn't
change in the other tab. This is imho not how a cloud OS should work.

But the container technology seems quite cool.

~~~
kentonv
Almost all Sandstorm apps support real-time collaboration. It sounds like you
tested draw.io, which unfortunately is one of the few that currently does not.

Sandstorm does not dictate what tech stack you use -- it supports any server
stack that runs on Linux. Supporting real-time collaboration is normally the
job of your tech stack. If you use Meteor, for example, you'll pretty much
have it by default.

Sandstorm does make real-time collaboration easier to operate at scale through
fine-grained containerization: each "document" runs in a separate container as
a separate instance of the app. Sandstorm ensures that there is never more
than one instance running for a particular document, so you don't have to
worry about concurrent writes coming from another machine. Scalability is
achieved by distributing the documents across machines.

[https://sandstorm.io/how-it-works](https://sandstorm.io/how-it-works)

~~~
yeowMeng
> Scalability is achieved by distributing the documents across machines.

Unless everyone wants to edit the same doc at the same time.

~~~
kentonv
Sure, but if you have hundreds of people trying to edit _the same document_ ,
you are going to have a bad time no matter what architecture you use. This is
why Google Docs reverts to read-only mode when too many people show up. We
could do something like that too: give the app an API to provide some static
content as a fall-back when a grain is overloaded.

------
educar
All the random domain sha1 like domain names are confusing. I also dislike
that many of the apps have missing features compared to the actual apps and
they get embedded in iframes. I understand all these are done for security.

Related: [https://cloudron.io](https://cloudron.io). Unlike sandstorm, it's
docker based and has a very good app store -
[https://cloudron.io/appstore.html](https://cloudron.io/appstore.html).

~~~
Veratyr
Unlike Sandstorm though, Cloudron hosting is limited to managed by them or
Amazon's cloud.

Being able to control your self hosted app environment is a big part of why
you'd do it.

~~~
educar
Yeah, I think that is correct. BTW, I just searched and agolia now and found
[https://urbit.org/](https://urbit.org/)

~~~
ORioN63
Urbit is a whole different beast, but shares the same goal:

Decentralization.

------
bigbugbag
Still looks a lot like yunohost[1] and cozy cloud[2] which for some reasons I
find more attractive than sandstorm. Then we have a rather new contender:
ArkOS

[1]: [https://yunohost.org/](https://yunohost.org/) [2]:
[https://cozy.io/](https://cozy.io/) [3]:
[https://arkos.io/](https://arkos.io/)

~~~
e12e
I'm not entirely clear if you're saying you prefer both yunhost and cozy cloud
to sandstorm (or just cozy) - but at least for cozy.io it doesn't really seem
like an alternative to sandstorm, as per the faq: "Currently Cozy supports
only Node.js applications.", while as can be seen from:
[https://docs.sandstorm.io/en/latest/vagrant-spk/packaging-
tu...](https://docs.sandstorm.io/en/latest/vagrant-spk/packaging-tutorial/)

"The tutorial uses a PHP app as an example. Sandstorm supports any programming
language that runs on Linux, not just PHP, such as Meteor, Python, Rails,
Node, PHP, C++, Go, Rust, and more. Read about vagrant-spk's platform stacks
to see how to optimize your package for your app's programming language."

As for yunohost it looks like it is more comparable - but essentially is a
"cpanel on steroids" \- a nice way to automate managing classic server
applications:

[https://yunohost.org/#/whatsyunohost](https://yunohost.org/#/whatsyunohost)

Thank you for mentioning these, I wasn't aware of either -- but I don't think
they really are an alternative to most of the use-cases sandstorm excel at.

Arkos seems similar to yunohost, although rather than an easy way to manage
Debian, it's an easy way to manage Arch on arm (a curios limitation IMHO):

[https://arkos.io/what-is-arkos/](https://arkos.io/what-is-arkos/)

~~~
bigbugbag
Let's put it this way, both are part of a larger effort to put the web back in
tracks by having more people self-host their data and web services. Then
sandstorm uses a third party google service to track visitors while cozy and
yunohost use a self-hosted instance of piwik. One of those is not putting his
money where his mouth is...

Sandstorm is made from an ex-google employee while cozy has the founder of the
european branch of mozilla in its ranks.

Then there is the AMD64 limitation of sandstorm, while the other offer support
for ARM, the de facto architecture for a low consumption personal server.

I think the sandstorm.io website is dense and hard to comprehend, it's not to
the point and from the get go I can tell it's pretending to be something it is
not (an OS).

Also the sandstorm demo feels clunky to me and I experience random crashes and
error messages while trying apps in sandstorm.

In the end my preference goes to yunohost because it's not from a funded
startup with a full team which may explain why it stayed simple and within
reach.

IMHO ArkOS is closer to freedombox than yunohost, also supporting ARM is _the_
feature not a limitation as it seems obvious that the personal self-hosted
server is running on cheap, low consumption embedded ARM. To me a curious
limitation is not supporting ARM.

~~~
e12e
I think I disagree with almost all your points here (or that they are a
reasonable criticism/list of things that should be changed with sandstorm) --
but I appreciate you taking the time to respond in such detail. (Modded your
comment up FWIW).

Didn't realize that Sandstorm (currently) targets only AMD64 - for others
interested, the rationale is here:

[https://github.com/sandstorm-
io/sandstorm/issues/94](https://github.com/sandstorm-io/sandstorm/issues/94)

I don't really see this as much of a problem - at a later point, if Sandstorm
is packaged for something like Debian, it's likely people will work on porting
it to Arm. As most of the software is already ported, this shouldn't be
insurmountable.

------
ams6110
OK so this _looks_ awesome and I had half a dozen ideas for things I'd like to
do with it in about 30 seconds. Is anyone using it? Would love to hear from
some experienced commentors.

~~~
ocdtrekkie
I've been using Sandstorm as my primary document writing and storage location
for a while. I've mostly been using the Oasis hosted service (and their alpha
server prior), but I've also installed Sandstorm on a couple of VMs for dev
and testing purposes. I'm pretty happy with it, and happy with the apps
available for it.

There's still some things I'm waiting for, such is the life of using a
platform that's still in development by a small team. Some way to organize my
grains is the big one left right now. But especially recently a lot of the
stuff "down the roadmap" is starting to feel finished. The Powerbox features
are getting there. Active Directory integration exists now.

Since somewhere in 2014 when I started playing with it, it's always felt like
it was moving forward.

~~~
SNvD7vEJ
Agree, I miss some way to organize, group or tag the grains/docs.

If a folder structure will be implemented, sharing options should also be
available for folders, I think.

~~~
kentonv
That's coming, probably in the next week! :D

------
middleclick
I am using Sandstorm and I love it. Nothing beats the power of hosting your
own private cloud. However, I really wish there was a way around the wildcard
certificate requirement. One of these days I am going to shell out $100 for a
wildcard cert or I can just hope Let's Encrypt will support it!

~~~
kentonv
Thanks!

You probably know this already, but for others: Sandstorm offers free wildcard
certs if you'll settle for a subdomain of sandcats.io.

And because I'm sure other people will ask, here's the FAQ on why we need the
wildcard certs:
[https://docs.sandstorm.io/en/latest/administering/wildcard/#...](https://docs.sandstorm.io/en/latest/administering/wildcard/#why-
sandstorm-needs-wildcard-dns)

------
theseoafs
Speaking of which, what is the status of Cap'n Proto? I was looking at using
it for a side project recently, but it doesn't look great on the surface:
there haven't been any updates on the site since March of last year, and that
post was to address a security vulnerability.

~~~
kentonv
Cap'n Proto is essentially a sub-project of Sandstorm. Lately, there hasn't
been a whole lot of churn in Cap'n Proto land as it is doing what we need it
to do. That said, there are a few features I'm becoming increasingly eager to
implement, probably later this year. The biggest one is three-party handoff:
when Alice holds a capability pointing towards Carol and sends that capability
to Bob, then Bob should form a direct connection to Carol rather than proxy
requests through Alice. This will be important for improving scalability of
the cluster version of Sandstorm, although at the moment it's getting along
fine without it.

But yeah, Sandstorm uses Cap'n Proto extensively, so if there's no updates to
Cap'n Proto, it's because its biggest user hasn't seen a need to make many
changes. :)

------
slewis
Can you explain the "Fine-grained Object Containers" concept a little more?

For example, how is this enforced: "JavaScript running in the user’s browser
can talk only to the document container for which it was loaded, no others."

Do apps need to be SandStorm aware? Thanks!

~~~
kentonv
Hi! There's an extensive explanation here: [https://sandstorm.io/how-it-
works](https://sandstorm.io/how-it-works)

To answer your questions:

> For example, how is this enforced: "JavaScript running in the user’s browser
> can talk only to the document container for which it was loaded, no others."

Each grain (or actually, each session on each grain) is assigned a randomly-
generated subdomain. These domain names are unguessable, so there's no way for
one grain to know another's domain, and anyway same-origin policy generally
prevents Javascript from interfering with other domains. (In the future, we
plan to further lock down the client side using Content-Security-Policy,
although we haven't done this yet.)

> Do apps need to be SandStorm aware?

Yes, apps need to be packaged for Sandstorm, and that usually involves some
modifications to the app. However, the modifications are usually mostly
deleting code: you delete your user management and rely on Sandstorm, and you
delete your multi-document management and instead let the user create multiple
grains at the Sandstorm level. Typically, this work takes a few hours, though
it varies depending on the app.

(It's also worth noting that apps which target exclusively Sandstorm from the
start are much easier to write than normal web apps: User management, document
management, security, access control, etc. are handled for you.)

~~~
slewis
Thanks. I'd poked through the howitworks. What threw me off was the reference
to Etherpad, which made me think apps could somehow run on SandStorm
unmodified. Makes sense now!

------
Hupriene
Is there a way to run e.g. an email server on Sandstorm?

Last time I looked it seemed like that would never be supported because it
would violate the sandstorm business model, which, as I understand it, appears
to be selling very cheap hosting of grains, made possible because only disk
resources are used when an grain is not currently in use.

This would seem preclude the hosting of services that need to be running all
the time in the background.

~~~
kentonv
Sandstorm's business model is selling enterprise-oriented add-ons and support
(e.g. ability to integrate with LDAP/SAML/Active Directory, or (eventually)
providing HIPAA compliance, etc.). The hosting service actually exists more so
that developers know that their apps are usable by everyone, not just people
who run their own server.

There's actually no need for a mail server to be always-on. It could start up
whenever a message arrives. Sandstorm is good at this kind of on-demand
starting.

We very much intend to support using Sandstorm as your email server. It turns
out, though, that e-mail is a complicated subject which operates on a
different set of protocols from the rest of the web (SMTP vs. HTTP) and has a
whole lot of hairy stuff you need to do to make it actually work (SPF, DKIM,
...). Since there are quite a few options out there already for handling
email, we've chosen to focus elsewhere in the short term, but I hope to loop
back to email in the coming months.

------
JD557
All the examples I see on your demo are web apps (and the documentation seems
to be focused on web apps as well).

How well are applications without a web-interface supported?

For example, could I easilly package and launch something like a teamspeak
server on sandstorm, or would I need to add some mock web server that kept the
real server online?

~~~
kentonv
You would need to add a web UI for configuring the server, as the user won't
be able to SSH in and edit config files. (This is by design: forcing users to
use shells and text editors is terrible UX.)

Currently, you would also need to request a raw IP networking capability from
the user. Only admins of the Sandstorm server are able to grant this
capability, for security reasons. So if you are self-hosting Sandstorm, you're
good, but you wouldn't be able to run the app on Oasis currently. We are
working on better solutions here.

Jason (a Sandstorm employee) has been playing around with packaging a Mumble
server, proving that this sort of thing is possible:
[https://groups.google.com/d/msg/sandstorm-dev/kfd-
zcSJqb4/2f...](https://groups.google.com/d/msg/sandstorm-dev/kfd-
zcSJqb4/2fnAvEPxBAAJ)

------
Santosh83
I realise I'm coming late to this thread, but just a quick question. But
before that, a big hats off to everyone involved with Sandstorm! Making a
complex platform point & click is a huge achievement.

Coming to my question, does Sandstorm have a HTTP server in its app platform?
While I'm excited about other apps like file sharing/storage, what I badly
want to do right now is to be able to host my site on my own machine using
dynamic DNS. Of course I can manually do it by installing Apache, configuring
it, updating a dyn DNS provider and so on. But if Sandstorm offers a point &
click way to do that, then I'm hooked right now. :-)

~~~
kentonv
If it's a static site, there are several apps on Sandstorm you can use:
Wordpress, Ghost, Hacker CMS (which is basically Jekyll + a text editor), or
Davros (to which you just upload files, and then publish). Any of these will
allow you to connect an arbitrary domain to the page so that visitors see your
content only, not Sandstorm.

If you want to publish a dynamic site with server-side logic, that's currently
not supported, unless you are OK with it being a Sandstorm app, which means
everyone who accesses it will see the Sandstorm UI around it.

~~~
Santosh83
Thanks for replying! A static site over SSL on a custom domain will do well
right now. Presently I'm using the motley combination of uploading HTML to
Github Pages and placing Cloudflare in front (for SSL on a custom domain), but
Sandstorm sounds like a more painless alternative that would also mean I
retain my data on my own machine. Will get started!

------
conradk
How do app updates work with Sandstorm ? For example, if I host an instance of
the Ghost blog. Now, say it updates from version 1.2 to version 2.0 (which
means probably a breaking database schema change or something). How would
Sandstorm go about updating it ?

~~~
dwrensha
A grain's filesystem consists of read-only app data mounted at / and writable
grain storage mounted at /var/. From Sandstorm's perspective, upgrading a
grain to a new app version just means launching the grain with the new app
version's read-only data, leaving the writable /var/ data untouched. The app
is then responsible for detecting if any migrations are required and for
performing them if so.

Some apps, such as WordPress, already have logic for automatically updating
the database when the code changes. Other apps need to add some simple
version-detection logic in their startup scripts.

~~~
conradk
Thanks for the explanation !

------
xg15
Very cool! Last time I checked Sandstorm, the powerbox system was still in
development. Now it seems finished, according to the docs. Is there are more
in-deptg doxumentation about the UI flows and APIs?

~~~
kentonv
It's not totally finished yet, but parts of it are finished and will start
being used by real apps in the next week or two.

The reason it has taken a while is not because it was hard, but rather because
we've been repeatedly pulled away by other, more-pressing concerns, e.g. basic
usability issues or features like the activity/notifications API (which turns
out very important for people trying to do real work on Sandstorm).

The most in-depth documentation is the .capnp files in the source tree. We
will be writing better documentation as the features stabilize (soon).

------
andrewstuart2
OS seems a little presumptuous. As does the majority of the "how it works"
page. Making wildly ridiculous claims that amount to "microservice are silly"
(my paraphrase) when really they've only managed to successfully sandbox
applications. Much like Android with its "a user for every app by default."

> Sandstorm takes a wildly different approach [than containerized services]:
> It containerizes objects, where an object is primarily defined by data

Or... You've containerized the data access service and enforced a per-
application sandbox. Plenty of precedent there.

> Microservices are seemingly designed to match the developer’s mental model
> of their system. Often, services are literally divided by the team who
> maintains them.

Good Lord, I sure hope that's not true. Maybe if the non-technical manager
designed the system.

> In the Sandstorm model, all of the components (from web server to database)
> needed to handle a particular grain run on the same machine.

So your container scheduler can optimize for I/O. Again not that far from
existing cloud schedulers.

I'm not saying this isn't a powerful model or platform. I am saying that I'm
very disinclined to want to work on such a platform, purely because I'm
worried its developers buy into the "everybody else is way off base" attitude,
when really they're only doing what many many successful architectures have
done before them.

~~~
kentonv
Hi! I'm the founder / tech lead of Sandstorm. I wrote that "How it Works" page
([https://sandstorm.io/how-it-works](https://sandstorm.io/how-it-works)) and
designed the architecture it is describing. Before Sandstorm, I worked at
Google on infrastructure for eight years. As you may know, Google has been
doing containers and microservices for over a decade. What I describe on the
page is largely based on that experience.

> Plenty of precedent there.

I've personally never observed a production (non-research) server architecture
which containerizes and sandboxes application servers on a _per-document_
level. That is, say, with Etherpad on Sandstorm, every document you create is
in a separate container isolated from the others.

I would love it if you could name some specific examples that work this way,
as I'd very might like to see what others have done.

~~~
solipsism
I get the grain-per-document thing. Sounds interesting for at least a subset
of applications, and disagree with the grandparent that it's common practice.

First question that comes to mind is how the data source is sharded like that.
You say in another post that you don't mandate a tech stack. So let's say I
want to store my data in MySQL. Are you saying each document literally has its
own private database instance?

I have many more questions, but they will depend on your answer to that
question. :)

~~~
kentonv
> Are you saying each document literally has its own private database
> instance?

Yes. Each "grain" (our catch-all term; could mean "an Etherpad document" or "a
Gitlab repo" or "a Wekan board", etc.) gets a slice of filesystem to do
whatever it wants with.

Of course, MySQL is totally overkill for this task. We generally encourage
people to use sqlite instead. But there are a couple apps that do actually run
MySQL in the container, mostly because switching to sqlite would have been too
much work. There are also several apps that use Mongo, but it turns out Mongo
3's WiredTiger engine is reasonably efficient for small data sets, so it works
out.

The model definitely is friendlier to some kinds of apps than others.
Productivity apps with clear data boundaries -- Etherpad (document editor),
Wekan (Trello clone), Rocket.Chat (Slack clone), Gitlab, etc. -- work very
well, and are our main focus. "Big data processing" apps are not well-suited
to this model. We're happy to cede those, as there are tons of people doing
great work in that domain.

~~~
simplehuman
In the case of Rocket.Chat, is each channel a grain or is the entire chat a
grain? It appears to me it should be a grain but I am not sure.

~~~
kentonv
Currently, the whole Rocket.Chat instance is a grain.

However, it would be "more Sandstorm-y" if each channel were a separate grain.
For that to work well, though, we need some UI features to emulate what
Rocket.Chat already does internally:

* The ability for grains to highlight themselves in the sidebar when there is activity or when you've been mentioned, so that you know to check them. This API actually just launched last week! So now the apps are being updated to use it.

* Some way to create "private message" grains between two people. We haven't done this yet but we have some ideas.

Soon, I hope to see Rocket.Chat add a "single-channel grain" mode where its
own sidebar is removed and it relies on Sandstorm's sidebar. This would
provide better security for channels which should only be accessible to a
subset of people. Probably, the multi-channel mode will stick around too, for
cases where you want the same people to have access to many channels.

------
brudgers
Scott Hanselman's interview with Kenton Varda about Sandstorm [2015]:

[http://hanselminutes.com/497/your-personal-cloud-platform-
wi...](http://hanselminutes.com/497/your-personal-cloud-platform-with-
sandstormio-and-kenton-varda)

------
SNvD7vEJ
How do I report a bug in an app?

The email app Roundcube does not seem to support unicode characters in the
senders name in received emails.

So the displayed name in received emails gets garbled if it contains e.g. a
swedish character like 'å', 'ä', 'ö'.

~~~
kentonv
In the Sandstorm UI, if you go to "Apps" and choose the app, you'll see a
"report issue" link in the upper-right. Generally, these link to the GitHub
issue tracker for the app or its Sandstorm packaging.

~~~
SNvD7vEJ
Thanks.

I have so far just tried the demo, but I find it surprisingly pleasant to use
and well polished.

Impressive work.

------
ipfsuser
Any plans to support saving/loading grains to IPFS?

------
jacke
always wondered by such projects. you have the resources, and you have made
analogues of paid services, and that's cool, but what's next? I think you can
offer people more than just a free Slack. Do you have any plan to integrate it
with commercial products?

~~~
eitland
If you are asking how they will pay their bills: They offer a hosted version.

If you are asking if commercial software will be available: it already was
from the start; draw.io is not open source but could i tefrate right away
anyway.

~~~
nullcipher
draw.io has become unmaintained and unsupported.

~~~
eitland
Yep, but still proves my point that commercial apps were available from the
start.

~~~
nullcipher
You mean proprietary and not commercial.

~~~
eitland
Ok, thanks for the correction.

Upvoted and appreciated.

------
aBoss
I really like it but it seems it cant work on the old linux server kernel.
PLEASE fix that

Thank you

~~~
kentonv
Unfortunately this isn't feasible. Sandstorm requires some modern Linux kernel
features to set up its sandbox. Kernels before 3.13 simply did not have the
features necessary, and there's no alternative approach we can use to get the
same effect. Sorry. :(

------
kevinSuttle
Looks pretty solid. Is there a comparison to other similar products like
OwnCloud?

------
erikb
Why is it an operating system? Looks like owncloud not like linux to me.

------
known
I'll try;

------
ktamura
Previously on HN:
[https://news.ycombinator.com/item?id=8972066](https://news.ycombinator.com/item?id=8972066)

As a marketer, I am all for launches, re-launches and marketing, but it's
unclear to me what's different now compared to a year ago.

~~~
kentonv
FWIW the submitter (who isn't affiliated with Sandstorm AFAIK) explains his
reasoning here:
[https://news.ycombinator.com/item?id=12104373](https://news.ycombinator.com/item?id=12104373)

But lots and lots has changed in the last year. :)

~~~
beefman
What's changed? I'm on your mailing list and I don't recall anything.

~~~
kentonv
Well...

* Oasis (managed hosting) was launched less than a year ago.

* That came together with a huge UI overhaul that introduced the sidebar / tabbed UI.

* It also came with the full app market, whereas before we only had a static HTML app list.

* We started offering free SSL certificates for self-hosted servers under sandcats.io.

* We introduced Sandstorm for Work, which integrates with LDAP / SAML (including Active Directory) for login, offers organizational user management features, etc.

* We added app signature verification via Keybase, proving that apps came from the twitter/github/HN user you expected.

* We implemented app update notifications with one-click updates.

* We added signature verification to the installation and update paths for Sandstorm itself.

* We enabled apps to serve WebDAV APIs, which enabled Davros (a file sync app) and Radicale (a CalDAV calendar app).

* The community wrote / packaged 30 new apps, bringing the total to 58.

* We added the ability to link multiple login services to a single account.

* We implemented the Powerbox grain picker, which allows you to share a grain through another app. (This will begin being used in real apps in the next couple weeks.)

* We added unified notifications and read/unread indicators, so you can keep track of activity across your grains.

* Many other, smaller things. See: [https://github.com/sandstorm-io/sandstorm/blob/master/CHANGE...](https://github.com/sandstorm-io/sandstorm/blob/master/CHANGELOG.md)

~~~
eitland
Sneaking in a feature request here:

Can we have update descriptions for apps as well?

~~~
kentonv
Yeah, actually, that wouldn't be too hard for apps that ship a changelog.

------
rosstex
I forget, what's the name of this OS?

~~~
aphextron
I think it's "Harder Better Faster Stronger"

------
lechevalierd3on
Whaou I just realized for the first time that this company is named after the
Darude's one and only hit.

~~~
kentonv
For the record, I wasn't thinking of Darude's Sandstorm when I named the
project. :)

EDIT: What I was thinking, at the time, was "sandbox in the cloud". But these
days I like to retcon the name to be about granularity. Sandstorm's major
technological innovation is fine-grained containerization, which is described
in detail here: [https://sandstorm.io/how-it-works](https://sandstorm.io/how-
it-works)

~~~
lechevalierd3on
I will never believe it.

~~~
nickpsecurity
Grains of sand in a cloud is a sandstorm. Storms and clouds coming in are both
said by people a lot. Synonymous in some situations. Some talk about
sandstorms. It's believable that the connection was made without knowing...
whatever work you're referencing. Never heard of it.

------
jondubois
It supports too many apps/frameworks - It's too ambitious and not focused
enough. Also, not all apps are the same when it comes to scalability - That's
a false assumption which pretty much of all PaaS providers make.

There is more to scaling an app than just adding more machines - Services need
to be scaled independently of each other based on usage; also, some services
should run on their own dedicated hosts, some services require persistent
storage also there are needs to be rules to define how services can talk to
each other, recover from failures, etc...

Developers NEED some way to specify HOW their services should scale - It can't
just be fully 'automatic' \- That's not flexible enough.

If you look at typical Kubernetes config files, they are quite long/complex;
this is because there are a LOT of factors/variables involved when it comes to
running services inside distributed containers and you can't just abstract
away from those details without seriously compromising flexibility.

~~~
kentonv
Sandstorm is not a PaaS and is not intended to be an alternative to
Kubernetes. It is not designed for running SaaS services intended to scale to
millions of users. It is designed for running personal or internal employee-
facing services, where there are anywhere between 1 up to maybe 10,000 users.
The kind of scaling you describe simply isn't an issue in this domain.

Moreover, Sandstorm takes a completely different approach to scalability
called fine-grained containerization. The problems and solutions are really
completely different from a Kubernetes-like approach. Read about it here:

[https://sandstorm.io/how-it-works](https://sandstorm.io/how-it-works)

~~~
Taek
Sandstorm is exciting to me for personal reasons. Scalability is not an issue
there, because all my apps already need to run on only one computer.

Security and encryption is really important to me. Synchronization as well.
Very excited about sandstorm because it seems to fix a lot of the gaps in my
current flow.

