
Show HN: Sandstorm Personal Cloud Platform Demo - kentonv
https://sandstorm.io/demo
======
AceJohnny2
Just to tie a few other things together, kentonv was the primary author of
Google's Protocol Buffers, and current author of open-source Cap'n Proto [1].

While at Google, he initiated the Jeff Dean Facts prank [2], and in his spare
time part-designed his own LAN-party-optimized house. [3]

So, you know, if he thinks he can provide a differentiating hosting platform,
I'd have a glance.

[1] [http://kentonv.github.io/capnproto/](http://kentonv.github.io/capnproto/)
[2]
[https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe](https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe)
[3] [http://kentonsprojects.blogspot.com/2011/12/lan-party-
optimi...](http://kentonsprojects.blogspot.com/2011/12/lan-party-optimized-
house.html)

~~~
kentonv
Oh my. Thanks!

Just need to correct one thing: I didn't invent Protobufs (Jeff Dean and his
colleague Sanjay Ghemawat did), but I did write most of version 2 and was
responsible for open sourcing it.

~~~
AceJohnny2
Thanks for the correction! In a way, it's kind of demoralizing that it's
actually yet another product of the Jeff & Sanjay Brain Trust. It highlights
how much they're in a league of their own, unattainable.

That said, my googler friends say they much prefer ProtoBufs v2 ;)

~~~
kentonv
Part of the reason Jeff and Sanjay invented half the tech at Google is that
they're incredibly smart. The other part is that they were there very early
on, and basically have had free reign to spend their time however they want.
Of course they spend it all inventing things, which means they output a lot of
awesome stuff. Certainly not just any hacker can produce the kind of quality
work they do, but I think there are a lot of other very smart people at Google
and elsewhere who _could_ be doing similar things if given the same deal. But,
most of those other people didn't bet on Google as early as Jeff and Sanjay
did, so...

------
kentonv
Thanks for the load test, HN! :) Several hundred users spinning up instances
of apps yet the server hasn't broken a sweat...

We still need your help to get this into production. Please check out the
crowdfunding campaign:
[http://igg.me/at/sandstorm](http://igg.me/at/sandstorm)

~~~
e12e
Great stuff. Could you share some data on the server and the traffic?
Ram/cores/uplink? This looks like the kind of stuff I've been thinking about
for a long time - will have look at the code and test it on a server. Is
everything in place to run a clone of your demo server easily?

~~~
kentonv
Sure!

Over the course of yesterday we had 747 demo users launch 952 app instances.

Currently we're on GCE for price reasons, though we're looking for good non-
Google, non-Amazon options for our eventual release.

Sandstorm tends to be memory-bottlenecked since we're launching separate
instances of each app for each user, although we make up a lot of it by
shutting down instances when they aren't in active use.

I was super-paranoid about the demo totally breaking under load (having not
tried this before), so I upped the server to GCE's largest-memory instance at
104GB of RAM.

That turned out to be completely unnecessary. It never used more than 6GB (and
CPU was mostly negligible). Doh. Oh well, better to have spent too much than
too little.

We also have a separate front-end server which does our SSL, so that our
private keys don't reside on the same server as apps, just as an extra
precaution. That machine is just a regular n1-standard-1 instance. CPU usage
was in the single digit percentages there as well.

You can indeed run your own Sandstorm instance. There are installation
instructions in our Github readme: [https://github.com/sandstorm-
io/sandstorm](https://github.com/sandstorm-io/sandstorm)

Feel free to send me questions.

~~~
e12e
Thanks for the detailed reply. I had some trouble building the whoke thing
under Debian stable (with a 3.14 kernel from backports, trying to force
everything to work under my user id in $HOME/opt. Not too surprising. Meteor
serms to be part of the problem (got it working with its self-contained binary
install) - as well as the apparent need for a newer libfuse and some c++
headers I couldn't quite figure out. Although I was building with clang-3.5).
I'll have a go at getting the vagrant box up, and then trying again in a
Debian testing vm/chroot. If that fails, I'll have to go with an ubuntu
vm/chroot :-)

At any rate, sounds like my 4 core gbuplink hetzner box with 16gb ram should
have no problem working with this (obviously overkill for a personal playpen,
but its good to know I have lots of headroom...).

Are you planning to stay with meteor going forward? The other components seem
much less complicated to get up and running, and more sys.adm. friendly. An
example being the need (?) for node/npm _and_ meteor's bundled node/npm...

~~~
kentonv
It sounds like you were installing from source. There are indeed a lot of
moving parts to deal with there. We also have a binary installer (and auto-
updater) which "just works" with essentially no dependencies. I understand
some people aren't comfortable with binary installers, but you have to
understand that building from source will always be more work. :) That said,
feel free to file issues for any obstacles you encounter that aren't already
covered in the docs.

We do plan to stick with Meteor. It's a pretty awesome framework. I find
Meteor is incredibly easy to get running, but this is again due to its binary
installer script which maybe you don't like. :)

I believe that Meteorite (the "mrt" tool which you had to install separately
using npm) is slated to go away before Meteor 1.0 as that functionality will
be rolled into Meteor itself. So, that part will get better, at least. For
now, you actually can (and arguably should) install it using Meteor's bundled
npm rather than the system npm.

~~~
e12e
Actually, the mrt tool wasn't a problem (I had node installed from source, in
my home-dir under opt from before -- working quite well with xstow. Meteor not
installing from source is partly due to me running Debian stable -- I'm not
surprised when bleeding edge frameworks won't compile, just a little bit
disappointed ;).

I was indeed installing from source, as that is a requisite for contributing
to the project (if I can't build it how could I possibly submit patches).

I have no problem with you citing any one distribution/version as supported
however -- it would be madness to demand support for "everything" from an
alpha source release! And I think aiming for cross-platform-distribution
comparability would be wasted effort at this point.

With Vagrant from wheezy-backports, everything came up fine -- just had to(?)
increase the vms ram to 512 mb for ipython notebook to work.

Now my only problem is figuring out how to get the hacker cms to publish
anything (to users without login). I've understood that the cms should be
stuffing files under /var/www (in my vagrant/virtualbox vm) -- so I'll have to
check if "publish" pushes anything there -- and if there appear to be an
nginx-instance set up to serve files from /var/www...).

As for meteor -- I have a few reservation of their architecture in general --
but my main problem is the complexity of the setup -- especially how hart it
seems to be to build everything needed from source. But then again, they're
not yet at 1.0, are they?

(Still seeing as how easy nodejs is to build from source, it is a shame meteor
manage to somehow mangle that process...)

~~~
kentonv
The recommended way to "publish" from Hacker CMS is to follow the in-app "DNS
setup" instructions, rather than point nginx directly at the storage. The
latter was the old way of doing it -- is there a doc you're seeing somewhere
that still recommends that?

~~~
e12e
No, (not other than the api docs for devs, that explains that one way to
publish from (any) app is to push files to /var/www). I guessed the dns-setup
was one way -- wasn't sure if that would work without a routable address
though (eg: 10.0.0.0/x on a lan). Or in my case, with the test setup in a
vagrant box.

[edit: That is, I couldn't find any docs on how to publish from hacker cms,
but I did find this for devs:

[https://github.com/sandstorm-io/sandstorm/wiki/Publishing-
to...](https://github.com/sandstorm-io/sandstorm/wiki/Publishing-to-the-
user's-domain) ]

~~~
kentonv
If you click on the "Setup DNS" button inside the Hacker CMS app, it gives you
instructions for how to publish to your domain.

------
nobodysfool
That's pretty amazing how quickly I can install and use an app. It's instant
as far as I can tell. This makes me wonder - how difficult would it be to take
a sandstorm app and migrate it to the internet? And how difficult is it to
migrate an app to your platform?

~~~
kentonv
Sandstorm uses a native-code sandbox on Linux which can run standard Linux
binaries, so generally porting an app to Sandstorm is pretty easy. Most of the
work is actually in removing the app's existing login system and relying on
Sandstorm's authentication instead. David touched on this in his post about
porting ShareLaTeX last week:

[https://blog.sandstorm.io/news/2014-07-25-sharelatex.html](https://blog.sandstorm.io/news/2014-07-25-sharelatex.html)

Most of our existing apps are actually ports of existing open source Linux-
based servers. Most were ported by Jason Paryani over the last month, which
tells you how easy it is. This includes apps written in PHP, Python, Node,
etc.

Note that the dev tools (and all the platform source code) are available to
anyone. There are some WIP docs on the wiki:

[https://github.com/sandstorm-io/sandstorm/wiki](https://github.com/sandstorm-
io/sandstorm/wiki)

It's probably actually harder to port _away_ from Sandstorm simply because you
have to add a login system and other such features. Adding is a lot harder
than removing. :)

~~~
chatmasta
So the apps on the platform are essentially forked versions of open source
apps. If you're editing the authentication system, which I imagine can
introduce some heavy dependencies, how do you plan on handling updates from
the original developers? Presumably there will be manual labor involved with
merging the origin master branch into the Sandstorm app. Do you plan to do
this yourself?

Nice implementation.

~~~
kentonv
> If you're editing the authentication system, which I imagine can introduce
> some heavy dependencies...

Not really true. Sandstorm acts as a proxy in front of the apps and adds
headers which identify the user. It takes very little code on the app side to
read these headers and use them rather than use the app's own authentication.

That said, we do have some changes which we'll have to rebase from time to
time. Eventually we hope that app developers will be interested in maintaining
these changes themselves and explicitly targeting Sandstorm, but that is
probably a ways away still. In the meantime, maintaining the edits doesn't
seem to painful.

------
wmf
Wired article explains the background: [http://www.wired.com/2014/07/out-in-
the-open-sandstorm-makes...](http://www.wired.com/2014/07/out-in-the-open-
sandstorm-makes-it-easy-to-control-your-apps-in-the-cloud/)

------
swah
A cool project in this "own your data" space is
[https://camlistore.org/](https://camlistore.org/) which IIUC could be the
storage system of a personal cloud.

OT: I always take a closer look when a computer screen appears in a video - in
this case I couldn't recognize what you're running there. My guess would be a
a tiling window manager? But the other screen shows Windows so I'm curious.

The code on the right window looks like Haskell due to the one liners.

[https://dl.dropboxusercontent.com/u/19746944/kv.png](https://dl.dropboxusercontent.com/u/19746944/kv.png)

(Also, its kinda surprising/refresing that Sandstorm isn't written in Go or
Node... )

~~~
kentonv
>
> [https://dl.dropboxusercontent.com/u/19746944/kv.png](https://dl.dropboxusercontent.com/u/19746944/kv.png)

That's me running Qt Creator (my personal favorite IDE) on Linux. You may be
disappointed to learn that the code is C++. :) Sandstorm's lower layers are
all C++ since we interface closely with Linux syscalls, though I think in that
shot I'm actually editing Cap'n Proto (which Sandstorm uses).

All of the machines in the video are running Linux, with GNOME 3 as the window
manager.

> Camlistore

Yep, I've talked to Bradfitz about Camlistore a few times.

Sandstorm actually gives each app instance its own directory on the filesystem
for storage, so that it can use any storage format it wants. So an app could
use Camlistore. It may also make sense for Camlistore to act as an independent
app which other apps connect to. But in any case, Sandstorm is agnostic to
these things.

------
techaddict009
I was trying out Demo and found that you do not have it wordpress in sandstrom
apps list. Hope to see it soon.

~~~
kentonv
Ah, the 800lb Gorilla of open source web apps. We wanted to highlight some of
the little guys first, but... it's coming.

------
etherealG
oh wow, the demo setup 'just works' and is instant to install apps. this just
keeps getting better and better.

