
Sovereign: Ansible playbooks to build and maintain your own private cloud - c0restraint
https://github.com/sovereign/sovereign
======
dguido
There are so many servers and apps being installed by Sovereign that I'm
certain few would be able to keep it secure
([https://github.com/sovereign/sovereign/wiki/Software-used-
by...](https://github.com/sovereign/sovereign/wiki/Software-used-by-
Sovereign)). The big win for the cloud is that you're paying a fraction of the
cost for access to a, typically, enormous security and operations team. If you
want to build software like this that allows people to self-host, you need to
scale down what you deploy to what a single person can reasonably manage. This
isn't it.

Fun todo: Install this somewhere, nmap it for open ports, then ask "How many
of these services had a remotely exploitable CVE in the last year?" "If one of
these services had one tomorrow, would I know to patch it and take action
faster than someone would takeover my box?" I don't see any containment
mechanisms on any of these services beyond what's included by default so a
compromise of one service likely leads to total compromise of the entire box.

I had to think about this a lot with AlgoVPN
([https://github.com/trailofbits/algo](https://github.com/trailofbits/algo)),
and we built a system with no out-of-the-box remote administration, strong
isolation between services with AppArmor, CPU accounting, and privilege
reductions, and limited third party dependencies and software. You can't count
on a full-time, expert system administrator.

~~~
wpietri
This touches on a problem I've been thinking about a lot. AWS, etc, have
solved the problem of hardware operation. Containerization is doing the same
for OS operation. I think the next thing is _app_ operation.

For some things, I'm happy to use SaaS providers, where they are responsible
for the whole stack. For others, I'm happy to use apps, where they just
provide the code, and I provide the platform. But for a number of things, I
want something in between: I provide storage and compute, they provide code
and operations.

Bitwarden for me is a good example. They're a password manager who provides
their backend as a docker container anybody can run. I like that, as I don't
really want them to have my data, and if they go out of business, I don't want
to be cut off from my passwords. But I won't run the backend myself, because I
don't have the time and expertise necessary to make sure it stays secure.

Another good example is photo hosting. I would rather keep all my photos on
space I control. But I also need modern, maintained software for syncing,
serving, and controlling access to photos and related data. I'm happy to pay
somebody to make and maintain that software, but not nearly as happy if that
means that at any point they might shut down and take my data with them.

I suspect we're headed toward a future where people like Synology and Digital
Ocean sell storage+compute, and then other companies sell and maintain user-
selected software that runs on those environments. Basically, some sort of app
store for servers. But I'd love to see this happen in an open, nonproprietary
way, as the drawbacks of Apple's and Google's app stores have become pretty
clear.

~~~
yjftsjthsd-h
Sandstorm was a really nice solution to this, but it required each app to be
integrated with it, which I personally think is what killed it. (Which sucked,
because for what it supported it was the best option available)

~~~
wpietri
Ah, interesting! Just reading the home page wearing my developer hat makes
this stand out: "Each document, chat room, mail box, notebook, blog, or
anything else you create is a "grain" in Sandstorm. Sandstorm containerizes
each one in its own secure sandbox from which it cannot talk to the world
without express permission."

The notion that every document is its own independent unit sounds pretty
menacing to me. Could be fine for some things, but getting things running
there is sounding like a fair bit of work to me, and very limited.

And then this part is especially bad: "[maybe someday] You won't have to deal
with payments | Eventually, we hope to make Sansdtorm implement in-app
purchases and deposit the proceeds directly to your bank account"

Right there a lot of incentive to integrate has leaked out. "Please build for
our platform in exchange for no money" is not quite the worst offer I've had,
but it's definitely not appealing. Looking through the Wayback machine, I see

But the part that really concerns me is that they seem to think that server
apps can run like mobile apps. To me one of the most powerful things about
SaaS products is that the aggregated use information drives both product and
operational improvements and allows rapid response to bugs and issues. So as a
developer choosing between this and a SaaS approach, this feels like having
one foot in a bucket to me.

For example, I recently outsourced my mail hosting to Fastmail precisely
because I want experts to run things. Would I be happier if the data were
stored somewhere I control? Definitely. But not if that means the experts
aren't paying attention anymore.

~~~
ocdtrekkie
Sandstorm's design is a little bit menacing! And it does require a fair bit of
work sometimes to fit web apps not built for Sandstorm into Sandstorm's model.
(The holy grail here is apps built for Sandstorm, but the platform needs to be
bigger before more developers do that.) Usually packaging for Sandstorm
largely entails locking an app into a single-document model, and stripping out
authentication (since Sandstorm handles it).

But the end goal is pretty well worth it: Any grain is incredibly secure by
default, and for the most part, app vulnerabilities are irrelevant. A grain
where only you have access doesn't need any sort of authentication or security
in the app at all. And since each document is it's own sandbox, sharing a
document with someone doesn't give them a way in to exploit access to your
other documents as might happen with a vulnerability in a more traditional
design.

The business model story for selling Sandstorm apps isn't super great right
now, you probably could have a licensing model that requested network access
through the Powerbox to check the license or something, but in many cases,
there's already a wide variety of great open source apps that are free and
just frustrating to host and manage without a platform like Sandstorm (or
Cloudron). (EDIT: Now that I think about it, Sandstorm used to have a paid
license key/feature key system that made no callbacks, I think the licensing
info was encrypted asymmetrically.)

As for your support of SaaS data collection, I just can't really agree with
you: People who want to give data to a developer can choose to do so, but I
think it's ethically wrong to collect data without permission. (Sandstorm
servers do have the ability to opt in to provide basic app usage data back to
Sandstorm's development team.)

I love FastMail, and have been an enthusiastic customer since 2016. :)

------
jlkuester7
This is a pretty cool setup! I have been tinkering in the self-hosting world
for awhile now and I would say my biggest piece of advice is to learn Docker.
(Not trying to shill for Docker here or start a Docker vs. Ansible flame war.)
In my personal experience, Docker images (and docker-compose orchestrations)
are more pervasive than Ansible playbooks and when it comes to self-hosting
for personal use, hosting all my services in Docker has made it much easier to
deploy/maintain them. (Not speaking to business use cases since in that
situation you should really have a deep familiarity with your tech stack and
be able to roll-your-own Ansible playbook/Dockerfiles.)

Also, there are so many great FLOSS alternatives to Google Apps. This repo
contains some, but here are some of my favorites:

* Drive/Calendar/Photos/Keep:
    
    
      - https://nextcloud.com/ (I prefer this over OwnCloud)
    

* Docs/Sheets/Slides
    
    
      - https://www.onlyoffice.com/
    

* Mail
    
    
      - https://mailu.io/ (basically a Docker-based deployment of Postfix/Dovecot/etc)
    

* Hangouts
    
    
      - Server - https://matrix.org/
    
      - Client - https://about.riot.im/
    
      - (I prefer Matrix.org over Jabber/XMPP)
    

* G+/Twitter
    
    
      - https://joinmastodon.org/

~~~
ggregoire
> Not trying to start a Docker vs. Ansible flame war

Don't worry, they serve very different purposes. You already probably know but
Docker is for running applications in isolation, while Ansible is for
provisioning and configuring hosts. For instance, you won't use Docker to
harden sshd on your hosts but Ansible.

~~~
bwbmr
Are there any good resources (besides OP) for using Ansible in a home-server
environment? Especially in a mixed docker + Ansible environment.

~~~
jefurii
After figuring out Ansible work I started using it to manage several personal
VPS machines and the various boxes at home. I just used the Ansible docs,
which are pretty good.

------
navaati
For my fellow HNers, this is "private cloud" in the meaning of Owncloud, not
of Openstack.

~~~
choward
I hear about Nextcloud a lot more than I hear about Owncloud. Does anyone know
why this project uses Owncloud instead of Nextcloud?

~~~
crashbunny
This project started before Nextcloud existed, I don't know why it hasn't
switched, though.

~~~
AdmiralAsshat
NextCloud was started as a fork of OwnCloud by a bunch of OwnCloud devs that
were unhappy with the direction the product was taking.

You can think of it in much the same way as OpenOffice vs LibreOffice: devs
fork to make a new product, the "original" product stagnates and is mostly
used for rent-seeking.

The downside of both is that, to my ears, both "OpenOffice" and "OwnCloud"
better signify to outsiders what the product accomplishes, while "LibreOffice"
and "NextCloud" really don't, unless you're already familiar with the product
or product history.

------
Tepix
Wow, didn't expect to see Sovereign at the top of HN today! I'm one of the
project contributors.

If this project piques your interest, please consider contributing! We could
really need more helping hands.

Ansible is easy to learn and most (not all!) problems due to new versions are
easy to fix.

Also, if you only want to use a fraction of what Sovereign has to offer to
reduce your server's attack surface, that's easy! Just follow the
instructions.

------
mindslight
On the general topic of Ansible and personal infrastructure:

Every time I attempt to use Ansible (or its kin) to manage my own network, it
feels overly obtuse and ultimately unhelpful. Its gains seem to be rooted in
configuring a large number of _identical_ servers, and isn't geared for a
handful of hosts with some commonalities and some differences. Writing
playbooks feels like a still-imperative wrapper around shell commands, just in
a bespoke and verbose YAML syntax.

Instead I am using my own script that runs a tree of files through a template
engine, drops them on each host being configured, and then runs triggers based
on what has changed. This seems utterly simplistic, lacks polish, eschews
common practices, etc. But the overall configuration seems straightforwardly
grokkable compared to the heavy tools.

~~~
asokoloski
I had a similar reaction, after trying out Ansible at my last job. We ended up
switching to fabric, which is all in Python. It was mostly good but had some
awkward warts, which it seems that they've mostly addressed in fabric2.
Anyway, it might be worth a look, based on my understanding of your use case.

~~~
mindslight
Fabric seems more down to earth, but doesn't itself solve the problem of
actually defining the configuration of each host.

Having said that, I am to the point where it would be really nice if my ssh
pushes ran in parallel, which is one of those robust niceties you give up by
going your own way. So I'll have to revisit Fabric because it would be
complementary - thanks for the reminder!

------
gramakri
For those in the market to run a private cloud, please try
[https://cloudron.io](https://cloudron.io) . Our motivation is to make it
simple to selfhost apps . The main advantage is that we take care of automatic
updates across all the apps we package. Happy to answer any questions.

Disclaimer: I am the co-founder

~~~
jlgaddis
Is it really self-hosting if you're running it on someone's cloud?

~~~
wpietri
Definitely.

I mean, you could play an infinite regress game. Do you own the hardware? Do
you own the cage the hardware is in? Do you own the building that the cage is
in, and the land that the building is on? And then we can go toward owning the
power company and the connections to anybody your servers talk to.

But in practice, self-hosting is about control. If what you're running it on
is a commodity cloud instance that you could get from a half-dozen providers,
then any one cloud provider has very little leverage over you.

~~~
Tepix
No. If you have dedicated hardware (rented or owned) and full disk encryption
you have decent control over your data. On a virtual server you have no
control and no privacy.

~~~
wpietri
Depends on what you mean by control. It sounds like you're worried about
different downside risks than I am.

------
djsumdog
I wrote something similar that's custom for my personal infrastructure:

[https://github.com/sumdog/bee2](https://github.com/sumdog/bee2)

There are some blog posts in the README that go into how I built a lot of it.
A lot of it is specialized for me though. I have a ton of rspec/tests but I
don't have a real config schema or entirely useful error messages. I might add
some in the future.

Looking at the list in this, I'd advice against nextCloud(ownCloud). I
recently setup their official Docker containers and the web piece works
alright, but their F-droid app continually crashes and I had to uninstall it
and the nextcloud-client in Gentoo's package manager segfaults at home and
refused to build at work.

I've read other stories of data loss with nextcloud. It might be better now
but my initial experiences made me use syncthing. Syncthing does use relays if
you're behind a NAT, but if you have openvpn setup, you can also force it to
use a direct IP address as well.

If you're thinking if self hosting and have the time, I'd suggest building it
yourself; borrowing (and properly accrediting/licensing) other open source
projects, their ansible scripts and containers and such. You learn a whole lot
about why this tooling is so complex.

------
TheFiend7
This looks super cool.

Though somewhat offtopic, this line absolutely cracked me up.

>A VPS (or bare-metal server if you wanna ball hard).

I can appreciate a sense of humor.

------
mekster
There seems to be a few questionable picks in the readme.

* Why pick ownCloud over NextCloud? The former's forum had 139 posts in the last 7 days and the latter's forum had about 1700. Also some of the features in the former product are locked for enterprise only.

[https://central.owncloud.org/about](https://central.owncloud.org/about)

[https://help.nextcloud.com/about](https://help.nextcloud.com/about)

* Tarsnap is a paid online service. You could try restic command to have encrypted backup to remote storages.

* cgit is an old project released more than 10 years ago and despite being written by the author of wireguard, we have far better stuff like Gitea (or its fork source Gogs) to have user access control with nice web interface for git project management.

------
jophde
I currently just let my desktop run constantly. It runs Windows in a KMS for
games and sleeping breaks it so I never even suspend. It only seems to use
about $10/month in power. I have been considering using DDNS through my router
and Cloudflare and trying to create a iOS/Android app that will automatically
upload my photos to the my DDNS for storage on my desktop. It feels a little
crazy but the idea of syncing my photos to my own machine with no middle man
is comforting.

DDNS seems like it's a local too good to be true for solving the dynamic IP
problem. I'd prefer to have a static IP for my gigabit Internet but sadly
Webpass doesn't allow it. Does anyone have experience doing something like
this?

~~~
hrjd
Ngrok as alternative to dynamic DNS?

~~~
jophde
DDNS maps a domain to your current dynamic IP. Its built into most routers.
They will push the new ip to a DDNS service when it changes.

------
platform
I looked at Soverign at the time I was setting up private cloud on a $5 VPS
(prgmr.com with 1.5 gb ram)

I went with YunoHost.

[https://yunohost.org/#/apps](https://yunohost.org/#/apps)

I initially tried sovereign, but once I figured out I had to pay for tarsnap
backup service, and that it did not have ansible for nginx setup (I needed
that experience for work stuff), I went with Yunohost.

Sofar I am happy with YunoHost and subscribed to send periodic donation to the
project.

Overall, though, if you are working with ansible at work, or want to advance
in devops field, learning ansible and contributing to Sovereign project would
be a good path to take.

------
crmrc114
Dumb question maybe... but why would you not just configure this yourself on a
single virt/host? Most of these services would take less than a day to
configure. So many questions on why this is a good thing.

Like, there are countless ways to configure your MTA and spam filtering- if
you are going to have to dig through this config.. why not just roll your own?

Can someone explain to me why you need ansimble for this? or am I just being
stupid and this is like an exercise to show what the toolchain can do?

~~~
tryptophan
If you're the type of person that configures servers on your own time for fun,
than you would likely find automating the process 'just because' to be fun as
well.

You do not need ansible for config of a personal server at all.

~~~
crmrc114
Oh cool cool, I thought I had a stroke or was missing something here. This is
a cool idea in that case.

------
haolez
As the CTO of an established company, I cannot imagine a situation where I
would prefer to maintain my own infrastructure vs using managed cloud
services.

If I get locked in on a specific product, it's way cheaper to redesign that
around an alternative vendor than it is to maintain a private cloud (Ansible,
Kubernetes and friends included).

As a nerd, I'd prefer to do things myself, but I have business needs to attend
to.

~~~
modoc
Cost. Depending on your scale and needs, running stuff on real hardware you
manage can be a lot cheaper than AWS/GCP.

------
slovette
Is there a benefit to doing this over something like Cloudron(1)?

I see this being for people that just want things to work without much of the
effort to make it so. If that’s the case, a simple web UI that treats all the
little solutions as “apps” in a way makes sense. Not plugging here, just
curious to the practical everyday differences.

1\. [https://cloudron.io](https://cloudron.io)

~~~
ocdtrekkie
Cloudron.io seems pretty nice, speaking as someone who uses/works on a
different platform (Sandstorm.io). Cloudron is going to cost you money if you
want the benefits of automatic updates and the like, which is a downside for
some (I think paying for good products/services is worthwhile), but I
absolutely think making management of a ton of apps simple like a phone is the
key to self-hosted online services. One of the biggest features of a good
self-hosting platform should be unified authentication and identity, so you
aren't managing your account on a dozen different apps.

I've always been impressed by Cloudron's well-maintained app library and
constant march of major feature improvements to the platform.

~~~
rlander
Just wanted to mention that Sandstorm is awesome. For some reason, despite
being around for years, I only found out about it recently. I really think
that it should be way more popular than it currently is. Lots of apps are
outdated, so I’m planning on investing some personal time to contribute to it.

~~~
ocdtrekkie
By all means, hit us up on the mailing list or in IRC if you have any
questions or need any help!

------
fak3r
I've used this project off and on for years, and it's always worked perfectly.
I'd have an infosec conference to go to, I'd setup a host with wireguard, give
my friends the cert, we'd all tunnel out though that, then tear it down after
the con. Total cost a few dollars (most are $5/month)

------
cs702
Does anyone here on HN have experience _using_ Sovereign in a team setting? I
have a few questions:

* _Mobile contact and calendar syncing_ : How well and reliably does it work?

* _Calendar group features_ : how well do they work?

* _Setup and maintenance_ : how much hassle is involved?

~~~
ses1984
Doing that stuff is hard.

Sovereign doesn't solve all your operational problems.

I think it's suitable for personal use. I wouldn't run it in a production
setting without thoroughly understanding all parts of the stack.

I would say it's good for personal use or to demonstrate what ansible is
capable of.

------
OJFord
Isn't this just duplicating effort that's probably already been done, in many
cases by the first-party maintainers, in Dockerfiles?

I don't mean to start 'Docker vs. Ansible', I just wonder why if you wanted a
quick way to setup a single-server 'own private cloud' you wouldn't just go
with what already exists, and list the images you want in a docker-
compose.yaml file?

(Which would additionally set you up for 'scaling' if you had any concern that
you might be able to save some cash with a two or three smaller servers than
one big one by the time you'd installed everything you want.)

~~~
thinkmassive
Not everyone wants Docker installed on their servers. It’s not a requirement
with this playbook.

Furthermore, if you do want to use containers, there are tools like ansible-
bender[1] that use Ansible to build container images.

[1] [https://github.com/ansible-community/ansible-
bender](https://github.com/ansible-community/ansible-bender) (edited the link
to point to the ansible-community repo)

~~~
OJFord
But I'd have thought if you care that Docker isn't installed on your servers,
you're probably not running this anyway?

It seems to me that the target demographic is people that just want the least
effort minimal faff way of getting some services up and running for personal
non-production use. And for that it was my suggestion that many of the
services probably already provide a Dockerfile upstream, so the easiest thing
to do would be to install docker-compose, list the images, and `up`.

~~~
thinkmassive
This project has nothing to do with Docker, and there are many reasons why
someone would want to avoid running Docker on a server (a separate issue from
using containers). I'm confused about why you think Docker is relevant to this
set of Ansible playbooks, and also why this would only be suitable for
"personal non-production use."

The only requirement for a remote host to be managed by Ansible is python, and
even that can be installed by Ansible itself using the `raw` module on an
initial run with nothing but ssh access.

No need to gather a bunch of random Dockerfiles from various places, tweak
them to be compatible, and create a docker-compose file from scratch... how is
that "the easiest thing" when this is a complete set of Ansible playbooks
where the work is already done?

~~~
OJFord
I meant easier than creating these Ansible playbooks from scratch, when many
if not all of the playbooks will do the same required steps as have already
been encoded in Dockerfiles in many cases by the maintainer of the upstream
project (the one being deployed) itself. There certainly could be value in
collecting that in one place still.

The first line of the readme says it's for a 'personal cloud'.

~~~
thinkmassive
The initial commit of this repo is from August 2013, a few months before the
first public beta release of docker-compose.

------
bilekas
Nice resource, will definitely give it a spin, also this is the first time
I've ever come accross `Tarsnap` looks really interesting too !

------
liotier
Nowhere do I see mention of how to update and upgrade this thing after initial
deployment... How does it work ?

------
haolez
I've used Ansible successfully to turn Windows laptops into Wintendo machines
:)

------
choward
One of the most important things when managing data is not losing it. Does
this have a solution for doing backups built in or do you come up with your
own solution like using something your web host provides?

~~~
nemoniac
Did you miss the bit about tarsnap?

~~~
choward
Apparently I did. Oops.

------
ggm
Version for a Pi.

Version for BSD.

------
Annatar
Quit this damn nonsense with Ansible or whatever garbage fashion fad is in
vogue these days and finally learn how to make OS packages so you can do
configuration management with normal shell scripting inside of them, because
that's what it's for. The amount of incompetence and insanity from
incompetence has gone too far. Damn it, this is exactly why IT sucks so bad!!!

~~~
davestephens
Maybe you've never worked with someone that sucked at shell scripting, or
packaging.

Ansible is awesome for enabling people to do reasonably complicated things in
a consistent manner, at scale, without having to write all of the boilerplate
code to be able to do so.

This is forgetting the fact that Ansible is reasonably opinionated, which is
great for lowering the barrier to entry and helping devs/admins to be
productive quickly.

When I just need to Get Shit Done, Ansible is awesome.

~~~
wtf1234
> lowering the barrier to entry

That's the opposite of a good thing.

> When I just need to Get Shit Done, Ansible is awesome.

"just" is the keyword. "just" instead of caring about long term
maintainability and security

~~~
apple4ever
LOL on your first point.

For your second, Ansible is specifically designed for long term
maintainability and security.

