
How We Engineered CMS Airship to Be Simply Secure - CiPHPerCoder
https://paragonie.com/blog/2017/03/how-we-engineered-cms-airship-be-simply-secure
======
ogig
I've been using S3 static sites more and more and I think that exposing any
kind of CMS control panel to the internet is a bad idea.

Take a look at the list of security concerns Airship tries to manage. Most can
be avoided by separating the content creation aspect and the publishing
platform. Content creators represent the smallest audience group your site
has, why then have a full application exposed to the internet running only for
them to easily change content? Content creators should have an offline app,
make changes, press publish and have those changes uploaded to a static
server. The whole CMS control panel paradigm seems kinda flawed to me.

~~~
devwastaken
Hardly is it that static content is the way to go when you're working with any
sort of dynamic data. If you're just making blog posts, Airship most likely
isn't for you.

An offline app would cause a number of its own problems in that regard,
because now you're actually having to design a backend that interface with
your webserver and puts the content to it. What happens when you have multiple
editors? You're now pulling information from a central resource, which is
going to be over the internet, whether its on a different server or not.

Not to mention, by having it 'offline', you can't have mobile apps to edit
content on your site.

Static definetely has advantages, but when scaled and working with data its
not that simple.

~~~
CiPHPerCoder
One of the projects on our to do list is actually creating a static publishing
module that can be used for either:

    
    
      - Offline editing for online publishing
      - Static mirrors to resist censorship in the face of DDoS
    

Updates would be sent from an Airship to a static hub, using challenge-
response authentication, with Ed25519 signatures.

Aside from the API that receives data and triggers an update (which is
intended to be set up on a different vhost), the attack surface will be
minimal.

The only reason this isn't already a thing is that I've had other projects
that were higher priority, including one I'll be releasing in a month or so
(but probably not submitting to HN; it's an entirely commercial project).

------
megous
What always boggled my mind was escaping data on the input side. Yet once it
was quite popular. PHP even had option to do it automatically. What a weird
idea. You escape data to prevent mistaking it for code whenever you use them
in some context. You don't know the context when receiving the input.

~~~
jarofgreen
The only people I know off doing that now are WordPress. Modern PHP really has
come a long way. (And in it's defence, a lot of good PHP at the time didn't
use this auto-escaping feature. It was the PHP's designers flawed [as you
point out] attempt to try and stop bad PHP developers shooting themselves in
the foot.)

In fact, while I found the article interesting I could have done with a bit
less of the barbs about other frameworks. If the only way you have of building
up your own tech thing is by tearing down other people's ...

------
Aardwolf
Names of tech products in hacker news titles can sometimes give a different
image upon first read.

This one made me think of engineers tinkering with mechanics on a giant flying
airship :)

------
theamk
> Since Airship self-updates, it needs to be able to write to itself.

> chown -R myusername:www-data airship

> chmod -R g+w airship

Have you ever wondered why some people say that PHP is insecure? That's one of
the major reasons.

This app claims to care about security, but at the same time it self-updates
and has no external code integrity control. This means that if the webapp has
a vulnerability which was exploited, it will be very hard to detect it -- was
this file changed by auto-updater or by malware? you don't know. And even if
the new version of webapp was released which fixes the bug, you can not be
sure that infection is gone -- because the update process may also be
compromised. And you cannot easily remove code dir and re-download -- the
settings are mixed-in with the code.

Compare it with any other stack -- Ruby on Rails, Django, JAR files, whatever.
Code cannot modify itself. The only writable thing is the database.
Additionally, the repository is often under git control. If there is a
vulnerability, update software and restart the server (if in doubt, backup the
database and nuke/rebuild the server instead). Even if you made a
configuration error and left code dir writable, "git status" is enough to tell
you any changed/added files.

And this is why you, as an admin, should avoid PHP if you care about security.
Sure, you can use sane techniques even with PHP code -- I am sure that's what
Facebook and Wikipedia do -- but most random PHP projects will require
writable code dir. When choosing a web app, choose non-PHP one first.

~~~
CiPHPerCoder
What 'dchest said, but also:

[https://paragonie.com/blog/2016/10/guide-automatic-
security-...](https://paragonie.com/blog/2016/10/guide-automatic-security-
updates-for-php-developers#privilege-separation)

Security at the expense of usability, comes at the expense of security.

CMS Airship went with the "self-writing code" option because we believe the
potential damage of a 1day vulnerability to be much a higher concern than
theoretical "this isn't really following best practices grumble grumble".

By all means, don't chmod/chown anything, and run continuum.sh as a more
privileged user if that's what you want.

Just don't disable automatic updates.

~~~
theamk
I strongly disagree -- I think "self writing code" has the worst failure mode
ever: any security vulnerability leads to permanent compromise which is
persistent across versions and requires non-trivial skill even to detect.

Because there will be security vulnerability, and people will be exploited,
and even if your update system runs every 4 hours, it will take dev team some
time to write up a fix and push an update. And obviously, the first thing
attacker would do will be to disable any integrity checks your auto-update
does. So in the most common case, the host will stay compromised forever.

This looks pretty horrible to me. I am appealed that you call this
"theoretical". But then, (in my experience) the security is not a strong side
of PHP community.

The sad part, it is not hard to fix. You already have separate update script
-- obviously "don't chmod/chown anything" is not enough, you at least want the
cache directory to be writeable, and while I have not looked at the code, I
suspect there may be other locations that web app will want to write to. You
can change your Dockerfile to use secure setup. It will take a bit of work but
it is definitely possible.

Until this is done, "simply secure" is only until the first compromise.

~~~
CiPHPerCoder
> I strongly disagree -- I think "self writing code" has the worst failure
> mode ever: any security vulnerability leads to permanent compromise which is
> persistent across versions and requires non-trivial skill even to detect.

The auto-updater does nothing to make this worse. That's true of any
vulnerable system. Once you're owned, you're owned.

> Because there will be security vulnerability, and people will be exploited,
> and even if your update system runs every 4 hours, it will take dev team
> some time to write up a fix and push an update.

It runs every 15 minutes, in the default configuration, and our usual
turnaround time for getting a bugfix out is less than 1 hour.

> The sad part, it is not hard to fix.

It's not broken, it's a trade-off. Nothing to fix.

> You already have separate update script

That's the kicker: If your threat model prioritizes "don't let the httpd write
to the web directory" higher than your usual non-power-user, then simply:

1\. Set up a cronjob to run the update script, run by a more privileged user.

2\. Don't chown/chmod.

The reason we do this _by default_ is because our threat model is ensuring
automatic updates work for everyone (including people who are not
sophisticated enough to set up a cronjob).

> You can change your Dockerfile to use secure setup. It will take a bit of
> work but it is definitely possible.

I guess?

The Dockerfile was provided by the community and is maintained by the
community; I don't use it nor do I touch it.

I'm not a Docker user.
[https://news.ycombinator.com/item?id=10269200](https://news.ycombinator.com/item?id=10269200)

> Until this is done, "simply secure" is only until the first compromise.

That's literally true of anything. If you get compromised, burn the machine
and recover from a backup.

~~~
theamk
> The auto-updater does nothing to make this worse. That's true of any
> vulnerable system. Once you're owned, you're owned.

No. If you have RCE in the regular app, the attacker cannot make attack
'stick'. Once you reboot the machine, or restart service (in case of systemd),
any malicious code no longer runs.

Now, if there is a privilege escalation bug the situation will get worse, but
having 2 unpatched bugs is significantly less likely. Of course, you want to
make sure unattended updates are enabled so your host OS is up to date.

As a result, most _other_ systems will actually self-heal from minor
compromises. Only your system (any many other PHP systems in general) will
stay compromised forever.

> It runs every 15 minutes, in the default configuration, and our usual
> turnaround time for getting a bugfix out is less than 1 hour.

1 hour from what? From someone sending email to your team? If a black hat
finds an exploit in your service, they will start compromizing the existing
web servers first. You will not find out about the bug until someone notices
something weird on their installation, and that will take way more than an
hour.

> 2\. Don't chown/chmod.

Have you actually tried this? Looking at your github, there seems to be
`src/tmp/cache` directory -- presumably at least this one should be chmod'ed?
What about others? What about plugins -- would they work fine without
writeable root?

I have tried to make other PHP packages use immutable code, and it was very
flaky and painful. You need to structure your program in a certain way to make
sure it works with immutable root -- one location for temp, another for
configs, third for the code. If your main development team uses writable code,
there is a high chance immutable code would not work well.

> It's not broken, it's a trade-off. Nothing to fix.

> The reason we do this by default is because our threat model is ensuring
> automatic updates work for everyone (including people who are not
> sophisticated enough to set up a cronjob).

Why do you say it's hard to set up a cronjob? It is simple -- your deb/rpm
package just includes file /etc/cron.d/aristrip, and voila -- you are done.
The only place where cronjob setup is hard is on these cheap shared PHP
hosting services. And I think no one should be using them -- get a $5 Linode
VM instead.

The thing is, the modern Linux systems went pretty far in terms of security
and isolation. You have multiple easy-to-use technologies for threat
protection -- docker, Apparmor, capabilities, systemd' *Path, etc.. They work
pretty well to make sure that attacker which got RCE cannot take a hold on a
system and elevate privileges.

Except your system is not compatible with any of them, by design. Because you
don't care. Maybe you should change your motto from "A Secure Content
Management System for the Modern Web" to "A Secure Content Management System
for the Cheap Shared PHP Hosting", because you can certainly do much better
for the modern web.

------
orf
After a quick peruse of their github repository I have to say there is some
dodgy[1] stuff going on.

As ever with PHP sites validation seems to be a mess. For example we've got a
`escapeSupplierName` function[2] used in some places[3], but a different regex
used when creating a suppier[4]. Variants of that regex also appear all over
the code, why not put it in a single place?

Properly escaping SQL injection is great (and something PHP apps seem to have
a continual problem with), but why did you decide to roll your own framework
for doing this? Are all of the well maintained, tested and great ones already
available not suitable for some reason? It's just your app turns into a hard
to check, inefficient[5] sludge, full of lots of code for handling framework
related things like an ORM, caching[6] (that code is practically duplicated in
a _lot_ of places[7][8][9][10]), escaping[11], mime whitelisting[12] etc and
not much about a CMS. This is where security bugs appear.

1\.
[https://github.com/paragonie/airship/blob/eb4293aee5be59e329...](https://github.com/paragonie/airship/blob/eb4293aee5be59e329015520de7e0e44c3593ee6/tools/hangar/src/Command.php#L233-L237)

2\.
[https://github.com/paragonie/airship/blob/c4c9384d4d7860738d...](https://github.com/paragonie/airship/blob/c4c9384d4d7860738d4965ed0be9f8a27c18e1ca/src/Engine/Bolt/Supplier.php#L77)

3\.
[https://github.com/paragonie/airship/blob/c4c9384d4d7860738d...](https://github.com/paragonie/airship/blob/c4c9384d4d7860738d4965ed0be9f8a27c18e1ca/src/Engine/Bolt/Supplier.php#L97)

4\.
[https://github.com/paragonie/airship/blob/95af23ca782e8ecb10...](https://github.com/paragonie/airship/blob/95af23ca782e8ecb1083d31f8614a7ca532e3207/src/Engine/Keyggdrasil/TreeUpdate.php#L354)

5\.
[https://github.com/paragonie/airship/blob/63cf0661ba21cbb3f3...](https://github.com/paragonie/airship/blob/63cf0661ba21cbb3f3bfff510854a716f4b5579b/src/Cabin/Bridge/Model/Files.php#L1055-L1062)

6\.
[https://github.com/paragonie/airship/blob/master/src/view_fu...](https://github.com/paragonie/airship/blob/master/src/view_functions.php#L646-L681)

7\.
[https://github.com/paragonie/airship/blob/master/src/view_fu...](https://github.com/paragonie/airship/blob/master/src/view_functions.php#L718-L736)

8\.
[https://github.com/paragonie/airship/blob/master/src/view_fu...](https://github.com/paragonie/airship/blob/master/src/view_functions.php#L491-L510)

9\.
[https://github.com/paragonie/airship/blob/63cf0661ba21cbb3f3...](https://github.com/paragonie/airship/blob/63cf0661ba21cbb3f3bfff510854a716f4b5579b/src/motifs.php#L18)

10\.
[https://github.com/paragonie/airship/blob/2a8a87934921ecda85...](https://github.com/paragonie/airship/blob/2a8a87934921ecda85cf6ec3cecc43800b88a200/src/Cabin/Bridge/Controller/Account.php#L222)

11\.
[https://github.com/paragonie/airship/blob/eb4638eb31510028f4...](https://github.com/paragonie/airship/blob/eb4638eb31510028f40d6b51d13f0005fc5f2d60/src/Engine/Security/Util.php#L135)

12\.
[https://github.com/paragonie/airship/blob/eb4638eb31510028f4...](https://github.com/paragonie/airship/blob/eb4638eb31510028f40d6b51d13f0005fc5f2d60/src/Engine/Security/Util.php#L81)

~~~
CiPHPerCoder
Thank you for taking the time to look through the code.

1\. Context matters: That's a CLI script used to build and sign core updates.
We bundle it with the source code so anyone can fork our project and use only
their own keys, easily.

2-4. This is a good point.
[https://github.com/paragonie/airship/issues/181](https://github.com/paragonie/airship/issues/181)

> Properly escaping SQL injection is great (and something PHP apps seem to
> have a continual problem with), but why did you decide to roll your own
> framework for doing this?

Because when Airship was started, everyone insisted on backwards compatibility
for PHP 5 and I wanted to make use of strict typing. The master branch
includes a static analyzer as part of continuous integration.

> Are all of the well maintained, tested and great ones already available not
> suitable for some reason?

Have you seen what I've done to PHP frameworks over the years?

[https://scott.arciszewski.me/open-source/](https://scott.arciszewski.me/open-
source/)

[https://paragonie.com/security/advisories](https://paragonie.com/security/advisories)

I wasn't in a hurry to pick up other peoples' technical debt, especially if
we're paying money for vulnerabilities:
[https://hackerone.com/paragonie](https://hackerone.com/paragonie)

5\. That's simply a paranoia/convenience feature. If you've got a better way
to automagically convert filename.txt into filename-5.txt if filename.txt,
filename-2.txt, filename-3.txt, and filename-4.txt already exist, I'm all
ears.

6-10. Good eye. I think that can be refactored to repeat itself less.

11\. I don't see a problem with this?

12\. I don't see a problem with this either.

~~~
orf
> Context matters: That's a CLI script.

Ahh, my bad, it's still an odd way of doing this. And it's vulnerable to
injection via the prompt parameter (not that it matters, but...). That pattern
is repeated in a couple of places like the updater.

> Because when Airship was started, everyone insisted on backwards
> compatibility for PHP 5 and I wanted to make use of strict typing. The
> master branch includes a static analyzer as part of continuous integration.

Strict typing is great, but you don't have to write your own framework to
benefit from it. Is PHP 5 not compatible with modern frameworks?

> Have you seen what I've done to PHP frameworks over the years?

> [https://scott.arciszewski.me/open-
> source/](https://scott.arciszewski.me/open-source/)

>
> [https://paragonie.com/security/advisories](https://paragonie.com/security/advisories)

> I wasn't in a hurry to pick up other peoples' technical debt.

Indeed there are some vulnerabilities found in PHP libraries in those pages,
but hardly enough to warrant your position.

You aren't in a hurry to pick up others technical debt, but you rush ahead and
create your own?

All I'm saying is you put out a post saying you designed this from the ground
up to be secure, but did you really? Is rolling your own ORM and homegrown
framework secure? Says who?

Yeah, you use modernish stuff like csp etc, but the code is still a typical
PHP spaghetti of mixed concerns and hard-to-audit flows. It's not really a
CMS, it's a NIH syndromed framework with a CMS on top of that. That's a
typical source of bugs.

> That's simply a paranoia/convenience feature. If you've got a better way to
> automagically convert filename.txt into filename-5.txt if filename.txt,
> filename-2.txt, filename-3.txt, and filename-4.txt already exist, I'm all
> ears.

It could be done in a single query, I'm on my phone so i cant write you an
example right now.

> 11\. I don't see a problem with this?

> 12\. I don't see a problem with this either.

They are both framework functions, and you wouldn't be writing them if you had
written this from the ground up to be secure IMO. It's kind of code smell.
Using a whitelist is good though.

~~~
CiPHPerCoder
> Indeed there are some vulnerabilities found in PHP libraries in those pages,
> but hardly enough to warrant your position.

Those are all my research findings. :P

> Is rolling your own ORM and homegrown framework secure? Says who?

Says the person who routinely finds exploitable vulnerabilities in other PHP
frameworks and content management systems.

There will always be things to improve. Feel free to try to find an
exploitable security hole.

> It's not really a CMS, it's a NIH syndromed framework with a CMS on top of
> that. That's a typical source of bugs.

It's not "NIH syndrome" it's "I don't trust any of these people to write a
secure framework and I have the experience to justify this concern".

> They are both framework functions, and you wouldn't be writing them if you
> had written this from the ground up to be secure IMO.

Escaping-on-output still implies that escaping happens.

~~~
orf
> Those are all my research findings. :P

Yes, and well done! But there are not many framework specific issues you've
found. None in Laravel for example. Is your homemade framework superior to
Laravel? If so, how?

> Feel free to try to find an exploitable security hole.

I can't be bothered to learn your home grown half framework so i understand
the convoluted logic. That's the point. There are vulnerabilities in there,
I've seen enough code like that to almost guarantee you. But like a lot of PHP
projects that roll their own everything it's no trivial task to do a code
review. If you had used a well made framework then you would have got rid of a
lot of code that only you really understand.

> "I don't trust any of these people to write a secure framework and I have
> the experience to justify this concern".

Your code doesn't look like well written, secure code. I'm sorry to sound
abrasive but it's true. You mix validation in random places, perhaps
inconsistently, and your project tries to do everything. This is not how
secure projects are typically designed.

> Escaping-on-output still implies that escaping happens.

It's only used in 8 places as far as i can see, in some seemingly random
places.

~~~
CiPHPerCoder
> Is your homemade framework superior to Laravel? If so, how?

Our cryptography is certainly superior to Laravel's.

[https://github.com/illuminate/encryption](https://github.com/illuminate/encryption)

[https://github.com/paragonie/halite](https://github.com/paragonie/halite)

Beyond that, nobody has paid me to look deeper and I haven't had any reason
to.

> There are vulnerabilities in there, I've seen enough code like that to
> almost guarantee you.

If so, then anyone reading this thread who's aware of any vulnerabilities will
be interested in:
[https://hackerone.com/paragonie](https://hackerone.com/paragonie)

~~~
orf
> Our cryptography is certainly superior to Laravel's...

Ok, but cryptography is only a small portion of what a framework does and not
necessarily the most important bit. I was asking more about your routing, view
layer, ORM, code layout, templating, validation, caching, ACLs etc.

> Beyond that, nobody has paid me to look deeper and I haven't had any reason
> to.

Look, you can't put up a blog post titled "How We Engineered CMS Airship to Be
Simply Secure" and say that when asked "well, how did you actually _engineer_
it to be secure?". In your post you talk about problems all solved by a
framework, and apparently you haven't even evaluated what is out there already
and rushed headfirst into building it all yourself.

This is _not_ engineering a project to be "simply secure", in the sense that
it is definitely not simple and probably not secure!

> If so, then anyone reading this thread who's aware of any vulnerabilities
> will be interested...

This is great! I wish more companies did this. However, as stated previously,
the ad-hoc nature of the code and the single-developer nature[1] makes this
pretty hard to do from a code review point of view. Again, using a framework
makes your code way easier to review and a lot smaller to boot. Anyone with
Django experience can pick up a Django app and start to review it, rather than
have to learn whatever bespoke framework the developers of the project dreamed
up.

Also I just wanted to add that I don't mean to come across as a dick, your
project looks interesting for sure and a lot of work has gone into it. PHP
CMS's _need_ to start taking security seriously, things like Joomla and random
Wordpress plugins are a blight and anything taking security seriously is
great! As someone who works in the security industry it's just a bit tiring to
see yet another homegrown PHP half-framework full of it's own intricacies and
potential issues with the ever familiar problems described in all my comments
above, coupled with a post saying how it was engineered to be secure from the
ground up.

1\. It's hard to tell, in this graph
([https://github.com/paragonie/airship/graphs/contributors](https://github.com/paragonie/airship/graphs/contributors))
is 'paragonie-security' a single person?

~~~
CiPHPerCoder
> Look, you can't put up a blog post titled "How We Engineered CMS Airship to
> Be Simply Secure" and say that when asked "well, how did you actually
> engineer it to be secure?". In your post you talk about problems all solved
> by a framework, and apparently you haven't even evaluated what is out there
> already and rushed headfirst into building it all yourself.

I've looked at Laravel, about two years ago.
[http://seclists.org/fulldisclosure/2015/Apr/57](http://seclists.org/fulldisclosure/2015/Apr/57)

I haven't looked deeper at what it's done since then, since nobody is paying
me to do so. I wasn't impressed then, and instead of waiting for them to get
around to doing things right (while figuring out how to preserve backward
compatibility), I started from zero.

You don't have to like that decision, but that's the decision I made.

> Again, using a framework makes your code way easier to review and a lot
> smaller to boot.

Sure, use a framework.

And then this happens: [https://kivikakk.ee/cryptography/2016/02/20/breaking-
homegro...](https://kivikakk.ee/cryptography/2016/02/20/breaking-homegrown-
crypto.html)

I designed my own because, unfortunately, most of the PHP community either
doesn't prioritize security enough to be trustworthy, or is simply in over
their heads.

We use third party libraries (Twig for templates, HTML Purifier for allowing
some HTML but avoiding XSS vulnerabilities) because they're well-studied and
trustworthy libraries.

However, I do not trust most other PHP developers to do anything security-
critical correctly. And thus, I will not design something meant to be secure
based on an insecure foundation.

If your only reason for Airship's status as "probably insecure" is because
it's "ad-hoc" and/or "bespoke": Sorry, that's insufficient. Got a PoC handy?

Otherwise: we're talking about pure hypotheticals that are far removed from
the real world and there's not much point in that. You can frame your
criticism as "things to make the code easier to read/understand" and whatnot,
and they stand alone. Citing them as "probably a vulnerability" is
questionable, however.

> It's hard to tell, in this graph
> ([https://github.com/paragonie/airship/graphs/contributors](https://github.com/paragonie/airship/graphs/contributors))
> is 'paragonie-security' a single person?

For the moment, I'm the only _technical_ person at Paragon. That will change,
and that account will be delegated to signing/committing the code changes made
by other employees.

> As someone who works in the security industry it's just a bit tiring to see
> yet another homegrown PHP half-framework full of it's own intricacies and
> potential issues with the ever familiar problems described in all my
> comments above, coupled with a post saying how it was engineered to be
> secure from the ground up.

You've raised legitimate concerns over:

    
    
      - Apparent NIH-syndrome, which in this case is justified
      - Minor code quality nits, which are now fixed
      - Bus factor
    

...but none of those are, directly, a security problem. :)

