

Ask HN: are you allowed to have compilers on production? - tingletech

At work we are migrating our applications from solaris on sparc to linux (SUSE/SLES) on vmware.<p>This turns out to be a different group in our enterprisey data center, and they have totally different rules than the solaris group.<p>Since the packages for SUSE/SLES seem sort of lacking and not always up to date, I have scripts to build my stack from source code.  I like to build the stack on each machine because you never know what libraries are going to be subtly different between environments.  Plus this is how I was taught to do it when I was a junior sysadmin 15 years ago in another role.<p>For "security" one rule in the linux group is "no compilers on production."  They say this is standard practice, and that we should build our stack on another machine, tar it up, and deploy it that way.<p>Does this make sense?  Is this really standard practice?  Is that really more secure?
======
amock
Building once and deploying the binaries everywhere is standard practice
everywhere I know of. If you have subtle differences between libraries on
different machines you should fix that rather than recompiling your code on
each one. Removing the compiler isn't less secure, and since they are the ones
making the rules you have to follow them. It should also dramatically speed up
your deployments when you don't have to rebuild on each machine.

I would recommend that you don't do things just because that's how you were
taught 15 years ago. There are probably some things that haven't changed since
then, but so many things are different. With virtualization you can run the
exact same image on all of your machines so that there are no library issues.
Almost all Linux distributions provide binaries rather than source code if you
use their package management systems and they work very well.

~~~
tingletech
Well, we are paying them, so we can ask for the rules to be changed. We just
need to have a business justification and negotiate the change.

I have no power to fix the fact that stage is in a different datacenter with
different admins who use different repositories for SLES packages. The two
data centers will not not accept vm images from each other.

SLES is not like most linuxes in terms of package support. And the production
datacenter does not even let us use full list of SLES packages, only the ones
in their local repo. Also, my build script works on solaris and linux so the
environment is pretty close between those two by building my own packages.

I don't need to change python/apache etc. that often, it is not like I need to
recompile for every code deploy. I have it pretty well automated on solaris.

If it is not more secure to remove the compiler in production, then I can make
a strong argument for them to change their rule.

~~~
amock
Even if you can make an argument that it is not less secure I would still
recommend being able to run the same binaries in production as you do in test.
Having a staging system that is as close as feasible to your production system
is wonderful for debugging. If I were you I'd recommend trying to get the two
data centers to use the same images. If that's not possible and all the prod
machines run the same image I'd build a package on one prod machine and
install that on all of them. If not even all the prod machines are the same
then building on each one might be the best you can do unless you want to
statically link everything.

Getting back to your original question, I don't see how having compilers in
production is bad for security except if the compilers have malicious code in
them. However, my team's policies do not allow anything on our production
boxes that didn't go through our build and deployment systems. If an attacker
can run a compiler he can probably run arbitrary code anyway.

~~~
tingletech
> If I were you I'd recommend trying to get the two data centers to use the
> same images.

Yes... we have tried... this is not going to happen. The battle I'm picking is
to get a compiler on production.

In SUSE dev and stage started as clones and production is very close to
dev/stage but built by a different group; but in solaris land things are more
divergent between dev stage and production because the openCSW packages were
all installed on different days from the main public repository which is
consistently being updated. Plus, since -dev is shared with ~30 developers and
stage is shared with about ~10 developers and folks ask the dev/stage admins
for different packages things start to diverge more as dependencies get
upgraded.

Where I have problems is trying to compile things like GD and openquicktime
that are used by some legacy processes that seem to be very sensitive to
certain libraries being at the right version -- maybe this is not going to be
an issue on SLES but I've been bitten in the past so many times by subtle
library version issues and I find I have the best luck if I always rebuild my
entire stack after any OS update or package changes.

I know it was another age (when in one day I might have to work on old SunOS,
new Solaris, HP/UX, AIX, BSDi, and IRIX boxes -- and dev and stage machines
were unheard of), but the grey beards (these dudes literally had awesome grey
beards) I learned this trick from had a lot of experience and I still believe
that building locally is the best bet unless you can guarantee the systems are
identical in every detail (which I will never be able to do because I'm not a
sysadmin anymore).

> If that's not possible and all the prod machines run the same image I'd
> build a package on one prod machine and install that on all of them.

I will only have one production vm for my app. The main application I run has
~6k unique visitors a day and ~130k unique visitors a month with an average of
4 pages per visit -- not super high load so I don't need multiple back end
servers -- at least on my shared solaris host. I haven't done load testing on
the vms yet -- but if I do need multiple production vms they will be able to
clone the vms for me in production.

------
aphyr
If you can cross-compile to this build target, so can an attacker. Maybe
removing compilers made sense ten or twenty years ago, or on
obscure/proprietary systems--plenty of exploits in old editions of Phrack use
local compilers--but I don't really see the point nowadays.

[edit] That said, standardization will make your life _much_ easier. If you've
got varying libraries between systems, tracking down problems is tougher.

