
Application Security in a DevOps Environment - ryan_lane
https://eng.lyft.com/application-security-in-a-devops-environment-53092f8a6048
======
Bhilai
I feel security programs are all about small nuances and there is plethora of
generic advice out there. So, though this is good generic advice overall but I
would have loved to see some more details.

> With this freedom, teams have the responsibility of securing their services,
> and ensuring security issues are fixed within an established timeframe.

How does the security team ensure that security bugs are being addressed.
Individual engineering teams sometimes do not have visibility into the whole
system and may think of an issue as low severity but combined with another bug
in another system and its now a sev:high (SSRF type of things.) With
individualized team dashboards, how does the security team handle this.

> The Security Team is currently rolling out metrics to our service dashboards
> tracking out of date patches, with an alarm to page the team when security
> patches are left unapplied. Giving the teams visibility into the risks lets
> them prioritize and schedule their patching, instead of relying on the
> security team to monitor patch levels.

How is the compliance to security policies ensured ? What if a team does not
want to patch or the patch breaks their existing tools. A key element is these
faced paced dev->deploy environments is exceptions. How are those handled and
tracked ?

~~~
k4ch0w
It varies widely from company to company. It depends on how dangerous of a bug
it is as well as the exploitability; this determines the remediation period.
Usually, very severe bugs like heartbleed or front facing deserialization are
fixed within 1-7 days, whereas a CSRF in some random POST call could be 90/180
days depending on how a company treats less severe bugs.

You have a security guy assigned to keep track of bugs and is notified after
the remediation window has passed. It's usually assigned to the person that
found it because they are able to easily understand the issue and then verify
the fix. They will confirm it has been fixed for all the machines it was found
on, then close the ticket out. However, often bugs that are considered
unlikely to be exploited will be kicked back, and long chains of emails
between the security guys saying "Please fix this bug! I'm just trying to do
my job", and developers saying "We have 0 dev cycles and a release coming up,
not a high priority! Can't you just close it?"

This email thread goes on for weeks/months, and the security guys push the
redemption period to help developers meet their deadline. It gets pushed back
so many times that the security guys get sick of reporting it because they
know it's a low severity issue, but still is a problem none the less. They
push it up the chain of command and force the PM of the project sign off on
the risk associated with what could happen if exploited, they almost always
have it fixed immediately then it's closed.

------
samstave
I am constantly confounded by the fact that there isnt a devops-as-a-service
unicorn that doesnt exist.

A company who provides DOPsaaS based on supporting all these companies.

We IT/OPs people are the glue that holds these visions to profitability - have
more power than we acknowledge

~~~
e12e
[ed: TL;DR Is your service more difficult to deploy to _production_ with all
best practices, real security, redundancies and backup - than it is to build?
Then you're not doing devops.]

I don't think there's great consensus on what devops means. According to
Wikipedia[w] the original definition was:

"DevOps is a set of practices intended to reduce the time between committing a
change to a system and the change being placed into normal production, while
ensuring high quality."

Which is a bit of a non-definition if you ask me. You can have that with
traditional good system administration.

I see it as two things: the change from how a service or product is viewed, to
a more holistic, system oriented view: the acceptance that "the system"
consists of the interaction of code, configuration, infrastructure and users
(along with changing/evolving requirements, because: humans).

And that one entity needs to own the entire process; the devops team. That
team might change a bit in composition; but it means that rather than write
some "application" and ship it to the customer on a floppy disk - most useful
software is a _process_ more than it is a _product_.

The other part (which is connected) is that as software delivery is a
continuous process, it becomes natural to incorporate/inform "development"
from the ops side (12-factor apps, plan for dev/stage/prod deploys, make a
program/service that is easy to monitor, test, run and manage -- not "hexedit
configuration in this zip of executable and config data and run it with
magic_jar_executable, make sure to restart it after it's written 48mb to
disk"). And vice-versa: be more explicit in treating administration as a
traditional software project; version control, tests, immutable deployment
states with snapshot / rollback support.

There's another group that seem to think "write software and throw it over the
wall to the admins" is a perfectly sound model; all you need is better admins.

And to be clear: bad admins is a bad idea we/they _will_ break your fool-proof
single fat jar/single fat c++/go executable with resources baked in.

And good sys admins can work.magic with your broken software.

But it's not devops when the two groups are on different teams, doing
different things - throwing rocks, insults and build artifacts over the wall
at each other.

That still just "dev" and "ops" \-- apart.

[w]
[https://en.m.wikipedia.org/wiki/DevOps](https://en.m.wikipedia.org/wiki/DevOps)

~~~
windowsworkstoo
Good comment - I am a DevOps lead and have spent time thinking about this.
PErsonally, I try to promote Dev/Ops - the slash is important - you still have
dev and you still have ops but there must be a culture of co-operation and
flexibiltiy on both sides. IT is very much a people problem rather than a
technical thing

------
borski
This is a great post, and specifically what we preach to our customers. Having
data that is exportable, respects the developers' time, and accurate is
immensely important, and specifically what we focus on. We get flak sometimes
for focusing too much on developers and not enough on security engineers, but
we really do think that's the future.

If you agreed with this post, or if you're interested in chatting about DAST
for web applications or APIs (launching next month, actually thorough scanning
of APIs that isn't just throwing a webapp scanner against it), feel free to
reach out. I'd love to chat.

