
Ask HN: How do you continuously monitor web logs for hack attempts? - sandGorgon
what is the generally accepted best practice to be monitoring web logs for anomalous accesses ?<p>do you guys just throw cloudflare in front and forget about it ? Or do you have engineers who work like data scientists - eyeball the logs ?<p>I have heard suggestions of using a firewall - but I&#x27;m genuinely curious on how do security focus companies discover things like &quot;oh, we got attacked by Bitcoin miners from North Korea&quot;. Are there sophisticated tools that do this for you.. or is there a generally accepted practice that has evolved for even regular engineers to do this ?<p>P.S. I&#x27;m personally more interested in an API-focused answer, but I guess the same thing applies for websites.
======
tptacek
The answer isn't quite "don't bother" but it's definitely not "install these
tools and get started".

Most of the things you can easily set up to watch for security events are
looking for problems you simply shouldn't have.

For instance: you can set up fail2ban, sure. But what's it doing for you? If
you have password SSH authentication enabled anywhere, you're already playing
to lose, and logging and reactive blocking isn't really going to help you.
Don't scan your logs for this problem; scan your _configurations_ and make
sure the brute-force attack simply can't work.

The same goes for most of the stuff shrink-wrap tools look for in web logs.
OSSEC isn't bad, but the things you're going to light up on with OSSEC out of
the box all mean something went so wrong that you got owned up.

Same with URL regexes. You can set up log detection for people hitting your
admin interfaces. But then you have to ask: why is your admin interface
available on routable IPs to begin with?

What you want is an effectively endless sink for developer logs (Splunk is
fine but expensive, ELK is fine but complicated to manage), and you want to
add structured instrumenting logs to your applications in all the security-
sensitive places, like authorization checks (which should rarely fail), and
especially a very solid, very long-term audit trail of admin-like actions, so
you have some prayer of tracking insider misuse. You could instrument your SQL
queries and catch any query failures --- which should also be rare. That kind
of stuff. I don't think there's a good shrink-wrap package that does that.

But really: focus on removing attack surface, rather than continuously
monitoring it.

~~~
ivanr
I came here to add an important detail about WAFs that's missing in this
thread. For context, I am the original author of ModSecurity (but haven't
worked on it since at least 2009). I wrote ModSecurity for three use cases
that still make a lot of sense sense; they are: 1) full HTTP transaction
logging (meaning request and response bodies), 2) rule-based analysis of HTTP
traffic (largely to trigger logging and otherwise highlight anomalies) and 3)
virtual patching.

The first two use cases are all about traffic visibility, which has been, and
remains a big problem. If we're talking about improving security, the third
use case—virtual patching—is the exciting part. The story goes something like
this: you discover a vulnerability in your code but you can't fix it
immediately, for whatever reason (take your pick). In practice, it may be
weeks until the problem is fixed. However, if you have ModSecurity installed,
you can write a virtual patch for your problem, often to fix it 100% reliably.

However, it's very important to understand that virtual patching doesn't mean
just slapping together some rule that appears to be doing the job.
Blacklisting doesn't work. To properly virtual-patch, you have to fully
understand the vulnerability and the entry point to the application. You have
to carefully whitelist all parameters in that particular location. In other
words, you have to almost replicate the work that should have been done by the
developer.

Even in this context, it's not all rosy. Virtual patching requires a great
deal of expertise to do properly. It's almost always better to spend your time
fixing the actual problem. Sometimes, however, that's not possible to do in a
reasonable time frame, and that's when a WAF (according to my definition), can
be a life saver.

~~~
tptacek
Thanks for this. I admire ModSecurity! But the subtext of my comment is the
needs of typical HN people, most of whom manage applications they wrote, for
small companies. Overwhelmingly, those people should spend any time they would
spend on things like ModSecurity instead shoring up the security of their
code.

------
hluska
I assume that I suck at security, everything I have ever built is badly
compromised and I'm too dumb to know it. I'm so bad at security that evil
criminals actually celebrate my birthday.

Since I'm this bad, I assume that any monitoring tool I could install on the
boxes themselves, up to and including all my logs are compromised. I'll
usually still install monitoring tools for compliance, basic diligence and
because they make wicked graphs that impress my superiors.

Since I'm that bad, I'd rather not invest 40 hours a week combing through fake
logs unless I'm validating that I've taken care of the basic security
fundamentals for my threat level.

Rather, I like to make sure that my security fundamentals are taken care of so
that I can stop the simple attacks. If my threat/compliance model dictates,
from there I'll possibly step things up into a web application firewall (WAF).

But, through all of this, I know that there are people out there who are so
much smarter than me that the best I can do is little more than an annoyance
for them. So, again, I like to assume that I'm badly compromised. This implies
that I'm very careful about what data I keep, and I take efforts to protect my
data at rest.

------
dsl
I break into stuff professionally.

There is nothing I love more than the "just throw
{Cloudflare,Akamai,Fastly,etc} in front of it" crowd. If you are going to
deploy a WAF [1] it _needs_ to be at the edge of your datacenter or cloud
environment. No matter what fancy feature your provider wants to sell you to
fix this, there is a way to bypass it.

* Make sure you are logging centrally, and logging the right things.

* Make sure someone is looking at the logs, it doesn't have to be a full time person, but it needs to be someones job.

* Depending on how big you are, hire people to break in on some regular basis. It could be anything from hiring a cheap contractor to run a bunch of automated tools, to full on 10 person adversarial teams. They will find some stuff for you to fix, but more importantly you will know whether the first two points are working based on what you see.

1\.
[https://www.owasp.org/index.php/Web_Application_Firewall](https://www.owasp.org/index.php/Web_Application_Firewall)

~~~
BinaryIdiot
> to full on 10 person adversarial teams

Okay this sounds incredibly interesting. Are there any open source groups that
do this "for free" sorta? Not asking to get free working, asking so I could
possibly jump in and learn.

~~~
westpfelia
There are websites that let you practice. hackthebox.eu is really popular. And
there is a bit of a test to even get access to that.

~~~
dartf
Wow! Thanks for this! I "signed up" and it looks very interesting so far

------
loblollyboy
In short - security is such a big and multifaceted topic that it’s hard to
know where to start, but I think that learning Kali Linux tools would cover a
lot.

I started working at a cybersecurity company - our product is ridiculously
stupid and I try and work as little as possible. In the time I’m not working I
try to at least learn stuff. Since I’m at a cybersecurity place I sometimes
try to learn about cybersecurity. It seems that even though for profit cs
companies are making crazy dough, a lot of open source solutions out there do
the same thing. There are so many attacks - they can be anywhere from the
application level to the level of physical devices like routers having some
kind of exploit. For any type of attack there is a good or multiple good open
source tools that can be used to execute or defend (depending on what color
hat you have), and a ton of them come standard installed on Kali Linux, but
before you learn these you have to understand networks. As a python/Django dev
I didn’t really have a deep knowledge and still don’t have a deep knowledge of
this. Nor have I done more than dabble with any of these tools. But I know
that there are a lot of videos on how to use these - including this 15 hour
(!) one
[https://m.youtube.com/watch?t=1177s&v=vg9cNFPQFqM](https://m.youtube.com/watch?t=1177s&v=vg9cNFPQFqM)
\- if you were to go through that you’d probably be a security wizard

~~~
khamba
Thanks for the youtube link. Seems interesting. I am going to watch it in
full.

------
semanticist
We also use Fail2Ban like a poor man's Web Application Firewall - for a Rails
app you know you'll never have legit traffic to anything 'cgi' or 'php' or
'exe', so crafting a couple of regexps to match those and then ban them works
well.

We use CloudFlare so we feed the IP bans into CloudFlare's firewall through
their API as well as local IPTables. (We also use CloudFlare's WAF - we like
redundancy in our paranoia.) If you had more demanding requirements you could
look at mod_security for Apache or Nginx.

We also implement rate-limiting using Fail2Ban, which helps to trip the dumber
automated scans.

Every time Fail2Ban blocks an IP address, it emails our admin team with the
matching log entries and a whois on the IP address, which we then eyeball to
make sure there's nothing weird going on.

TBH, easily 90% of the IP addresses we block got caught looking for 'wp-
admin/admin.php', which is probably a good reason to never use WordPress.

~~~
eli
Are these bots causing problems like consuming excess resources? If not it
doesn't seem worth the effort to ban them.

These days I only block bots when they're e.g. filling out forms or posting
junk comment that a human has to moderate.

~~~
raverbashing
It seems you never had to deal with the amount of crap bots throw at every
minor website out there.

If you don't want your logs flooded with vuln. scanning and referrer spam, use
fail2ban or a waf.

~~~
eli
I have, in fact, been doing this for years. Why should I care about having
pristine logs?

~~~
raverbashing
Yeah, who cares about the extra disk space, the extra overhead, the waste of
resources, filtering through noise to understand some issues with the app,
right?

Isn't it lovely when every other referrer is trying to promote some online
pharmacy or testing to see if you have a wp vulnerability?

Sure you can filter it, but I'd rather deal with garbage at the source.

~~~
eli
A lesson I learned the hard way: if random bots are causing significant
resourcing issues, then you need to address that directly. And if their impact
isn't significant, why are you spending time on it? The disk space my log
files consume is trivial.

------
guessmyname
> I'm personally more interested in an API-focused answer

OSSEC [1] is my go-tool for this type of tasks, it offers a language-agnostic
interface to monitor any type of events. I used it along with my team to power
the early version of the Sucuri Firewall, which later became the GoDaddy's
security system. The project has matured a lot and is heavily tested using
millions of HTTP requests every day. We also implemented a behavior-based
algorithm to detect malicious requests, this part is not open-source but I can
tell you that it was relatively easy to integrate it with the core interface.

Splunk [2] is also a good option, slightly more difficult to configure though.
I have had the opportunity to meet some of the developers working in this
company during a short visit to their Vancouver office and was genuinely
surprised with the level of engineering that goes around this product.
Definitely, worth checking it.

NewRelic [3] while the purpose has nothing to do with security, it also offers
an easy-to-use interface to monitor events in the system. With some extra
configuration you can make it work as a rudimentary firewall and benefit from
all the great features that the product provides.

Disclaimer: I am an active contributor to the core features, but I am neutral
on my recommendation.

[1] [https://en.wikipedia.org/wiki/OSSEC](https://en.wikipedia.org/wiki/OSSEC)

[2]
[https://en.wikipedia.org/wiki/Splunk](https://en.wikipedia.org/wiki/Splunk)

[3]
[https://en.wikipedia.org/wiki/New_Relic](https://en.wikipedia.org/wiki/New_Relic)

------
e12e
What you need is information, what's easy to get is data. Generally, if you
have data on what "normal" looks like, and data on what the current state is,
you can view the difference as information that indicate an abnormal state; be
that a full/failing disk - or someone downloading all tour data via blind sql
injection.

That's the theory. I've yet to achieve this in a timely fashion, within budget
limitations - but on my list is to have another look at graylog2 for logging
(if you have the money, splunk is probably easier - but they do charge for the
value they provide).

But I'm hoping to find the time to mash graylog2 with something like collectd,
and maybe a general event collector for exceptions/stack traces. The goal
would be to have it all on a similar timeliness in a unified interface (this
spike of exceptions happened after this series of requests from there, and
after the disk went full). And ideally with a sensible way to automate
"normal" vs "exceptional" (alert on high cpu, low disk, high network bandwidth
etc).

Sure, people should read logs/alerts - but it's a challenge to filter out
irrelevant (ssh pw bruteforce against a ssh server with only key auth, Web app
crashing due to #wontfix (yet) bug) - so the relevant becomes obvious
(successful key based login from different continent - where none of your team
are).

[https://github.com/Graylog2/graylog2-server](https://github.com/Graylog2/graylog2-server)

------
gzur
You don't monitor web logs, you ship the logs to cold storage every [crontab
spec units for time :)] and monitor your hosts and network for unusual
activity.

If anything happens, THEN you look at the logs. The signal/noise ratio of the
average web host logs makes combing through them just not worth it.

There are tons of products out there - have a look at OSSEC
([https://www.ossec.net/](https://www.ossec.net/)), it's open source and free.

------
hannob
Don't.

You shouldn't care about hack attempts. There will be thousands each day. They
don't matter.

You should care about your systems not being easy to hack. Unless you're a
very high value target then "practice basic security hygiene, install your
updates, don't reuse passwords" is enough to achieve that.

------
gesman
Using Splunk to monitor traffic either via web logs or via more detailed
packet capture solutions.

Just good web logs is enough to detect all kind of malicious anomalies.

Splunk machine learning functions can cluster data and detect anomalies in
anything without rules or signatures:

[https://imgur.com/a/IFhUZuH](https://imgur.com/a/IFhUZuH)

------
linsomniac
I'd recommend only really addressing the most obvious attacks, rather than
trying to follow up on everything that even looks like it.

I once got an "emergency" call at 7am (I had been up doing something rather
late), about a machine I run attacking this guy. After some digging with him,
the machine was our public-service NTP server, and the packets were on
udp/123, in response to packets his system was sending out. Ended up with him
yelling at me and hanging up, and me shutting down our public NTP server for
good.

------
busterarm
1) fail2ban 2) log everything to an ELK stack (or your preferred alternative)
and set up dashboards and reports to feed you malicious crafted URLs 3) Reject
traffic in your web proxy based on a maintained list of hostile URLs. Ban the
source of this traffic. Don't send back 500s, just slow down and eventually
stop answering the requests. HAproxy lets you do this easily with a few simple
rules and a stick table.

------
wepple
most successful organizations ive seen don’t think about their API/app, but
rather their stuff of value: usually data. If someone lifts your entire
database, can you spot that on the way out the door? How quickly would you
notice 100% of your fattest internet pipes spewing data at some poor souls
website? You mention bitcoin; how long until you spot all your machines at
100% CPU?

------
lmeyerov
Separating this out:

1\. Monitoring logs for security -- people don't, but machines do. Initial
security comes more from segmentation architectures (bastions, whitelisting,
..) and network/host intrusion detection systems. Logs do come into play later
for app-specific rules (unpatched apps, fraud tipoffs, ...) and eventually
automated UEBAs (exabeam, ..) and analyst threat hunting (plug:
[http://www.graphistry.com](http://www.graphistry.com) !).

2\. Forensics for post-breach attribution & containment/mitigation. Let's say
something does happen (... it will.) Logs are important here.

\-- Threat intel: Known indicators of compromise (IOCs) -- IPs, URLs,
downloaded file hashes, ... -- get published. Team setup always-on correlation
rules against their internal logs via say splunk or hadoop, and if hit, they
know the threat intel of it. Or they'll get alerts from OpenDNS that an
internal device is talking to a known bad actor. That's generally pretty
shallow threat intel on the attack (ex: VirusTotal.com entries), but a start.
There are multi-step ways to iteratively correlate across these sources, but
takes effort. Attacker attribution happens here, but typically more important
is identifying further incident IOCs in case of a bigger breach.

\-- IR / DFIR: Teams care more about defense and explaining what happened
internally (incident killchain, scope, ...). Say there was a malware alert
from AV, if the dwell time was hours or the device important, teams should
investigate what else happened for the device/user/etc, or if anyone else got
hit. THAT'S where logs really shine. Small/immature teams will check
individual tools (firewall device, ...), while more mature teams check
centralized logs.

We find manual log analysis shallow in practice for a decently sized org... so
another plug for [http://www.graphistry.com](http://www.graphistry.com) for
setting up more comprehensive multi-step analysis flows for the human side of
looking through stuff like malware, phishing, and account hijacking. Tools
like ours work on top of log stores like Splunk/ELK/Hadoop/etc. (more
scalable) or bring their own DB (less scalable.) This can get pretty intense:
some companies log all the way through process activity.

At the extreme end (FBI, Mandiant, ...), companies will quarantine a device
for activities like analyzing RAM & disk images.

------
cm2187
I sleep a lot better since I use IP white lists on any port/is that is not
public facing. Central reliable location with a signed whitelist that the
servers checks regularly and adds to its firewall. No security measure is
bulletproof, particularly against zero days but that reduces the risk by
orders of magnitude.

~~~
archi42
Nice idea, but how does it improve over "have important stuff on an
administrative VPN"? To me the whitelist seems more effort (but we often have
non-static IPs at home; with dual stack lite aka native(?) ipv6 plus a private
ipv4 behind a huge, ISP-wide NAT making things even worse), and it doesn't
give you additional encryption.

~~~
cm2187
Agree. I have scripts regularly updating the whitelist with dynamic IPs every
5 min. But I don't use VPNs for various reasons.

OpenVPN on windows is very slow (unlike on Linux), it's hard to get above
5-7MB/s, which is a problem for large file transfers (and that was pre
meltdown and spectre patches). It's also hard to configure, and the server
still has WAN connectivity so it's easy to misconfigure it and get no
protection. I also found that having an OpenVPN LAN with no gateway in
addition to your default LAN on win7 tends to interfere with Chrome on the
client, which will occasionally complain that it has no connectivity.

Being a belt and suspenders guy, and only dealing with windows machines, I
found that a good solution is to combine it with IPSec, which is
straightforward to configure on windows, fast, and adds encryption when the
underlying protocol doesn't have any (SMB2). You can configure the firewall to
only use IPSec for certain ports. But it's a nightmare to interop with non
windows machines.

------
andyjpb
Marcus J. Ranum's "Artificial Ignorance" concept is pretty good. I did an
implementation and have been running it for more than 10 years:

[http://www.ranum.com/security/computer_security/papers/ai/](http://www.ranum.com/security/computer_security/papers/ai/)

Someone's also done a Python implementation:
[https://github.com/lchamon/ai](https://github.com/lchamon/ai)

...and it looks like there's one in syslog-ng Premium Edition:

[https://syslog-ng.com/documents/html/syslog-ng-pe-latest-
gui...](https://syslog-ng.com/documents/html/syslog-ng-pe-latest-
guides/en/syslog-ng-pe-guide-admin/html/concepts-artificial-ignorance.html)

------
kwi
I highly recommend you try out Sqreen.
[https://www.sqreen.io/](https://www.sqreen.io/) It's basically newrelic for
security. Takes 2m to add to your web app and it works super well. It blocked
an attack automatically on our service recently!

------
subcosmos
A tad off-topic, but one thing I've enjoyed doing in the past is giving my
attackers something to get distracted on, allowing me to observe their
behavior.

[https://www.youtube.com/watch?v=smvwW3vW5rM](https://www.youtube.com/watch?v=smvwW3vW5rM)

~~~
archi42
Wow, it's interesting to see how bad this hacker is with a shell. Even the
"better" attacker from the video you linked in the comments doesn't seem to
really know what he is doing at times.

Sadly, I feel not all hackers are dilettantes.

------
unixhero
I can not recommend watching this Defcon talk enough:
[https://www.youtube.com/watch?v=_SsUeWYoO1Y](https://www.youtube.com/watch?v=_SsUeWYoO1Y)
\- "You Spent All That Money"

~~~
freehunter
Dude I'm a security consultant for a major vendor and I can't tell you the
number of times I've

A) Found an incident while installing new equipment

B) Installed new equipment last year and came back this year because they
still got hacked

and/or

C) Ripped out their old gear and put in my company's gear after a hack because
the sales guy told them they wouldn't get hacked with our (passive, monitor-
only) gear.

You need the right tools, no doubt. But you need to understand your needs
first. Centralized logging is the first step, but without knowing what you
want to get out of it, those logs just sit there unnoticed. Even intelligence
platforms like Splunk or ArcSight, unless you know what you're looking for and
write it into rules, it's just sitting there.

Knowledge is what matters, the tools just make it _easier_.

------
mejakethomas
During my time at Wanderu we set up Filebeat on all app machines, forwarding
json logs to Kafka, and then on to ELK and Pipelinedb for
analysis/alerting/session killing/ etc.

It worked very well, as there are almost always application edge cases that
should not be blocked immediately with Cloudflare or at a FW level.

ELK was used for exploration and/or setting alerts on thresholds that we had
found/defined beforehand.

Pipelinedb was used to run continuous aggregates on a stream, augment (pop
json log lines off a stream, enrich them with Maxmind, etc) logs in
"realtime", and immediately surface malicious behavior. We'd then
programmatically kill sessions or add Cloudflare rules upstream, based on the
behavior that surfaced in a Pipelinedb continuous aggregate.

Wanderu: [https://www.wanderu.com/](https://www.wanderu.com/) Filebeat:
[https://www.elastic.co/products/beats/filebeat](https://www.elastic.co/products/beats/filebeat)
PipelinedB: [https://www.pipelinedb.com/](https://www.pipelinedb.com/)

~~~
alexland
I'm curious how you got Filebeat to push to Kafka. Did you write a custom
modification to Filebeat/libbeat?

------
kazinator
I use a home-grown solution called txrban:

[http://www.kylheku.com/cgit/txrban/tree/](http://www.kylheku.com/cgit/txrban/tree/)

It consists of several processes written in the TXR language. One for
monitoring Apache, one for sshd and one for Exim (mail server).

TXR's standard library has "tail streams": streams which automatically follow
a rotating log file.

These txrban scripts simply open tail streams on their respective log files,
use TXR pattern matching to scan for what they are a looking for, and then use
functions in the tiny txrban utility library to do the banning. Banning is
instigated by issuing iptables commands.

The banning isn't persisted, but when you restart the scripts, they
retroactively replay everything by scanning the available logs from the
beginning.

Banning events are reported via syslog. A "dry run" configuration variable
allows testing for the rules without actually modifying your iptables.

I never packaged this program into a redistributable form. It contains some
hard-coded log-file paths that work on my server, and no documentation.

------
bkovacev
Could something like poor man's primitive check work?

\- Create a custom http header and find a unique way to create/identify it. So
theoretically another layer of authorization, just like APP ID for instance.

\- Create a middleware that would scan the request and check whether it has
the required custom header

\- If not log everything and create a notification/slack/allow/deny and if
it's good let through the request.

------
bsenftner
I used to run a web service, an API, that created 3D models of people from
photographs. It was a character creation service that made likeness models of
real people from a photo. I knew we'd get a lot of hack attempts, and a
consultant recommended we get a high end Sonic Wall appliance, a $20K piece of
hardware that cleanses all network traffic to and from my data center.

Best decision EVER! We'd get hack attempts, we'd get DoS attacks to the tune
of 130K hits per hour, and that Sonic Wall shrugged it all oof like nothing.
For a while, it looked like every script kiddie was trying to hack into our
servers. No one got through, and we could care less how or what they were
doing in their attempts. We had real work to do, that did not include fucking
around with those idiots.

We also had a block list for customers from China. Not a single transaction
from China was legitimate, all forged credit card data, so we just blocked
their billions of potential customers because not a single purchase from China
was real.

------
StreamBright
For me monitoring the logs is already too late to do anything about the
hacking attempt. Usually web servers write logs after the request was served.
One approach (I have seen commercial products do it) is to hook into the web
server code and inspect every request real time before it hits the application
servers and make a decision if it is a legitimate request or not.

~~~
yani
It does take a number of malicious requests before someone can find an
oppening. If you ban them after the 1st or 2nd request, you will be safe. It
is going to be a miracle if some vulnerability scanner finds and oppening
after 1 request, not impossible but HIGHLY unlikely

~~~
hluska
> If you ban them after the 1st or 2nd request, you will be safe.

Sadly, it is too easy to get a new IP address now. I think as part of our
security mindset, we should consider it trivial to scan for several hundred
vulnerabilities using relatively unique IP addresses.

I'm not saying that you shouldn't ban traffic as it's required under several
compliance programs and it is another form of demonstrated diligence. But,
unfortunately, banning individual IP addresses only works against the absolute
lowest hanging fruit of the criminal world.

------
cat199
previously worked at a major shared webhost a few years back (which used
Apache)

we used mod_security with common rulesets + a custom list which was
reviewed/updated based on what hacks were coming in (with hacks both passively
monitored by customer base/compromised spam reports and via active scans for
malware deposited by attackers which were then cross-correlated/traced back to
application logs to find the entry point/compromise.

support staff was ~600 (including front end call center and billing people),
with probably 250 of those doing L2+ support and ~50 doing the security work
which fed into the ruleset, for a server farm of ~2-4000ish (and exponentially
more sites/accounts since it was shared webhosting). In some ways our network
could have been viewed as a giant honeypot because of all of the outdated
webapps that were being run by the customers, plus their compromised PC's
saving passwords, etc.

~~~
yani
I see a lot of mod_security misconfiguration across hostings. While I think it
is a great tool, these misconfigs make it difficult for me to recommend it.

------
jwatte
What is "an attack?" You need to define this better to get more focused
answers.

Log monitoring is good for finding bad access patterns, but can't be trusted
to find a problem as it happens.

Look for repeated errors. Hitting the same service over an over? Login attempt
failures from the same few IPs? It's either a script, or the Bing crawler.
Write application level code and network level configuration to mitigate the
problem.

How about legitimate users doing social engineering attacks on each other?
Sybil attacks in ratings or voting? Attempt to reuse mobile purchase tokens on
different accounts? Some new SSHD credential leak zero day? Using your cat
picture upload service for storing pictures of dogs? Each of these need
different approaches, and a good logging system that can change with your
needs is super helpful.

But you can't read the logs in real time, looking for bad patterns. That needs
customized analytics.

------
rebekk99
We use a combination of
[https://www.templarbit.com](https://www.templarbit.com) and Splunk.
Templarbit allows us to get real-time notifications when someone is running a
vuln scan, violates one of our security headers or is trying payloads that
look funny.

------
ChuckMcM
Fail2ban looking for web pages that don't exist on your web site (like
../wpadmin) works pretty well at getting rid of script kiddies.

I suppose you could add a honeypot page that took form submissions and looked
for any sequence attempting a CSRF, XSS, or SQL attack and ban the source
there as well.

------
hkr_mag
This is a list of what you can do for application security with Nginx (mostly
with open source tools): [https://github.com/wallarm/awesome-nginx-
security](https://github.com/wallarm/awesome-nginx-security)

My talk from Nginx conference: [https://www.nginx.com/blog/build-application-
security-shield...](https://www.nginx.com/blog/build-application-security-
shield-with-nginx-wallarm/)

Important note. Care about vulnerabilities. Not about attacks. Buy Burp
license. Run appsec training for all of your developers; it's easy while
you're small.

Disclaimer: I am a co-founder of Wallarm mentioned in preso.

------
eli
I used to spend a lot of time combing my logs with custom scripts, but
honestly I think the effort is better spent elsewhere. There're countless
automated attacks a day on a big site but they really aren't anything you need
to worry about.

------
SnowingXIV
I’ve learned that at the very least to ensure that everything is being
properly backed up and that the restore works. Testing this regularly puts my
mind at ease that even if my site, application, whatever gets destroyed,
stolen, etc I can bring it back to life in a very short time. It’s very
difficult to ensure that something is 100% secure (not even sure if possible
since the major players like Twitter make mistakes). Of course, this doesn’t
solve the issue of data being taken but that’s why it’s important to
understand the data you actually store and if you truly need to store it or if
some could be handled another way.

------
twunde
There's really no way to eyeball logs in any sort of complex environment. What
you're looking for is a system that will automatically analyze logs and/or
traffic for malicious events and alert on them. What you're looking for are
Introsion Detection Systems (IDS) and Security Information and Event
Management (SIEM). For IDS there are a whole bunch of open source solutions.
Essentially these systems ingest logs into a centralized location and run them
through a rules engine, which gets updated with new rules on a weekly or daily
basis.

~~~
jlgaddis
> _What you 're looking for is a system that will automatically analyze logs
> and/or traffic for malicious events and alert on them._

(I don't pick on you, so I hope this comment doesn't come off that way.)

What are "malicious events" and how will you know them when you see them?

I would argue that, instead, your system should automatically analyse logs for
benign (a.k.a., completely normal, expected, innocent) events, toss them out,
and leave humans to review anything that's left over.

With any new "system", it will take a bit to build up a complete "whitelist"
so that the amount of data to review is down to a manageable level but you'll
be rewarded with much more meaningful and actionable data when you get there.

~~~
amelius
> What are "malicious events" and how will you know them when you see them?

I suspect they are simply defined as "rare" events.

------
jenkstom
fail2ban, but I'm not sure that's what you are looking for. You'd have to
customize it for an API.

------
FundThrowaway
Ha I remember awhile ago doing something similar with our AWS server logs,
turned out we were always under attack 24/7, rather amusing but not very
useful apart from scaring juniors a little bit.

------
summadat
2nd'ing fail2ban, it's simple and awesome.

Beyond that, tripwire or it's modern equivalent. Traffic monitoring, "deep
packet" inspection on the network side. Run all outbound through a proxy.

~~~
tbrock
Nah, I’ve waned to use fail2ban multiple times and failed.

Unless something has recently changed it is some of the most poorly documented
software on the planet.

I’ve never been able to configure the thing or figure out what it’s doing
or... really anything besides find the homepage.

It’s also just not worth the trouble. Globally open SSH port? That’s like 1992
level Linux box in your parent’s basement stuff.

~~~
semanticist
While it monitors SSH by default out of the box, people here are talking about
creating custom filters which will monitor their Apache or Nginx logs.
(Although, that said, we also have it monitor our non-standard SSH port with
v. aggressive blocking rules, even though it's also behind a firewall. Defence
in depth, and all that.)

The documentation isn't the greatest, but this page explains the concepts well
enough and was easy to find using Google.

[https://www.fail2ban.org/wiki/index.php/MANUAL_0_8](https://www.fail2ban.org/wiki/index.php/MANUAL_0_8)

~~~
tbrock
I understand the concepts but... how do you actually use it?

~~~
semanticist
A filter matches lines in the nginx log using regular expressions. If the line
matches, it uses another regexp to extract the IP address, and then calls out
to scripts to block that IP address.

I'm not going to post the exact configuration files I use, but the GitHub repo
for fail2ban contains examples.

[https://github.com/fail2ban/fail2ban/blob/0.11/config/filter...](https://github.com/fail2ban/fail2ban/blob/0.11/config/filter.d/apache-
shellshock.conf)

That's a filter that protects Apache against the 'shellshock'
([https://en.wikipedia.org/wiki/Shellshock_(software_bug)](https://en.wikipedia.org/wiki/Shellshock_\(software_bug\)))
vulnerability, for example.

------
pilooch
Dont do this in production without backup tools but we use unsupervised
learning for anomaly detection. We're machine learnists so this is natural for
us and we originally built the system for a mega Corp and made it open source
(though not all parts are yet public). Catch is it's not real time but he
system outputs outliers every few hours or daily depending of the size of your
traffic (daily for around 50M proxy daily logs). Outliers are then reviewed by
scripts and few operators / analysts.

------
gmuslera
Computers are better at the eyeballs-on-logs tasks. You teach them what is
"normal" to be ignored and they may warn you about what is left, even the old
logwatch is pretty capable for that.

Of course, there some basic security practices. Firewall, fail2ban or similar
to stop/ban misbehaving IPs, monitoring systems metrics to detect abnormal
activity/usage (not just for intrusion detection), and maybe feeding logs to
i.e. ELK to notice new/unknown patterns.

------
crucialfelix
I use logwatch. It gives a large informative report every day. Mostly it shows
me that fail2ban and my secure configuration are doing what they are supposed
to do.

------
tegeek
There is a product called Unomaly [1] which does exactly what you wrote.

1\. [https://unomaly.com/](https://unomaly.com/)

------
HauptJ
WAFs like CloudFlare help, but stuff still gets through. Last night an obvious
brute force attack went undetected through CloudFlare. I just finished writing
a post about it: [https://hauptj.com/2018/05/08/brute-force-login-
attacks/](https://hauptj.com/2018/05/08/brute-force-login-attacks/)

------
tmaly
I use Fail2ban
[http://www.fail2ban.org/wiki/index.php/Main_Page](http://www.fail2ban.org/wiki/index.php/Main_Page)
for auto blocking.

I take an 80/20 type approach and see what types of access is not covered by
current rules, then I add a custom rule to cover that type of access attempt.

------
vectorEQ
The most important things for this are: \- A good idea of what is normal
traffic and behaviour on the web service. \- Good threat intel.

These 2 combined will make it fairly obvious what needs to be done, but
obtaining these 2 items is not a trivial task, infact, they are both
continuous processes.

------
jgimenez
I maybe have a counter-question. Once you identify an attack, what do you do?
Firewalling the attacker is an option but not really effective, the attacker
might just change IP address and keep going. We've only inconvenienced them a
bit. What else can be done?

------
sprkyco
I work for: [https://www.signalsciences.com/](https://www.signalsciences.com/)
and our tool is specifically designed for use cases like this. Let me know if
you would like more info cody-at-signalsciences-dot-com

------
strictnein
> I'm genuinely curious on how do security focus companies discover things
> like "oh, we got attacked by Bitcoin miners from North Korea". Are there
> sophisticated tools that do this for you..

I work in a Fortune 100 Security Operations Center (SOC). We have a dev team
that builds tools for our Incident Response team, Threat Intel team, etc, etc
and an engineering team to manage those tools. And lots of other teams that
specialize on lots of other things (malware reverse engineering,
spam/phishing, red team, etc). We have to build a fair amount of our own stuff
because we operate at levels above what a lot of commercial tools can do. We
have hundreds and hundreds of custom alerts that fire based on various network
traffic and sysmon activity that our threat team has written rules around.

This, of course, is out of the question for most.

So, what should you do? First, get your logs all in one place. The ELK stack
is probably the easiest way to do that. ES makes things searchable and Kibana
is a pretty good UI for a lot of this stuff.

Second, if you aren't monitoring your internal network, you really have no
idea what is going on, so look at what you can accomplish quickly with
distributions like Security Onion (which includes the ELK stack):
[https://securityonion.net/](https://securityonion.net/)

Third, there's a fair amount of Open Source Intel (OSINT) on threats. You'll
need to find a way to integrate that information and scan your logs for it.
Malware, like Coinminers, will have certain Indicators that will tell you if
they are on your network. If you're on the ELK platform, look at something
like ElastAlert. You write rules in Yaml and then set up Alerts to fire into
your Slack channel or Email you or whatever. There is no real shortcut here.

Commercial tools that can do a lot of this stuff includes FireEye's TAP, but
they definitely aren't free: [https://www.fireeye.com/solutions/threat-
analytics-platform....](https://www.fireeye.com/solutions/threat-analytics-
platform.html)

And, as mentioned elsewhere, absolutely remove any attack surface that you
can. Scan your IP ranges and domains with tools like Shodan.io,
HackerTarget.com's DNS tools (DNSDumpster.com has a nice, free DNS search),
etc and make sure you have a handle on what, exactly, you're exposing.

And, finally, if you're big enough then every single day a computer will be
compromised in some way. There is no company in the world who doesn't
experience this so you need to have a clear set of tools and playbooks to
handle that. If you can respond to most incidents in days (not weeks), you'll
be better than 90%+ of the companies out there.

Edit: clarified acronym

~~~
westpfelia
Im currently building a SoC for a fairly large company in Germany and before
that I worked for what probably by now through M&A is the largest MSSP in the
world. I dont know if I agree with you that having a MSSP is out of the
question. we had plenty of clients who only paid us maybe 2 to 3 thousand a
year if even that. Most MSSPs will charge on a contract to contract basis
revolving around log volume and estimated daily event creation. If you are
only creating an estimated .1 events per day you could very well have a rather
cheap contract.

~~~
zxcmx
If it costs 2 or 3k a year your MSSP is probably not spending more than a few
minutes a year understanding your logs. Realistically at that price the most
likely number is zero minutes.

There is room in the industry for players that aggregate logs in interesting
ways and can share threat intel and track attack patterns between customers,
but a lot of the time with MSSPs you will just find yourself paying for a
really expensive splunk instance.

Logs are fairly useless without occasional review (to at least verify you are
collecting the things you think you are) and a decent understanding of the
infrastructure they support.

I don't know your business model or capabilities though so this might not
apply in your case. Hopefully not.

But there are plenty of MSSPs out there that will ingest logs, charge out the
%]%{^{ for "monitoring" and add very little value. After the nth time you get
alerts for PHP vulns on stuff which is clearly not running it, or something
stopped sending logs months ago and no one noticed, or you find that firewalls
logs from anything not in an unpublished support list don't really get parsed
(yeah they will ingest it but just not tell you that nothing meaningful will
happen with it..)...

To be fair, monitoring by third parties is one of those things that you can't
"do and forget about it", you need shared responsibility and a clear
understanding of who is doing what. None of this is cheap though.

------
tgsovlerkhgsel
For web logs specifically, even if you had a system that would 100% tell you
what is a hack attempt and what isn't, you would probably not want to look at
the output.

The web server _will_ be hit by thousands of "hack attempts" a day. Most of
them will be directed at software you don't even use.

You care about _successful_ hacks, not random attempts (targeted attempts may
be different, but good luck finding them in the noise of web logs). Web logs
are probably the noisiest, least useful place to look for those (although they
can be useful to determine how an attacker got in/where they came from after
the fact).

The only thing that could make sense for web logs is looking for brute-force
attempts, but that is better done with application-level logging (and
countermeasures like CAPTCHAs).

Things you could look at instead are e.g. firewall/netflow logs: If your
server is making unexpected _outbound_ connections, there's a good chance it
got owned and someone is now downloading their toolkit onto it.

------
realandreskytt
SpectX is a tool that can run fast sql-like queries on top of distributed text
files. It was initially built to parse logs by an infosec team. Hope it helps!

------
xer
It's called SIEM, Security Information Event Management, there is a lot of
companies doing this and providing cheap and expensive solutions.

------
elchief
OSSEC is a nice FOSS tool for this

------
cft
Another question- how do you comply with GDPR and do that? What if a black hat
sends you a SAR request?

------
arca_vorago
ossec + syslog , how you display the data (zabbix, elk, etc) is up to you.

------
zallarak
error/exception monitoring

