Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Do you have any suggestions for a more robust method for anomaly detection in seasonal time series data?


Every case is different (as the article says, generalizing these things is not easy) but the basic approach always involves two things:

1) Build a model of the expected time series. There is no way of avoiding this. To find an anomaly you must define "that which is expected", either in terms of the actual data, differences, or moments.

2) Measure the distribution around the expected values based on past data.

3) Apply some test that answers some version of the question, "What is plausibility of the belief that the new data are drawn from the combination of model plus distribution?" The trick here is that you aren't interested in all anomalies, just "significant" ones, which may have different temporal behaviour, etc.

The important thing is to test relative to the distribution you actually have, which is never going to be particularly normal, especially in the wings, which you are going to really care about when attempting to be maximally sensitive to real anomalies. Normal distributions almost always underestimate the tails, which makes them prone to false triggers, which another poster here has pointed out: you really do not want.

Robustness against "unknown unknowns" in the anomaly distribution is one thing you want to be particularly careful about. Weird things happen all the time in real data, and you are generally looking for anomalies of a particular kind, with particular characteristics. The ideal anomaly detector will catch those without going off on every odd thing that happens.


Don't underestimate how far you can get with simple Holt-Winters exponential smoothing to generate a forecast and then note deviations from same. With the Taylor (Taylor 2010) mods to same you can get up to three different types of seasons.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: