You’re effectively using https://en.m.wikipedia.org/wiki/Little%27s_law to deduce component performance as service time. I’ve found the peer filtering mechanism to work as well. It can also work if the entire node is slow, depending upon characteristics of the workload.
I think peer filtering starts working well at some medium sized number of peers x events. Off the top I guess at five peers with a steady, non-zero event rate for a few hours, a slow peer should stick out obviously in the 95th percentile.
However, op mentions that opaque workloads make peer identification difficult. IIUC this is a cool way to use two correlated metrics to extract performance outliers with some neat-sounding math that I didn’t study.
