
Profiling Go Applications with Flamegraphs - ingve
http://brendanjryan.com/golang/profiling/2018/02/28/profiling-go-applications.html
======
emfree
A nice writeup, thanks. There are a few variations on this workflow that I've
found useful in practice; perhaps they'll be helpful to some folks:

\- Linux perf can profile unmodified Go programs. This is handy when your
application doesn't expose the /debug/pprof endpoint.
([http://brendangregg.com/FlameGraphs/cpuflamegraphs.html#perf](http://brendangregg.com/FlameGraphs/cpuflamegraphs.html#perf)
has detailed instructions)

\- Recent versions of
[https://github.com/google/pprof](https://github.com/google/pprof) include a
flamegraph viewer in the web UI. This is handy when you want a line-level
flamegraph instead of a function-level flamegraph.

------
Buge
Does anyone else have this problem with go indentation:

    
    
        func (sc *SimpleClient) Timing(s string, d time.Duration, sampleRate float64,
            tags map[string]string) error {
            return sc.send([...]
    

The wrapped parameter list is indented to the same level as the function body.
Is this how it's supposed to be done?

~~~
flashmob
No problem with that indenting here (personal opinion), I guess makes it
easier to read if your editor doesn't have soft-wraps. However, generally, if
the parameters list overflow to another line like that, it becomes a good
candidate for refactoring!

~~~
farnsworthy
Yeah, it could be a new struct, especially if another parameter or two comes
along (though the simplest thing has its benefits, too).

------
squiguy7
I would have liked to have seen how the author measured the hot spots in the
graph. It wasn't immediately obvious to me that the `WriteToUDP` function was
taking up a significant portion of the call.

Another thing that is omitted is what type of buffer was used. I'm assuming it
was a `[]string` slice but why not use a `sync.Pool`?

------
ronnier
I’m trying to figure out why I find Go so hard to read. Am I mostly alone with
this?

~~~
adamkl
It’s considered idiomatic in Go to use very short variable names [1], which
seems masochistic to me. Every time I look at Go code, it’s like I’m back in
high school using Borland Turbo C all over again.

[1]
[https://github.com/golang/go/wiki/CodeReviewComments#variabl...](https://github.com/golang/go/wiki/CodeReviewComments#variable-
names)

~~~
carbocation
I've mostly interpreted that to use single character names for method
receivers and loop variables. I think those are defensible because you know
where to find them if you need their definition (in the method definition or
in the loop).

I don't think that assigning terse names (such as the variable "ra" from the
OP) to variables that might be declared anywhere in a function is helpful.

~~~
nahname
Though that would still be idiomatic go code. For what it is worth, I do agree
with you. Short variable names on a function are a hassle. You have to lookup
what they mean, which slows down understanding.

n = copy(p, b.buf[b.r:b.w])

Oh better go to the top and see what those are again...

[https://github.com/golang/go/blob/master/src/bufio/bufio.go#...](https://github.com/golang/go/blob/master/src/bufio/bufio.go#L190)

~~~
dilap
I find to understand a piece of code, I need to know what all the involved
variables are, which I'll have to do regardless of long or short names.

Once I know what they all are, I find it's easier for me to track them through
the function if they have short names.

Something java style like

    
    
        numBytesWritten = copy(destinationBuffer, reader.buffer[reader.readPosition:reader.writePosition]
    

is much harder for me to follow.

------
gnufx
How are flame graphs a better way of displaying profile data than viewers like
paraprof [1] or cube [2], which have tree/flat and inclusive/exclusive views,
as opposed to essentially just an inclusive tree view? Consider seeing the
effect of something called all over the place in a complex system.

[1] [http://www.vi-hps.org/upload/material/tw-score-p/vi-hps-
tw-s...](http://www.vi-hps.org/upload/material/tw-score-p/vi-hps-tw-
scorep-05_ParaProf.pdf) [2] [http://www.vi-hps.org/upload/material/tw-
score-p/vi-hps-tw-s...](http://www.vi-hps.org/upload/material/tw-score-p/vi-
hps-tw-scorep-04_Cube.pdf)

------
LaFolle
Flamegraphs are good for CPU profiling (as far as I've used them), I've used
them to debug an issue when one of our programs was "consistently" consuming
high CPU. As the "stack" was mostly on CPU, during sampling, this stack was
picked up most of the time, hence in flamegraphs, it was represented with a
relatively longer bar.

But do they help to debug issues related to CPU spikes? I have my reservations
because they are generated using sampling, and flamegraphs won't give a clear
insight into who is causing the spikes.

~~~
LaFolle
One way to solve this I can think of is, if cpu spikes are deterministic, then
we can take 2 profiles: 1\. span consisting of no spikes 2\. span consisting
of at least 1 spike

And then "diff" them to get stack causing spikes.

Is this the correct way?

In general, how do people solve CPU spikes issue?

------
oneweekwonder
Whenever I see something like this I think of the 2015 video netflix[0] gave
also using flamegraphs but for node.

Personally I want something like this for py.

In searching for the vid I see netflix also have a medium post(2014)[1] about
it.

[0]:
[https://www.youtube.com/watch?v=O1YP8QP9gLA](https://www.youtube.com/watch?v=O1YP8QP9gLA)

[1]: [https://medium.com/netflix-techblog/node-js-in-flames-
ddd073...](https://medium.com/netflix-techblog/node-js-in-flames-ddd073803aa4)

------
malandrew
Video of this in action:
[https://www.youtube.com/watch?v=aAhNDgEZj_U](https://www.youtube.com/watch?v=aAhNDgEZj_U)

