Hacker News new | comments | ask | show | jobs | submit login
Using /proc to get a process' current stack trace (ops.tips)
190 points by cirowrc 4 months ago | hide | past | web | favorite | 66 comments

FWIW, the equivalent in FreeBSD is 'procstat -kk $PID' Eg:

  % procstat -kk 5592

  PID    TID COMM                TDNAME              KSTACK                   
 5592 103222 less                -                   
mi_switch+0xe1 sleepq_catch_signals+0x405 sleepq_wait_sig+0xf _cv_wait_sig+0x154 tty_wait+0x1c ttydisc_read+0x1f2 ttydev_read+0x64 devfs_read_f+0xdc dofileread+0x95 sys_read+0xc3 amd64_syscall+0x369 fast_syscall_common+0x101

procstat can also do interesting things, like show current rusage state:

  % procstat -r 5592
  PID COMM             RESOURCE                          VALUE
 5592 less             user time                    00:00:00.010805
 5592 less             system time                  00:00:00.002444
 5592 less             maximum RSS                             3172 KB
 5592 less             integral shared memory                   192 KB
 5592 less             integral unshared data                    80 KB
 5592 less             integral unshared stack                  256 KB
 5592 less             page reclaims                            199
 5592 less             page faults                                0
 5592 less             swaps                                      0
 5592 less             block reads                                4
 5592 less             block writes                               0
 5592 less             messages sent                              0
 5592 less             messages received                          0
 5592 less             signals received                           0
 5592 less             voluntary context switches                59
 5592 less             involuntary context switches               0

Also you can easily get both kernel- and user-level stack trace with dtrace.

BTW, I totally suck at formatting stuff in various forums. Is there a way to post pre-formatted text here? Eg, like triple back-ticks will do in slack? ``` stuff... ```

Prepend with four(?) spaces.

Thanks! I cleaned up the above based on this..

If you are running java instead of a c program the proc stacktrace shows you just the virtual machine state. You can still get a stacktrace of your java threads[1].

How about other languages? Python, ruby?

[1] https://stackoverflow.com/questions/4876274/kill-3-to-get-ja...

> If you are running java instead of a c program the proc stacktrace shows you just the virtual machine state.

No, this is a _kernel_ backtrace: what is happening in kernelspace on behalf of your process. If the work is being done in userspace (that is, in state R; the thread isn't in a syscall or page fault handler), you'll see essentially nothing here. I just tried it on a userspace busylooper and got this:

    [<0000000000000000>] exit_to_usermode_loop+0x57/0xb0
    [<0000000000000000>] prepare_exit_to_usermode+0x20/0x30
    [<0000000000000000>] 0xfffffffffffffff
Java, Python, C++ nothings all look pretty similar.

If you want a userspace stack trace, you need a different tool. If you're using an interpreted (or perhaps JITted) language, yes, you probably want something language-specific.

Also note the current stack trace is a per-thread concept, not a per-process one. If you're looking at a multithreaded program, you want to target the thread(s) of interest with "/proc/<PID>/task/<TID>/stack".

I wrote something that will get you the python interpreter stack from any running cpython process : https://github.com/benfred/py-spy/ , and rbspy can do the same for ruby https://github.com/rbspy/rbspy

This is about the kernel's stack trace during a system call, not the user space stack trace.

Yes, and if you want to know the user space stack, you may be able to attach gdb to the process: https://stackoverflow.com/questions/2308653/can-i-use-gdb-to...

Also `jstack` from the shell

Nicely formatted site. Anyone know if/what static site generator/theme is being used? Couldn't find anything on the footer, site tags, or his GitHub that would reveal that...

Agreed! It looks like Hugo and possibly this theme: https://themes.gohugo.io//theme/cocoa-eh-hugo-theme/blog/exa...

Note, the GNU debugger (gdb) can attach to running processes. This should give you a stack trace with readable addresses, cf. https://stackoverflow.com/questions/2308653/can-i-use-gdb-to...

The /proc interface tells you what the process is doing in the kernel (eg, after a system call), which is orthogonal to getting a user-space stack trace. Both are helpful. The userspace trace is arguably more helpful. The nice thing about the kernel trace is that you're likely to have symbols (or be able to download them); that is unlikely to be true for userspace if you're running a commercial binary, for example.

That's a cool tool.. I did that manually many times and always thought I should write a tool. Glad to see it exists already :)

You may want to check the original topic on that thread ;)

I want this capability for embedded ARM systems. I should be able to call a function to have the current stack trace printed:


The most widely distributed embedded ARM system software in the world, Android, offers this. They use mini debug info (normal debug info sections compressed, IIRC) and then you can signal a daemon in the background to do a stacktrace on your application.

Not on production images, though.

> The most widely distributed embedded ARM system software in the world, Android

[citation needed]

Also, when talking about embedded systems, people aren't normally talking about full Linux distros.

I've done it, but it increases binary size by a whole lot.

What worked better was a last chance hardware fault handler that wrote a partial core dump out to flash, and a tool to create an ELF core dump file from that that GDB can accept.

This is a very well written and formatted article... I found it easy to read!

Instead of teaching people how to do this portably across all UNIX-like systems, by sending SIGABRT to the process, the article is steeping them in GNU/Linux only way of doing things. This feels exactly like the '90's of the past century, where a lot of people with computer-related careers had no idea that there were other operating systems and other ways of doing things (better): an intel-based PC tin bucket with Windows was the one and only truth for them. Now it's exactly the same except Windows has been replaced with GNU/Linux. 28 years later and the only advancement some people have made is running the proverbial sed 's/Windows/Linux/g'.

That's a very strange response. This is an article about Linux specifically. The author tags the article with Linux and mentions that it's part of a series of articles about Linux. It's not in and of itself a bad idea to have articles that specialise in a specific unix flavour.

Secondly, SIGABRT will cause the process to abort and dump core, will it not? That would give you a userspace stack trace (if you load the core into a debugger) whereas this is how you get the kernel-side stack trace of a still-running process.

I don't know of a way to get the kernel stack trace of a process in a cross-platform way. Is there such a thing?

SIGABRT in most places is not going to both get the stack trace and keep the process running. In fact, I would argue that if you are running something that continues working after it receives a signal telling it to /abort/, it's a bug. What do the word abort mean to you?

As for SmartOS, as someone who ran it at home and in production for years: Keep flying that flag, I guess. I liked it. But I also realized that Joyent, even with the Samsung acquisition, does not have the resources to keep it going in any meaningful way for anyone beyond themselves and people who have the exact same usecase as them. Things like lx branded zones are clever. I miss SMF, and am not a fan of systemd. But bpf is better than dtrace. Container management is easier than zone management. I've got less bugs dealing with KVM on Linux than I ever did on SmartOS. I spend less time compiling things from source and having to find random patch files to make things work. I know plenty about Solaris and SmartOS and HPUX and AIX and the BSDs and I don't think anyone is making the incorrect choice in deciding to learn Linux over any other UNIX-like.

That ship has sailed, man. And there's no compelling reason that it shouldn't have.

"Container management is easier than zone management."

Linux has no containers, cgroups aren't anything conceptually close to zones. There are 56 different solutions to virtualization on Linux, all competing, mainly because everyone there is still flapping on deployment and lifecycle management. We'll just have to disagree and vehemently at that.

As for Joyent having no resources: you debug it and fix it yourself. That's precisely how Linux got to be the hegemony that it is today. Oh, how quickly we forget, how short our memory is...

Using SIGABRT gets you a stack trace in the same manner as a burnt house excuses you from sweeping the floors.

> Instead of teaching people how to do this portably across all UNIX-like systems, by sending SIGABRT to the process

Sending SIGABRT doesn't do what the article is talking about, on any UNIX. Perhaps you would learn something from listening to the kids these days, like the distinction between a kernel stack and a userspace stack.


Can you post a transcript of doing so and getting the kernel stack? On not just Illumos but other UNIXes, to demonstrate that it is portable?

This mainly about grabbing the kernel stack trace. There's not a portable way to do that.

Also, doesn't SIGABRT generally kill off the process?

A kernel should always be compiled with symbols / source code inside of it -- that the Linux kernel doesn't have full support for CTF says more about it than it does about CTF.

Yes, a SIGABRT will get you a core file and will kill a process, but if your process is hanging in an endless loop (like the author's was), one already has far bigger problems, and keeping such a process running will not amount to much.

I can't think of a single OS that gives you kernel stack traces on SIGABRT.

Can you disambiguate CTF? I only know that as capture the flag, which doesn't really make sense here.

Edit: ok, I figured it out.

No theyre not going to give you that raw information because it's a kernel ASLR bypass. You can totally get all the same information with the dwarf symbols, but you're going to have to opt in on a kernel for it to mean anything.

Edit2: bitching about how people aren't using portable Unix techniques, and then citing features that are Solaris specific isn't a great look.

Tell us what you figured out…

Haha sorry. It's a Solaris version of DWARF or pdb style metadata for their kernel.


For that to be useful, you'd need the addresses in question, which is why it'd be an ASLR bypass. The kernel needs to give higher level information by default so it can sanitize the output.

And the debugging symbols exist on Linux anyway, they're just DWARF (which ironically is the more standards compliant way as opposed to CTF).

CTF is just a specially compressed format and specification of embedding the source code into the binary; it's a complement to DWARF format, not the format itself. The documentation you cite even says it's similar, meaning that it's not DWARF.

The main purpose of CTF is to provide a succinct representation of the graph of C types used in a program. It's generated from DWARF; anything encoded in the CTF section can also be found in DWARF info. CTF has nothing to do with "embedding source code" and isn't useful for stack unwinding or symbol resolution.

And that's what I said: "DWARF or pdb style metadata".

It's not a Solaris specific feature; FreeBSD has it as well:


Linux is the one lagging behind, as usual.

How is embedding source code an "ASLR bypass"?

OK, it's a dtrace thing that got ported to FreeBSD with dtrace. That still doesn't make it "Unix" any more than Linux's proc access.

Embddeding debugging information in a way that allows to you to use it (ie. in a way that you'd care about the format as an end user) implies giving you kernel addresses, which implies an ASLR bypass. If it's an implementation detail, then DWARF works great.

And if "FreeBSD has it too" is our standard for portable UNIX features, then /proc counts thanks to linprocfs.

/proc on Linux wasn't implemented like /proc is on other operating systems; it's the only /proc implemented that way. The interface, if it could even be called that, is completely proprietary to Linux; output is ad hoc with no consistency. As usual.

Again, which bypass? If you attach to a process you will see machine code and it will be stored at memory addresses.

Could you write a blog post in response that describes the better way to achieve this? There are possibly related items which you could throw in for others on achieving better portability.

Sounds like a good post for Hacker News! If you find it or write it, please go ahead and submit.

Yes, it feels strange to hear millennials talk about UNIX, when they are actually talking about GNU/Linux, and many things don't apply to e.g. OS X, Aix or many other variants.

...the article doesn't say unix at all with the exception of when the go code at the end is in fd_unix.go in it's source trace.

I was supporting the parent post about doing stuff in Linux only way, instead of UNIX in general.

There isn't a UNIX way for grabbing the kernel stack trace.

Even if that were the case, a free software hegemony is surely better than a proprietary Microsoft hegemony.

Write for yourself -- I've no problem paying Apple Computer for a solution which I plop down on a table, turn on, and start using immediately after that without having to fiddle with it.

They regularly fuck up security so much that you aren't getting what you pay for. In the last major release they both allowed anyone to login as root by just not supplying a password, and writing the FDE password on the disk in plaintext as the "hint" instead of the actual hint.

Umm, what is the alternative O/S that has complete hardware support?

What is the non alternative O/S that has complete hardware support? fwiw openbsd / freebsd have been painless to install on every machine I have tried in the last 5 years.

How's the support for bluetooth on openbsd these days?

Also, how many of the aforementioned bsds use drivers that were ported from the Linux kernel tree?

BSD user here. Of course there is porting occurring, but this is the nature of open source, is it not? Oftentimes, the BSD devs will take whatever hardware specs they can get and code something from that. OpenBSD frequently writes drivers that are orders of magnitude smaller than comparable Linux drivers, while getting the same functionality. NVIDIA drivers are but one example. Some hardware vendors are generous with hardware specs for OSS, most are not. OpenBSD, as an example, will not have any GPL'd code in the base OS, as it's not truly free. Ports are another issue.

> will not have any GPL'd code in the base OS, as it's not truly free

That's a disingenuous way to put it. The GPL is a free software license. It is "free" as in libre by all common definitions of "free" in regards to software.

The GPL is not compatible with the BSD dev's preferred license, which is much more likely the reason they avoid using such code.

By all conventional definitions, the BSD and GPL licenses are "truly free [software licenses]."

You're welcome to argue that the BSD license is better (because, for example, it lets sony create derivative playstation OSs without providing that source code to their users, ensuring their users have less freedom than if it were linux) or that the GPL is better (because it would prevent the previous), but they're both free licenses.

> will not have any GPL'd code in the base OS, as it's not truly free.

"My freedom to lock down someone else's code is more important than my user's freedom to have access to the code."

No idea about other BSDs, but in case of FreeBSD that’s generally just Intel and AMD graphics drivers. All the rest is native.

And this goes both ways - at one point Linux folks famously managed to violate the BSD license when porting some WiFi driver, iirc, which is notable, since it’s not that easy given the licenses nature :-)

I dont understand how ports help / hinder usability of drivers. Could you please elaborate?

They don't. However, they do reduce the ability of the target O/S of the port to claim that it isn't a clone of Linux. The top-level post was complaining that people don't consider non-Linux O/S, and then failed to offer compelling alternatives.

I did not, I stated that nothing has changed: before people only knew PC buckets and Windows, now PC buckets and Linux is the only truth and religion. Nothing else exists. So the more things changed, the more they stayed the same. That's progress!

I also listed several alternatives just a few comments below, but apparently you didn't read that. You know of the old saying "you can bring a horse to the water, but you can't make him drink"?

Any illumos-based operating system: SmartOS, OmniOS, Tribblix, or any BSD-based one: OpenBSD, NetBSD, FreeBSD.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact