This program more or less implements a debugger, using ptrace to attach to its target, suspend it, then trampoline through malloc and dlopen (the routines a program would use to load dynamic libraries on its own behalf). This isn't far from what gdb would do if you used "call" to dlopen a library.
Unfortunately, it's not race-free - if the target's already inside malloc, Bad Things will happen. A better strategy would be to invoke clone() to spin up a worker thread, somehow finagle a TLS segment good enough to fool libc and dlopen into working Well Enough, and loading the library via this thread (while the main thread is allowed to continue execution).
If you're going to do something as intrusive as attaching the debugger interface to the process, you could just check to make sure the code isn't in malloc.
Instead of malloc, wouldn't it be safer to trigger an int80 straight to the mmap syscall? You only need to overwrite a few instructions, and put them back in place when you're done. Then you have freshly allocated memory pages for your exclusive use, and I'm guessing things get a little simpler.
WinAPI makes this especially easy too, since you can bypass the allocator and just grab pages; I believe that's how Detours does it, and I know it's how our code (which is public, somewhere) does it.
Maybe, but you still need to avoid invoking the dynamic linker while inside the dynamic linker. And you don't want the code you call inside the solib invoking malloc again either.
