Obviously you can't check for the presence of a rootkit while under a rootkit, in the general case.
Checking environment variables wasn't part of the challenge. The challenge was likely not intended to check for rootkits, because there are a thousands of other ways to place a rootkit apart from already loaded libraries. (Why check only open files? Closed files can also contain unwanted things.)
If the purpose is to check system integrity, just check all packages. That is much easier and faster.
If there is even the slightest possibility that the system is already compromised, do it from rescue media.
Couldn't we have a nice overview of what kind of signed modules are valid in their integrity and authenticity based on cryptography?
(Also I wanted to point out that LD_PRELOAD was specifically mentioned in my comment, but it doesn't really matter, it's the lack of integrity checks across the /usr folders that are part of the problem. Glibc, $PATH, sideloaded .so files, kernel hooks...it's such a vast problem space of insecure development practices that by now we need a better OS architecture because all (old) tools down the stack rely on 100% trustable programs being installed, which after the invention of the internet is not a reality anymore.)
Checking environment variables wasn't part of the challenge. The challenge was likely not intended to check for rootkits, because there are a thousands of other ways to place a rootkit apart from already loaded libraries. (Why check only open files? Closed files can also contain unwanted things.)
If the purpose is to check system integrity, just check all packages. That is much easier and faster.
If there is even the slightest possibility that the system is already compromised, do it from rescue media.