If it was hard-to-hack then I would expect (at least) the following:
* Output messages can't be discovered using "strings"
* Program is self-encrypted
* Password isn't even stored, just hash result.
(And I consider myself an amateur at this kind of thing).
Like you said, if you really wanted to write a hard to hack binary, just use a strong hash without the plaintext on a hellish password. Heck, just leave the hash in the strings output :)
ie I would have expected to see 'ptrace', 'scanf' and 'printf' in the strings output, but they must be obfuscated in some way (otherwise I guess there's no point using the dlopen/dlsym trick at all.)
I only see one call to dlsym (at 8048506), so it seems to me the program is doing something tricky to build each symbol name string and then calling a routine there to dlsym() it.
That's about where my x86-fu fails me, though, and I remember I should be working on other things. :/
80484e0: 83 f2 55 xor $0x55,%edx
>>> def ascii_to_xored_hex(s, xorval):
... return ''.join(['%02x' % (ord(c) ^ xorval) for c in s])
>>> ascii_to_xored_hex('ptrace', 0x55)
>>> ascii_to_xored_hex('printf', 0x55)
mrj10@mjlap:~/Downloads$ xxd hackme | grep 2521
0000680: 008d 7600 2521 2734 3630 0090 2636 343b ..v.%!'460..&64;
mrj10@mjlap:~/Downloads$ xxd hackme | grep 2527
0000690: 3300 6690 2527 3c3b 2133 0090 6afb 4c8d 3.f.%'<;!3..j.L.
To the disassembler, these strings look like and-xor sequences. e.g., for 'ptrace':
8048684: 25 21 27 34 36 and $0x36342721,%eax
8048689: 30 00 xor %al,(%eax)
I have a hardcopy of Paul Carter's "PC Assembly Language" book at home that I started reading once but never finished. Some day... :/
A few friends also (unsuccessfully) tried in parallel to get the password and they were using IDA (http://www.hex-rays.com/idapro/) but I have not personally tried it. It seems like a good option (although it is not open source, which irks me a little :P).
I also tried to use an existing ASM-to-C decompiler called Boomerang (http://boomerang.sourceforge.net/license.php) but the output was a complete mess to understand (and compile). Maybe I'll try writing one of these when I'm bored on another lazy Friday :)
Any other (preferably open) recommendations for Linux?
The REC decompiler (http://www.backerstreet.com/rec/rec.htm) isn't horrible. For simple stuff, it'll give you reasonable looking C-ish code. For anything slightly more complex, it may produce wrong code. It's not so good at eliminating duplicate variables, but manually removing them isn't hard, they're easy to see.
I've recently been reversing a few stripped DLLs on Windows. REC worked well on the short functions but severely changed the logic of a few more complicated ones, especially doing bit shifts, concatenating bytes, and doing complex loops.
I've seen IDA. I'd love to use it but it's expensive and I don't reverse engineer enough to justify asking for the company to buy it. That, and I'd also have to learn how to use it effectively, which would add time and possibly stunt me learning the basics first. Since I'm most certainly doing my work for commercial purposes, the demo / educational versions of IDA aren't usable for me (license agreement says so).
EDIT: REC studio does not appear to be free (as in speech) software but it is free (as in beer) to use for most purposes and it runs on Windows / Linux / Mac.
Not sure what architectures that version supports outside of x86 (the latest commercial version does ARM and the Advanced variant does MIPS.)
But IDA is probably the best you can get. Haven't had a look at the new versions, but even the freeware version is useful (on windows at least)