This (2013) article[1], when talking about "High Entropy Bottom-up Randomization" says that:
"For compatibility reasons, this feature is disabled by default and must be enabled on a per-application basis. This is because some 64-bit applications have latent pointer truncation issues that can surface when dealing with pointers above 4 GB (significant bits set beyond bit 31)."
Although OP had an issue with an offset rather than an absolute position, I imagine it would also handle that case.
If you already had this in rr, you should have been able to set a hardware watchpoint on 0x36810e3eb and just reverse-continue to the point where the relocation was applied.
Similarly, instead of doing "a few hundred stepis" to get to the call insn that jumps to a garbage address, you can let execution run to the crash point and the reverse stepi once. The "reverse" functionality parts of rr are one of its strongest features and well worth the time investment to get acquainted with them.
(I bet my own debugging workflows and habits are stuffed full of this kind of "works but is definitely not optimal" move, though, so I don't mean this as a criticism of the article.)
This is a good point, but I feel like there was a reason that I couldn’t do this. With Wine under rr, the replays were a bit tricky to get working in the first place. I think it is possible i could’ve done this still, but I first needed to find the number of events to skip for that part of the replay. See this issue:
Semi-related: Is anyone aware of a Proton neophyte's guide to fixing a Windows game? I've got a few old games that get stuck various places and I imagine that there may be a simple Windows call that needs implementing, but I don't know the first thing about how to diagnose that.
I used to work on Wine, at first as a hobby and then later as a job (for CodeWeavers). As part of that I contributed to the Wine Developers Guide. The Wine docs to help new devs are (or were) remarkably good given the enormous complexity of the project. Unfortunately "how to fix a bug in Wine" can still devolve very rapidly into:
Step 1. Learn Windows programming.
Step 2. Then learn Linux programming, the Wine architecture, X86 assembly, low level debugging etc.
The days when fixing games involved just straight implementation of simple library calls were mostly over, even back when I was doing this a long time ago. Broken apps or games are much more likely to be hitting obscure "bugs" in Wine (which can mean failure to replicate Windows bugs), or other even more subtle compatibility issues, like the one this blog post documents.
Thanks for the overview. Guess I'm not surprised that the low-hanging fruit has all been plucked, hopefully someone at Valve fixes $RANDOM_GAME for me. :-)
Because I was graduating university and needed a full time job. Around that time Google offered me a position, which came with an office (and therefore, new friends to make, an important consideration if you're 22).
This was 2006 and there were only about 5000 people in Google engineering worldwide, so it was quite a prestigious offer. Friends/family actually couldn't quite believe it because the media was full of stories about how selective they were. The company's reputation was stellar, the pay was good and so it made sense.
Also, CodeWeavers had started getting into the Mac port around that time. The main reason I worked on Wine was to try and support the free Linux desktop. Writing code to benefit Apple didn't appeal. I figured if that was the choice I may as well just go work on something fully proprietary but that worked on Linux anyway (like ... Google's web apps).
I would start by running the game from the terminal and read all warnings and errors, compare those with games that work because some of the warnings are always there but don't affect the games.
I once had a game that always crashed under wine. Instead of trying to understand the bug, I got as far as figuring out that this was a virtual function call into nowhere and then just 909090'd the crap out of it. The game then seemed to work perfectly and I couldn't even tell what was missing/different after removing a totally random function call.
> I never quite figured out how to get debug symbols to map correctly with this Wine-under-GDB setup going on, so I had to manually explore the address space to figure out what I was looking at.
Yeah I had the same problem and never solved it. In fact I don't think I even got a handle on what the problem is at all. In what format are symbols created by mingw-gcc (I think PDB??) and in what format either Wine's debugger or GCC can consume symbols (DWARF???)
I have a patch that teaches GDB how to do this here: https://github.com/JuliaComputing/gdb-solib-wine. Unfortunately, I don't really have the time to upstream this, but perhaps it's useful or maybe somebody sees this and wants to take on the upstreaming.
Awesome! It’s definitely a step in the right direction, and I’d love to see this upstreamed, though personally I have never contributed to GDB before. Do you have any idea how this is impacted by builds of Wine that are using the new MinGW-based PE modules for almost everything?
That’s the weird part: MinGW does use DWARF, embedded into PE similar to how it is embedded into ELF, with just one little non-standard trick for long section names. On Windows, GDB supports loading these symbols just fine. But if I forcibly get them to load on Wine under GDB, it doesn’t seem to work even if the load seems to succeed. Confusing.
I didn't know about GDB follow-fork-mode child / follow-exec-mode new! I always just fiddle around trying to launch the child process directly to debug it.
Ahh, sorry I never completed the story. The program was simply fixed. I believe the way they wound up fixing it was simply static linking libpng and zlib. I found another workaround too, which was a very obscure linker flag, but I can’t recall what it was as it has been a few months now. If you run into this problem with a random binary and can’t get it fixed properly, you can use Microsoft’s REBASE.EXE or EDITBIN.EXE to rebase the preferred base address to be within the range of a RIP relative CALL. If you are savvy with PE, writing such a tool by hand is very straightforward; you need to write a PE parser that can map virtual addresses, relocate as if you are a real PE loader, update the preferred base address, and write back to disk. The only case where something like this won’t work is when the executable has some kind of on-disk CRC checking, or something along those lines. (So probably, VMProtect and Themida encumbered binaries can’t be modified this way. But that’s OK, as most tools of this sort only works with Microsoft’s toolchain, which does not have pseudo-relocs.)
Out of curiosity, if the tool was compiled for MinGW doesn't that mean it would run natively on Linux? Why was your friend trying to run the MinGW version?
MinGW does allow you to write Windows-specific code. The code in question could be ported, and in fact I think it might be now, but it still puzzled us why it didn’t work on Wine. I only wanted to help on this particular avenue for the sake of making Wine better. I had figured that it was most likely an issue with Wine that could be rectified. Unfortunately, emulating Windows ASLR behavior in userspace accurately may prove to be fairly difficult, so it’s probably not worth it :) I’m not convinced pseudo relocs actually work reliably on Windows in this way, although I could be wrong.
This is great debugging but I wonder why the author immediately drops to debugging Wine directly, rather than running WinDbg/cdb under Wine and trying to debug the app via this way first. This feels a bit like debugging your webpage JS via attaching lldb to Chromium!
Actually, I do have an answer for that: debuggers don’t work very well under Wine. It would be nice to get them working better.
The standard tool to use for Wine is winedbg, but as I wrote, this doesn’t work here because there’s no way to break before the application entrypoint (which is too late.)
"For compatibility reasons, this feature is disabled by default and must be enabled on a per-application basis. This is because some 64-bit applications have latent pointer truncation issues that can surface when dealing with pointers above 4 GB (significant bits set beyond bit 31)."
Although OP had an issue with an offset rather than an absolute position, I imagine it would also handle that case.
[1] https://msrc-blog.microsoft.com/2013/12/11/software-defense-...