When people are learning to program, I always tell them that regardless of their preference between books, videos, classes or peyote, the most important thing they need to persevere is a problem or project that they passionately want to see through to the end. This urgency translates directly to giving a shit, which is something I believe is critically missing when implementing a TODO clone in language framework X.
My hierarchy above can be viewed from another lens: in what order does having ample supply of this trait open the aperture for all of the others that follow?
Being really patient and having access to Stack Overflow can make up for a lack of skill. Doing something for a long time can occasionally make short work of a project if you see the answer quickly. However, nothing else in the chain matters if you aren't urgently motivated to reach the conclusion. When you feel genuine need to see something through, everything else clicks into place.
Great writeup! A good reminder that in reverse engineering you can get pretty far just with static analysis and a bit of educated guesswork. Also a fantastic demo of using statistics to get a foothold into how the files are structured.
If anyone is interested in some similar research, I did a video writeup using very similar techniques for reversing the graphics format of a cutesy platform game - https://www.youtube.com/watch?v=e0x7KU4wLGw
The statistics part is interesting but seems a lot of analysis work and guesses to get what's happening there - I was left with a feeling that after the initial analysis showed that the map layout isn't straightforward, it would have been easier to just look at the code touching it, e.g. take that DOS executable, put a memory breakpoint on one of these bytes with unclear meaning, and see what the code is doing with it.
(Note: I'm assuming from the reference to "map layout" that this comment is directed at the original article, and not the video mentioned in the parent comment.)
Reading the assembly language is what many (perhaps most?) people would do at that point. But not everyone is as comfortable with reading assembly, and I wanted to show that a lot can be done without taking that step.
Also, "easier" is a relative term. It was far, far easier for me to examine the data files sitting comfortably in front of my Linux box, than to try to pull together a decent debugging setup inside of a dosbox before making any headway. There's always more than one way to do it.
I absolutely loved that game! I played the Windows version when I was younger.
That guy is a mad genius. It's great knowing there are people who can figure out a two-byte compression scheme (and even know what it is to begin with) by just looking at the binary dumps.
Horrible article. It had me glued to the screen in rapt interest and I ended up losing track of time, causing me to be late for a meeting.
Actually, I have been biting at the bit to get into reverse engineering after discovering some of Chris Domas' [0]. However, my hand has been stayed simply by a lack of somebody to learn from and nerd out with on the topic. Would you be able to share any communities you are aware of in this regard?
Really though, awesome article. Thank you for taking the time to write and share it!
Thanks for the kind feedback. Unfortunately I don't know much about communities. Most people who do reverse engineering focus on examining code/assembly, but my limited experience is solely with data files.
This was a great post with references to many tricks and tools I was not aware of that might help me in similar endeavors. Do you have any particular advice for reverse engineering image data files (not encrypted, of a completely proprietary format)?
I'm afraid that's something I don't have any experience with. However, another commenter mentioned a cool video he did on just this subject, at https://news.ycombinator.com/item?id=21728298 -- so maybe check that out?
When it comes to learning these sorts processes, there is a hierarchy of related phenomena required to succeed:
motivation > intuition (experience) > patience/persistence/determination > skill (aptitude) > tools
When people are learning to program, I always tell them that regardless of their preference between books, videos, classes or peyote, the most important thing they need to persevere is a problem or project that they passionately want to see through to the end. This urgency translates directly to giving a shit, which is something I believe is critically missing when implementing a TODO clone in language framework X.
My hierarchy above can be viewed from another lens: in what order does having ample supply of this trait open the aperture for all of the others that follow?
Being really patient and having access to Stack Overflow can make up for a lack of skill. Doing something for a long time can occasionally make short work of a project if you see the answer quickly. However, nothing else in the chain matters if you aren't urgently motivated to reach the conclusion. When you feel genuine need to see something through, everything else clicks into place.