
Reverse engineering the binary data format for Star Wars: Yoda Stories - krispykrem
http://www.zachtronics.com/yoda-stories/
======
sehugg
_I 'm not sure why there's a sports car in the game's tileset_

I have a suspicion -- it's from one of Mark Hamill's more-regrettable roles.
What a great Easter Egg! (unless it's actually part of the game somewhere)

[http://fffmp.fffmoviepostersc.netdna-cdn.com/wp-
content/uplo...](http://fffmp.fffmoviepostersc.netdna-cdn.com/wp-
content/uploads/3922.jpg)

~~~
krispykrem
Holy crap, that's absolutely the same car. I'll update the writeup with your
discovery. Thanks!

~~~
cremno
I guess the two tiles are really unused, aren't they? If they are, that's
great stuff for the _The Cutting Room Floor_ wiki
([http://tcrf.net](http://tcrf.net))! Don't get lost on there as it's somewhat
like TV Tropes.

------
roghummal
This put a big ol' smile on my face. I remember Yoda Stories (there was a
similar game, Indiana Jones and his Desktop Adventures, same engine?)

Very thorough RE of the data format. Posts like this are why I still come to
HN!

~~~
joshschreuder
Yeah the author mentions that it was probably a later version of the same
engine: _The VERS identifier clearly starts a "version" section, which
contains the following four bytes: 0x00, 0x02, 0x00, 0x00. My guess is that
this is version 2.0 of the file format, as Yoda Stories was actually the
successor to an Indiana Jones game that appears to use the same engine,_

------
Jemaclus
Every time I think "hey, I have this funky DAT file that I need data from. I
bet I could reverse engineer it." And then I open the file and my eyes glaze
over. Then I read a headline like this, and I think, "Hey, maybe this will
help." And then I realize... nope, still way over my head.

Sigh.

------
gear54rus
10/10 good article.

I wonder if something remotely similar is possible with todays games. With all
the custom data formats and compression algorithms being used.

~~~
krispykrem
Absolutely! Are there any games you'd like to see in particular?

~~~
gear54rus
Nothing in particular.

I love reading RE posts but I don't plan on making any mods to any games in
the near future. Therefore I don't want to specify any 'targets' for your
efforts unless I'm willing to extend them :)

I was talking about something big and sophisticated, like Crysis or
Battlefield (from the top of my head). I'm under the impression they have
something more than uncompressed bitmaps and 8bpp resolution but I've never
done anything like this to be certain.

Thanks! Keep up the good work.

~~~
jon-wood
The CryEngine documentation would be a good start for Crysis
[http://docs.cryengine.com/display/SDKDOC1/Home](http://docs.cryengine.com/display/SDKDOC1/Home).
Battlefield would require some reverse engineering since it runs on Frostbite,
which EA's internal game engine, and not publicly available.

------
pjc50
The four bytes of ID followed by four bytes of size is an old and somewhat
standard technique:
[http://en.wikipedia.org/wiki/Interchange_File_Format](http://en.wikipedia.org/wiki/Interchange_File_Format)

------
Ogre
The author also created SpaceChem, which is one of the best
programming/engineering games out there. For just programming games, I'd call
it my second all time favorite just behind Robot Odyssey, an ancient Apple ][
game.

This gives some great context to where SpaceChem came from. Thanks Zach!

~~~
pdw
SpaceChem is included in the current Humble Bundle, with still 10 hours
remaining, FWIW.

~~~
MiWDesktopHack
Sweet! Thanks for the tip. looks like EFF just got another donation for $6.54
USD, and thank you Zach for donating your software for charity!

------
NamTaf
I want to pick up more skills like this. I've just done something similiar at
work to break in to some event recorder files we could only view as the
application provided no utility for exporting the data into CSV (among many
other limitations). I was pretty lucky and fluked a lot of it, mainly by just
identifying a pattern and messing with the hex values of a file then viewing
what it did in the viewer application provided.

I'd love to expand my skills and try this out on a number of other projects.
Is there some good starting material which can push me in the right direction?

------
Sarkie
Great article, brings back hex hacking, I remember doing the same with Virtua
Tennis for the PC, then someone else wrote up an article after doing the same
thing.

[http://www.gamefaqs.com/pc/557900-virtua-
tennis/faqs/19110](http://www.gamefaqs.com/pc/557900-virtua-tennis/faqs/19110)

I think I made a little tool to unlock all the options and enabling players
you couldn't otherwise get IIRC.

------
jwarren
Great article. Thanks for being so rigorous with the step-by-step process and
for annotating your thoughts on each step, it's very instructive.

~~~
emillon
That's the difficult part about documenting CTFs and reverse engineering
tasks. In the past I tried to do it afterwards but it's a bit clunky. Now I
try to do it at the same time, like a journal, and it's getting better (still
not as good as OP :))

------
MiWDesktopHack
This is a really well written article; full of great, reusable tricks and
techniques in reverse engineering. Clear screenshots, humorous yet technical
content, neat results. Hats off to you, sir, keep up the good work and I look
forward to reading more of your tutorials.

------
grokys
This looks to me like the file is in something like the IFF format:
[http://en.wikipedia.org/wiki/Interchange_File_Format](http://en.wikipedia.org/wiki/Interchange_File_Format)

------
muyuu
Gameplay video: [https://www.youtube.com/watch?v=AQhFSf--
yUI](https://www.youtube.com/watch?v=AQhFSf--yUI)

------
13
Thank you for that write up, it's very detailed and showed your thought
process extremely well.

------
voltagex_
Love it. Any pointers for an easier old (or new) game to have a go at pulling
apart like this?

~~~
PublicEnemy111
I'd definitely check out Halo 2. The modding community was massive. In fact,
modding Halo 2 was what introduced me to programming :)

The maps have a very basic encryption(checksum). Map signers are all over the
internet if you don't feel like doing the work yourself. There's also many
high level tools to play with the maps. I remember taking an a vehicle from
one level and placing it in another was a trivial process. Youtube has a
plethora of videos where modders show off what they could do

~~~
voltagex_
For anyone else looking - Halo 2 is hard to get legitimately now. eBay copies
are looking like upwards of $90AUD!

~~~
jdboyd
That must be an Australian issue because on eBay and in local stores, it is
still $10ish.

~~~
voltagex_
I'm not sure - [http://www.ebay.com.au/sch/Video-Games-
Consoles-/1249/i.html...](http://www.ebay.com.au/sch/Video-Games-
Consoles-/1249/i.html?_from=R40&_nkw=halo+2+pc&LH_PrefLoc=2&rt=nc&LH_BIN=1)

Amazon has a (used?) copy for ~$17 - AFAICT it requires online activation so
this may not work well.

~~~
PublicEnemy111
I was referencing the xbox version of the game. If I remember correctly, the
pc version of the game actually allowed cheaters, which takes all the fun out
of it IMO

Im sure you could find an old xbox and the game for less than $90AUD. Or, you
could buy an xbox, flash the disk drive, download a halo .iso and burn it to a
verbatim disk. Thats what I usually do

------
twodayslate
Can you upload those sprites?

~~~
3ifbyw
[http://i.imgur.com/tcFHs14.png](http://i.imgur.com/tcFHs14.png)

~~~
cremno
I found Waldo! Just kidding, but I found Indiana Jones. I also doubt, that the
sprites on the left side of him, are part of this Star Wars game.

Edit:

I did the same for the Indiana Jones game:
[https://i.imgur.com/c1kKfV7.png](https://i.imgur.com/c1kKfV7.png)

(thanks ImageMagick's convert and montage)

Edit 2:

Well, they're part of the game according to
[http://starwars.wikia.com/wiki/Star_Wars:_Yoda_Stories](http://starwars.wikia.com/wiki/Star_Wars:_Yoda_Stories):

>Indiana Jones is featured as an easter egg in a mission, as a sequel to a
mission in Desktop Adventures. His similarity to Han Solo is remarked upon.

------
DigitalJack
[https://www.synalysis.net](https://www.synalysis.net) is a program
specifically for reversing binary files.

------
k3liutZu
That was such a great game.

~~~
Bockit
I'm stretching my memory a little but I think there was an Indiana Jones game
using what I vaguely recognised at the time as the same engine, which I have
equally fond memories of!

~~~
eg0
[http://en.wikipedia.org/wiki/Indiana_Jones_and_His_Desktop_A...](http://en.wikipedia.org/wiki/Indiana_Jones_and_His_Desktop_Adventures)
is what you're looking for.

I'll have to see later if the same tricks apply to both games!

------
throwawayHN123
the site is taken down!!!!!!!!!!!!!!!!!!

------
bob31
Thanks! :)

------
V01D-eXe
Can someone please write a C/C++/Go/Rust version as I am C# Illiterate .

~~~
jabbernotty
If I may be so blunt, I suggest you help yourself. You will no longer be 'C#
illiterate' when you have.

I'm having difficulty understanding in what way C# is difficult to read for
someone who can write in the languages that you mentioned.

~~~
aw3c2
Must be a troll or a comment on the wrong thread. I know neither language,
have no clue about programming and have no trouble understanding the logic in
the code snippets.

~~~
coldpie
It is frustrating to see tools like this written with C#, because most C#
developers have no mind to cross-platform compatibility. Often they'll build
the UI using the Microsoft-specific frameworks, which Mono doesn't support.
.NET programs also don't often run well in Wine, leaving me with a ton of work
to do to get it running on Linux.

It's not really the language that's the problem so much as the incredibly
proprietary environment in which it's used.

~~~
darklajid
This code should work as is using Mono.

But the complaint was weird in the first place?

~~~
coldpie
> This code should work as is using Mono.

Sure, this one does, but nearly any .NET program with a GUI won't. It's very
frustrating to run into useful programs that I can't run, and most often
applications that fit that description are written in a .NET language.

> But the complaint was weird in the first place?

Yeah, true. There's nothing hard about reading C#, just running it.

~~~
darklajid
You make it sound as if there were no portable (or 'non-windows') C#
applications. Stuff I care about:

[http://banshee.fm/](http://banshee.fm/)

[https://wiki.gnome.org/Apps/Tomboy](https://wiki.gnome.org/Apps/Tomboy)

[http://f-spot.org/](http://f-spot.org/)

[http://keepass.info/](http://keepass.info/)

~~~
coldpie
At least in the game-hacking-with-a-GUI space, there very nearly aren't any.

~~~
boomlinde
Wow, you're really narrowing this down now. OK, there aren't many tools in the
portable C# game-hacking-with-a-GUI space. Why is that such a great issue, and
why is it relevant to this article?

~~~
coldpie
It's just annoying, as someone who likes to dabble in that space. There's a
lot of good, interesting work done and it's frustrating that it's done using a
proprietary technology I can't use when there are tons of other options.

This article is about game hacking using C# and never addressed the issue of
portability, because game hackers who use C# never think about portability.
That was my point.

~~~
boomlinde
Maybe it doesn't address the "issue" of portability because it in fact is a
portable program. Maybe it doesn't address portability because it's an article
about reverse engineering a game. Maybe it doesn't address portability because
the code is meant to illustrate his process rather than for others to port it.
I still don't see how your criticism is relevant at all.

