Doesn't AFL rely on being able to compile the application with it's hooks? It's been a long time since I've messed with Photoshop, but it's a complete mess of random dead code inside (even includes poems about a dead dog believe it or not), chances are you're looking for a completely obfuscated bunch of assembly in the hundred plus megabyte binary. I love Hopper, but not that much.
According to wikipedia [0], the code that recognizes money was given to Adobe as a binary blob. So no go even if you have acces to the Photoshop source code.
[0] http://en.m.wikipedia.org/wiki/EURion_constellation
Yes, you're right. I'm admittedly not at all an expert on this type of thing, but surely some randomised testing, starting from a real banknote and making small mutations, would be doable.
Oh sure, you could make a quick imagemagick script to make random variations and get a pass/fail with AppleScript, or even do some exploring with a debugger while opening one. Main barrier is motivation, I have no desire to counterfeit money (and who would use PhotoShop for that anyway), so I decided to let that rest.
