
Ray tracing in notepad.exe at 30 FPS - ingve
http://kylehalladay.com/blog/2020/05/20/Rendering-With-Notepad.html
======
mrspeaker
I remember doing the "use PostMessage to send a WM_CHAR" trick as part of
program I made to auto-draw in a FaceBook drawing app.

It would read a pixel value from a file, then "click" in the FaceBook app's
color textfield, send 7 delete messages to remove the #000000 color, type in
the new color value, then "click" at an X/Y position. It did it 10 times a
second.

It worked great until I accidentally focused another app while it was running:
and the 70 delete-messages-per-second got sent to Microsoft Outlook.

It took me a while to figure out what was going on as screenfuls of emails
evaporated before my eyes.

~~~
grenoire
Well, basically the same story but with Auto Hotkey scripts!

~~~
TheSpiceIsLife
Can get messy real quick!

I have:

    
    
      F9::
      Reload
      Return
    

in all my AutoHotKey scripts, so hitting F9 will cause the script to reload
and thereby stop anything running. Also handy when doing edits and you want to
reload to test.

------
saagarjha
Hacks like these are fun; but there's a number of interesting takeaways from
something like this. One is that applications supporting accessibility or
scripting interfaces are really, really important: they let other apps
interact with your program in ways you may have never imagined, "for free".

Another is that platforms have been locking down on cross-application access
for quite a while, for security reasons. Reading or writing a process's memory
is a big no-no these days, since it effectively lets you inherit the
privileges of that application, and some operating systems will prevent you
from doing it altogether no matter how much privileges you have. The latter
operating systems have also begin to lock down the APIs that give programmatic
control as well, which is somewhat sad…

Finally, when doing things like this try to see if you can find the "highest
level" to do such hackery at. Ideally, the app has functionality built-in to
do what you want, and if that doesn't work see if the accessibility APIs can
help. Then see if you can find functions in the binary to call, and then as a
last resort you can touch data structures in memory. This is because using
higher-level APIs is more likely to keep things working: you might be changing
the string in memory, but some code at a "higher level" might not expect that
and misbehave. Perhaps it cached the string's length or thinks it has
exclusive access to that memory so it overwrites your modifications. Have fun
with your hacks, but be careful!

~~~
nolok
The Valorant hack through Raspberry Pi emulated mouse* that has been going
around really shows that while you're right, ultimately those kind of things
are like DRM in that they fight a fight that they cannot win by design.

You can't stop the user (or other application pretending to be him) from
accessing the content of something while allowing him to access the content of
something, and you can't stop the user (or other application pretending to be
him) from interacting with your software while being interactive for him.

* actual mouse is plugged into the pi, pi is plugged in the computer in usb and pretend to be a mouse, video output is piped into the pi, software magic running on the pi "fix" the mouse movement to make perfect aim when need be through image analysis [https://www.youtube.com/watch?v=d1jz8qbzfIk](https://www.youtube.com/watch?v=d1jz8qbzfIk)

~~~
Polylactic_acid
There were also mice you could buy which were programmed to undo the recoil
pattern of guns in a game. The game responded by blocking the usb ID of those
mice. But there is nothing stopping them as showing up as another brand of
mice.

------
tomc1985
I am so glad this demo exists, because it identifies one of the many great
things that modern sandbox security is trying to destroy

I had a lot of fun as a kid fucking with Windows in Spy++. Oh your shareware
program disabled the "Continue" button?? Let me just go in there and reenable
it. It was like the DOM inspector but for Windows itself.

Later I got to use these skills to port a Firefox addon to Internet Explorer.
It was a contraption, and it worked. Ironically that project died because I
couldn't get the installer to work right

~~~
z3t4
You dont want coolgame app to be able to read and alter sucurebank app. the
funny thing with for example snap apps is that most want full privileges...

~~~
colejohnson66
Doesn’t Windows require Administrative privileges to access another process’
memory? I know it does to get direct access to a hard drive. Granted, it would
be nice if one had to explicitly whitelist an app that wanted to read other
process’ memory.

~~~
asveikau
You don't need admin to do it to yourself, otherwise debuggers wouldn't work.
It will fail if you do it for an unrelated user though.

Basically the permissions are enforced at OpenProcess(), and these are the
permissions you can ask for: [https://docs.microsoft.com/en-
us/windows/win32/procthread/pr...](https://docs.microsoft.com/en-
us/windows/win32/procthread/process-security-and-access-rights)

------
milesvp
I did some windows hooking/hijacking years ago when I was working on a poker
bot, and found that I couldn't get access to some text output of the poker
client.

Fun fact. The C++ hooking library I found, Detours, which at the time was the
most common (only?) library for hooking windows API calls. It was written by
microsoft research for accessibility reasons. There are 2 windows API
functions (maybe more) that will render text to the screen basically as a bmp
making it impossible to gain access to the text being written. Not sure if
this was an oversight by the MS windows group, or it was intentional to allow
for developers to obfuscate text output in various ways. Thing is, this breaks
screen readers. So microsoft got to create a library that unbreaks their API,
which I personally found very amusing at the time.

Also you should know, that the library makes it obvious it's being used if the
client has enough privileges. I don't remember the details because I never
needed to care. But it's some combination of adding a process in task manager,
and making an obvious fingerprint in memory. I think this was to appease
complaints of use by more nefarious purposes than screen readers. My
understanding is that companies like Blizzard know how to find you're using
detours, if you don't go out of your way to modify the library before
compiling it.

[https://www.microsoft.com/en-
us/research/project/detours/](https://www.microsoft.com/en-
us/research/project/detours/)

~~~
DHowett
This is only tangentially related to your post, but it appears as though
Detours is open-source!

[https://github.com/microsoft/detours](https://github.com/microsoft/detours)

------
aabajian
I'm a radiology resident. I did a similar thing with PowerScribe 360, the most
popular dictation tool for radiologists. I injected an auto-complete dropdown
and an inline spell checker. The Win32 SendMessage APIs are quite powerful if
the underlying program is built on top of Win32.

~~~
Scoundreller
Only problem is that a lot of apps are being run “in the cloud” over Citrix
these days. Suddenly all you have is a streaming image.

Sometimes you can work with the x and y coordinates if you resize each screen
each time.

Sometimes that can be very fast because you can click on the controls before
they’re even drawn on your screen.

~~~
folmar
Citrix often works so poor that _I_ click controls of program I use often
before they are drawn.

------
CivBase
I'm less interested in how he got Notepad in particular to do this and more
interested in how convincingly he managed to represent a 3D object by
projecting it onto a grid of ASCII characters. I've seen plenty of ASCII art,
but I've never even considered what it might look like animated.

I went looking for something similar that was interactive and found ASCIICKER.
It's everything I wanted and more.

[https://asciicker.com/y6/](https://asciicker.com/y6/)

~~~
pp19dd
Whoa, Nelly. I just freaked out after hitting q and e to rotate. I thought it
was just a 2d representation and ... it's not. Thanks for finding this.

------
simias
It's funny how CheatEngine seems to see a lot of use as a legit "professional"
dev tool. I personally don't use it because I don't run Windows a whole lot
but I've had a few bug reports on github where people would use CheatEngine to
investigate bugs even though the code was fully available so you could use any
full-featured debugger instead.

I suspect that some younger developers simply started tinkering with
CheatEngine before they learned proper programming so they feel very
comfortable using it instead of a regular debugger. I started programming by
tinkering with BASIC game listings so I feel a certain kinship with this
approach.

Also CheatEngine is probably easier to set up and will work regardless of
whether you have access to the source or not. And unlike more advanced tools
like IDA you don't have to worry about expensive license or pirating since
it's free and open source.

That being said if you're reporting bugs on open source projects on GH please
try to use more appropriate debugging tools, I don't know what to do when
somebody starts posting dumps of the Windows VM address space to show me how
something isn't working as it should...

~~~
mhh__
In terms of simply poking around with memory cheatengine's UI is much more
direct than (say) GDB or similar

~~~
userbinator
GDB is an absolute nightmare to use with Asm, it's almost like it was designed
to discourage it. Look up the absolutely perplexing behaviour of the
"disassemble" command[1] for one example --- the first time I encountered it,
I exclaimed "WTF!?" out loud.

WinDbg, and DOS's DEBUG that came before it, feel far more natural with such
(and the former can also debug at source-level.)

[1] "No function contains specified address." \-- when there is absolutely no
requirement for anything to resemble a function in order to be capable of
being disassembled.

~~~
saagarjha
Hit a breakpoint? My fingers are already typing x/10i. LLDB does stupid things
otherwise as well (I think it will automatically "move your cursor down"?) so
this is just the command I always reach for.

------
rodiger
I love how useless this is. Hacking for the sake of hacking. Really cool!

------
rkagerer
_It’s not possible to read data in a different process’ address space directly
(or at least, I didn’t stumble on how to do it). Instead, I first needed to
copy the contents of a page range to the memory scanner’s address space._

How about injecting your code into that process? (Either from your own DLL via
LoadLibrary and SetWindowsHookEx; or WriteProcessMemory directly into it then
CreateRemoteThread) [1]

I love that Windows lets you do this kind of stuff. Security nightmare, sure,
but it allows use cases the creators couldn't envision (e.g. I believe
UltraMon uses it to add and override window functionality). The first time I
injected a hook into another app from Visual Basic it felt like I had a new
superpower.

 _I was going to add some code to force notepad to use the fonts I wanted
(Consolas, 11pt), but for some reason sending WM_SETFONT messages kept messing
up how fonts were displaying_

For this one you could automate the menu dialogs to display the current font.
SendKeys should do the trick, then read the control values with the same
methodology already used in your project.

[1] [https://www.codeproject.com/Articles/4610/Three-Ways-to-
Inje...](https://www.codeproject.com/Articles/4610/Three-Ways-to-Inject-Your-
Code-into-Another-Proces)

------
phoe-krk
Someone use that technique to hook a WSL bash shell in there. I can't wait for
notepad to be my terminal.

~~~
sesuximo
Do it!

~~~
heinrichhartman
Actually. I just added something similar to my emacs yesterday:

    
    
        (defun sh-line ()
          (interactive)
          (let ((cmd (buffer-substring-no-properties (line-beginning-position) (line-end-position))))
            (end-of-line)
            (newline)
            (insert (shell-command-to-string cmd))))
        (local-set-key (kbd "M-RET") 'sh-line)
    

With this Alt-RET will evaluate the last line in your shell, and insert the
output.

~~~
TeMPOraL
Speaking of Emacs, shells and WSL, I recently had to shell out from Emacs
under WSL to git-bash under Windows proper, to run cmake to run MSBuild, and
get the generation/compilation output all the way back. Turns out, while
regular M-x shell-command would work, M-x async-shell-command (i.e. shell-
command with appended &) would not - it wasn't able to communicate with the
spawned process. I needed it to be async so that a compile wouldn't block my
Emacs.

The solution I came up with is a bit lower-level, looks like this:

    
    
      (make-process :name "some_name"
                    :buffer "buffer"
                    :command ("cmd" "to" "run")
                    :connection-type 'pipe
                    :coding 'utf-8-dos)
    

Two key lines are :connection-type 'pipe, which solves the I/O issues, and
:coding 'utf-8-dos, which ensures the output streamed to an Emacs buffer isn't
polluted by ^M (CR from CRLF line ending).

Now, if anyone figured out how to configure a connection to a language server
across the WSL boundary, I'm all ears.

~~~
heinrichhartman
Does `M-x compile` not work for you on windows? This is what I use to do run
build scripts async.

------
shadykiller
I once used the similar strategy to hook into DC++ used for lan file sharing
in our school. There was a chatroom game which had contest for unjumbling
jumbled words. My app would inject into the DC++ messaging pipeline, read the
unjumbled word and type back the correct answers.

It infuriated long time players whose score I beat in just a few days. It was
fun :)

Still have the source code here -
[https://github.com/shadabahmed/quizroom_hacker](https://github.com/shadabahmed/quizroom_hacker)

------
bluedino
I never got to program for classic Mac OS, but I wonder what a similar program
would look like, considering they didn't have memory protection or anything.

~~~
tambourine_man
A lot of cool hacks where possible because of that.

One very useful, was an extension that allowed you to “save” a file even after
you‘ve hit “don’t save”

------
mbreese
How common is it to be able to read/write another program’s memory space? It’s
not something I’ve ever done before, but it doesn’t seem like it’s a very good
idea from a security standpoint. Are there other use cases that I’m missing?

Still, this is a really cool hack.

~~~
pjc50
This is a great example of how security has changed from the applications
being installed by the system administrator being "trusted" and access control
focused on users, to the main threat being malicious applications all
nominally under the control of the user but with unwanted behaviour.

 _Directly_ going into another process's memory is a bit antisocial and
unreliable, so it's normally only seen in debuggers and cheat engines. Some
systems use shared memory deliberately to communicate, but that's setup
specially (MAP_SHARED etc).

What's far more common is shared code "injection": Windows shell extensions,
for example. Keyboards and input method helpers. Or OS hook mechanisms:
Autohotkey. COM/OLE. And if you want to be baffled and terrified, "OLE
Automation" which lets you RPC into Microsoft Word.

I tried searching for detailed explanations and came across this absolute
horror:
[https://supportline.microfocus.com/documentation/books/nx30b...](https://supportline.microfocus.com/documentation/books/nx30books/opolea.htm)
(how to use ActiveX from Object COBOL)

~~~
milesvp
Just wanted to point out screen readers and other disability tools are a very
important reason to read and modify another application memory. My
understanding is this was a primary motivator for the detours library. Though
I can't seem to find the MS Research blog post talking about this nearly 20
years ago, I'm wondering if there's some revisionist history that's happened
:(

~~~
saagarjha
Screen readers should not require reading (or modifying!!) a process's memory;
platform UI toolkits make this information available programmatically.

~~~
TeMPOraL
Except when they don't, or don't give enough context. A top-level comment
here[0] gives an example where text was internally rendered to a bitmap prior
to display, so a MS library was needed to hook the relevant API calls to get
at the source text. Unfortunately (in this exceptional case), native UI
toolkits aren't always as nice as webpage DOM. We'll probably start
experiencing the same issue on the Web as soon as WebAssembly gets popular
enough for people to develop UI toolkits on top of it.

\--

[0] -
[https://news.ycombinator.com/item?id=23251509](https://news.ycombinator.com/item?id=23251509)

~~~
saagarjha
I should have probably said "good platform UI toolkits" :P

------
polytronic
That's awesome! An example of true art in software engineering! Kudos!

------
gerdesj
As soon as I saw: "Sending Key Events To Notepad" I simply thought AutoIt and
moved on.

There is no denying how cool the end result is but input to any Windows
thingie can be scripted with AutoIt in a nice easy way with a very expressive,
BASIC style language that has existed for decades. You can get it to churn out
a self contained .exe.

I don't know why Windows sysadmins piss around with anything less for most
tasks on an OS that needs a GUI. You can always call into or out of PS/cmd or
whatever as needed.

~~~
83457
Would it be efficient enough to render a rotating cube at 30fps?

~~~
gerdesj
Tools for the job

~~~
83457
And that is exactly my point.

------
sesuximo
this is a great introduction to programming on Windows

~~~
bluedino
Reminds me of stuff like "Advanced Windows 95 Programming" by Jeffrey Richter

------
anonymfus
IIRC you can just use WM_SETTEXT message instead. Don't know about FPS,
probably in between of the two methods you used.

~~~
mywittyname
I'm not sure this would work. The "game" is running in a different process
than notepad and WM_SETTEXT accepts a memory pointer as input. This pointer
would be meaningless because it's coming from a different process.

~~~
netmare
Windows does automatic parameter marshaling for system messages (0 to
WM_USER-1), which WM_SETTEXT is a part of. Since the meaning of the parameters
is well-established, the system can allocate memory in the target process and
replace the pointers with meaningful values for the target wndproc.

------
j_z_reeves
I never heard about cheat engine before. Are there similar tools for other
platforms? I know you can hook GDB to any running process, but once you're in
the process, discovering what is occurring is often pretty difficult.

I'm guessing this is how a lot of the game shark cheats were discovered back
in the day.

~~~
hexmiles
for linux there is scanmem and it's gui gameconqueror, which offer the same
core functionality of memory scanning

[https://github.com/scanmem/scanmem](https://github.com/scanmem/scanmem)

------
gfaure
This tweet mentions a trick which uses Notepad as a logging sink.
[https://twitter.com/steveklabnik/status/1263190719721766918](https://twitter.com/steveklabnik/status/1263190719721766918)

~~~
steveklabnik
Here's a link to the source comment:
[https://www.reddit.com/r/programming/comments/gnazif/ray_tra...](https://www.reddit.com/r/programming/comments/gnazif/ray_tracing_in_notepadexe_at_30_fps/fr8uy2l/)

------
ygra
That post could have been a lot shorter had they known about UI Automation.
Not sure whether it's far enough, but it definitely would have tried that
probably before even using WM_CHAR, simply because you can just tell the text
box to adopt a new text.

------
gprasanth
This is severely cool. Human endeavour never ceases to amaze me. For me, the
most beautiful aspect of being human is the length people go to, to express
themselves through their creations.

------
alpb
It seems like the refresh rate of Notepad itself is lower, as one can see
half-drawn frames in the GIF. But a really cool experiment nonetheless.

~~~
kaetemi
Caused by the separate background redraw pass of Win32.

------
macca321
I've got a script somewhere that auto saves notepad. I sometimes wonder if
it's marketable

------
ebj73
This must be mentioned somewhere in the Book of Revelation, surely.

------
SakeOfBrevity
This is the stuff that keeps me on my toes, thank you bro

------
NKCSS
Stuff like this is why I love HN :)

------
imvetri
Nice work!. Thanks for sharing

