This was the question I was actually trying to answer when I found out about C-M-j (my other thread), I ended up reading the readline documentation back and forth + all the questions about it on SO etc... But to no avail.
If you have a buffer that spans multiple screen line, can you move up and down between them, like 'gv' in vim?