It's explain at the link in the first para [1]. It's the catchy way of talking about screwing with site layout and scroll performance when manipulating the DOM with onscroll events. It became a thing to look for after Google's 2013 IO and dev conferences focused and removing jank from the web and Android - both in user facing (speed up DOM manipulation) and in code (prevent jank to begin with).
Not sure why the need to invent a new word for this. Scroll "twitch" would have been a perfectly acceptable choice I would think. ie. Twitchy scrolling.
I've never heard jitter used in client side web development, so it may just be custom in this domain. That being said, claiming either one as proper seems a bit misguided
