Scrollbars being hard to use is mostly because most users have an easy time scrolling (laptops use the two-fingered scroll, mice all have buttons or touch-sensitive surfaces), and scrolling requires only mouseover not focus (try it! scroll the window behind you!). I've almost never used a scrollbar as an input mechanism, only output to see where in a document I am.
