That's very similar to what Qt already provides (QPainter, for raw image/windows, more sophisticated widgets with event support. Further- and this is something I've been looking for a while- they have a graphicsview that allows resolution-independent rendering of millions of objects, with events, in an larger-than-window graphics plane.
