Indeed you shouldn't! That's why all serious image manipulation programs have this capability built in.
Most of them also let you write your own plugins for those cases where you need more than what the interface provides.
I think going to all-code drawing is more limiting than freeing. It seems more like a case of having a hammer and seeing that everything can be made to look like a nail.
As for plugins, that's not really integrated into the workflow in the way I'm envisioning. I shouldn't have to write heavy code just to experiment with a quick artistic idea! Think web development. It should be as simple as editing a JS file and refreshing the page.