Because pictures are helpful, but neither necessary, nor, in most cases, sufficient. Just look at the use of pictures in science and technology generally (e.g. circuit schematics, Feynman diagrams), and try to imagine understanding what is being presented without reading the words.
I have seen examples where pictures can help a lot (I am thinking of a pictorial demonstration of the Pythagorean theorem, for example), but I am not aware of a constrained pictorial formalism that is both general and expressive enough to do the job of a programming language (directed graphs may be general enough, but are not expressive enough; when extended to fix this, they lose the generality.)
>The complete dismissal of visual programming by text-based programmers is often infuriating.
I, for one, would be thrilled to see an example that actually works. Arguments that it should work don't count for much, otherwise.
There are some hybrids that are pretty useful in their areas of applicability, such as state transition networks, dataflow models and Petri nets (note that these three examples are all annotated directed graphs.)