When I write code, for example a layout algorithm for a set of gui elements, I visually see the data in my head (the gui elements), then I run the algorithm and see the elements 'move' into position dependent upon their dock/anchor/margin properties (also taking into account previously docked elements positions, parent element resize delta, etc). This is the visual I need to see on screen! I need to see my real data being manipulated by my algorithms and moving from A to B. I expect with this kind of animation I could easily see when things go wrong naturally, seeing as visual processing happens with no conscious effort.
Instead visual programming thinks I want to see the textual properties of my objects in memory in fancy coloured boxes, which is not the case at all.
It didn't take very long to realise that as a solution SSIS was well suited to some simple tasks where the logical actions on screen matched those taking place on the database.
But as soon as those tasks became even slightly more complex, and this mirror was broken, the whole thing sort of fell apart. Then I was struggling to find ways to defeat the system to make it do what I wanted. It was with this realisation, i.e. that the solution was the problem, that I stopped using it.
But it's not just SSIS that suffered this way, ActiveBatch is another example and I'm sure there are plenty more.
It's not meant for professional programming, it's about empowering the end user with programmable tools.
I've used it for real projects and it actually solves the spaghetti problem. Compare this: with this:.
You're right that until now its usage has been limited to domain-specific languages for automating applications. But with the emergence of functional programming in the mainstream and visual tools that follow that paradigm, I predict that soon there will be visual tools that allow quite general programming. (I've already seen working prototypes, and they deliver on most of the promises).
Have you seen the tools by Chris Granger (Light Table, Eve)? Although they are not as spectacular as Victor's, those show a lot of promise to become practical tools.
Chris Granger himself stopped working on LightTable because he didn't believe in the project, and if you use it, the live editing gives minimal gains in productivity. As well as being buggy, slow, distracting and potential dangerous when blindly running half complete IO operations.
As for Eve, well.. his first release was a visual programming tool that inflated 3 line sql statements to take up a whole page with, you guessed it, boxes and arrows. Now he is working on a 'card wiki' idea which effectively tries to hide all the program logic behind pictures and hyperlinks. Neither are good ideas, or once again, scale in any meaningful way.
As soon as you add encapsulation and abstraction, the system can scale; it's no different than hiding code logic behind function declarations and object schemas. If you add easy composition, like modern functional languages have, such systems could be more scalable than traditional imperative languages; code outliners like Leo follow the same structural principle, and they're great for programming.
* Ok, Wikipedia contains articles and not code, but my point is that the structure can handle lots of hierarchical content in multiple categories.
Additionally simply adding 'encapsulation and abstraction' does not equal scale. Live programming for instance requires context and re-running of code on every edit (and we could be talking millions of lines of code that need to execute), re-reading of data files, re-running of prior user input and so on. There's a reason it is only implemented for small systems (read, useless practically).
IMHO programming a large system shouldn't be all that different from writing an encyclopedia, or rather a math textbook; the same combination of informal explanation and formal demonstrations could be the shape that "visual" programs could take. Literate programming is written in English, and it provides much of the same advantages of the encyclopedia or technical book. Systems like IPython / Jupyter work in that direction, and they belong in the family of visual programming (although they use "classic" languages for the formal part, nothing stops them from also including graphical models as part of the code specification).
Programming languages should be made for human consumption, and one of the promises of visual languages is that they use secondary notation to convey the meaning of the program, so that parts that are not visible to the compiler are still relevant.
I believe that programming environments should converge towards a hybrid of programming, using the best representation for each concept - like text (to represent abstract concepts), diagrams (to represent more physical properties - time, data flows, and dependencies between modules), and raw text for the human-readable comments that explain how everything is tied together. If you build a programming environment like that, my comparison is not unfair at all.
>Live programming for instance requires context and re-running of code on every edit (and we could be talking millions of lines of code that need to execute), re-reading of data files, re-running of prior user input and so on. There's a reason it is only implemented for small systems (read, useless practically).
Modern IDEs are capable of doing a lot of that when in debug mode, I see no reason why having a language with a graphical syntax should make any difference - it just places a little more weight on the parser and graphic card.
The problem here is the "impedance mistmach" between humans and computers. Wikipedia is not a good example: it is exclusively for humans. If there is a "program" for the computer, it's minimal: "search this", "when I click here, take me to this page" (you'll notice everything more complex than that, even in Wikipedia, requires its own mini-language). This is the kind of trivial systems which are easily encoded by "visual" programming.
The problem is that real computer systems need to be understood by the computer as well as by humans. In order for something to be understood by a computer, it must be specified in formal terms; English -- natural language, actually -- is terrible for this. Once you're working in a complex system with a formal language, the limitations of these "visual" tools become evident.
Not exclusively. Bots and Wikidata have shown themselves pretty capable of doing complex automated tasks on top of the structured data contained in the encyclopedia. And the wiki platform plus templates is a very general and flexible data structure that makes it easy for humans and software to collaborate.
> In order for something to be understood by a computer, it must be specified in formal terms
This doesn't need to be that way, at least not for 100% of the system. End-user development tools like spreadsheets and Programming By Example can be used to ease the impedance mismatch without requiring the human to specify by hand every single step in a formal language, so that partially ambiguous inputs can be used to specify valid programs.
Only the task of building system software should require the full technical approach; most business applications can benefit from looser approaches - in fact being less strict can be a benefit in the requirements specification phase, just like User-centered design has shown us.
Combine that with logic solvers and deep learning, and in the future we could get some systems that can use very high-level commands to build fairly complex automated behaviors.
As for the rest, I think it's a pipe dream except for the most basic, "build this out of predetermined blocks" apps.
My point is that the structure of a wiki, where each unit of content is a separate building block, can be used to build a visual system in the same way that you can build a textual one; the basic components of storage and APIs are the same, only the thin syntax layer is different. You are no longer dependent on the single stream of raw text following a strict syntax, which is the basis from which compilers generate binary code in traditional languages; you'd be able to build programs by arranging code and data in a network within a semi-structured platform, which is one of the characteristics of visual tools.
You're right that stuff like this has been a pipe dream for at least half a century, since the Mother of All Demos; but in the very recent years I've seen a lot of the required pieces falling into place, so I believe it to be possible.
If we apply the same amount of engineering work and refinements that went form Engelbart's demo to modern desktop systems, we could have a functioning system with this paradigm; the basis are in place for it, and many of us have the vision of the benefits it shall provide.
Of course, part of building a program may require describing some components using a textual representation, where it makes the most sense; but with a wiki-like "outliner" platform, parts of the working system could be programmed using visual code, and be integrated with those other components programmed in traditional languages. Heck, GUI builders and visual tools for database schemas already work that way; the only thing they lack is the capability to be expanded with new components, which this new family of visual tools have.
Because it doesn't. Watch Bret Victor's last talk about Eve. All those ideas fail when you give them to novices and try to do anything more complex than describing a few static relationships.
Maybe there's a way, but I'm skepitcal.
Of course experimental tools will be severely limited compared with industrial IDE systems that have been polished literally for decades; it will take years before we get these new tools into shape and learn what their best use cases are. But I'm not skeptical, because I've seen it happen before, and the conditions for it are now better than ever.
I've thought about what could be done to improve on their weaknesses (like strong typing, bridging the gap to databases, better table naming), even wrote some code in this direction, but anything I can think of involves either making them less visual, less user friendly or both and doesn't even begin to address the lack of generality
Finally I have to point out that some spreadsheets have been around for longer than many programming languages in current use so they don't actually lack maturity, yet they have seen little improvement in over a decade.
They significantly drop the barrier to entry and allow basic controlling of systems for non-programmers.