I agree with you by and large except for this part.
> COBOL's promise was ... we wouldn't need programmers anymore..average person doesn't know how to explain & solve a problem
COBOL wasn't intended to be used by an "average" person but rather those with deep domain knowledge. They would know the business processes so well that they could transcribe it in COBOL with little or no need to learn how the computers worked. In some ways similar to analysts/data folks using SQL to communicate with databases.
While at it let me share a few more aspects of the top of my head.
COBOL and 4GLs in general were primarily intended to be used to build business applications; payroll, banking, HRMS, inventory management and so on. Even within that emphasis was more towards batch processing operations to reduce the burden on people doing routine bulk operations like reconciliation.
COBOL harks back to the times when there was no dedicated DBMS software. Which is why you see so much focus on how files are organised and the extensive verbs around files which somewhat resemble SQL today.
In my experience, often it’s hard to find that person with deep domain knowledge, and even when they do, it’s unstructured, they take things for granted they shouldn’t* and the have no appreciation of the demands of formalism.
Getting anything you can use to construct a work plan, never mind a detailed feature list, out of clients can be a dark art.
*To the point I have repeatedly experienced a point close to the end of the project where they go “What do you mean you don’t handle a case I have failed to mention for the entire duration of the project?”
I recall a spec doc from a domain expert that said something like:
"The transaction consists of a credit stub and a debit stub. If the debit stub is missing and is of type X then we do A and if it is of type Y then we do B."
How to know what flavour the missing item was? Absolutely no mention of that...
It's interesting that domain experts all exhibit the same cognitive issue - their assumptions are just so ingrained that they cannot articulate it at all.
The fact that they "know" a missing stub would have a type is because they actually have some more information than they let on, and this information is only known by the expert. For example, they know if the submission was from party A, it must be type X.
But that fact might not ever be recorded in the computer system, in a way that the old business process would've had a record of.
And this is just one small example - imagine something more complex!
So realistically, the job of a programmer is to force an expert to articulate all of their assumptions. IMHO, the best way to do it is to be sitting with the expert, and observe exactly what they do.
My experience with "business domain experts" is that the majority of them are simply executing a process that someone else defined a long time ago. Their definition of "success" is usually that all steps within the process execute successfully without error and they can move on to the next transaction or activity. Very few of them are capable of taking a step back and considering what the process is actually trying to achieve and whether there might be a better way of accomplishing it. This leads to constant "paving of the cowpath" where archaic processes just get replicated in new technologies every so many years.
That is a fair point. The programmer will have to learn the process, but also understand the process' goals and intentions.
If they do, then they by definition become a domain expert. It's just that this takes a while, and projects usually don't give enough time for such to take place unfortunately.
I work in the energy industry at the moment, we spend a lot of time looking for a stakeholder / domain expert, and they all seem to point at each other; nobody has the complete picture of one domain / feature, or nobody has the confidence to own and decide upon all aspects of it. That is, that "side" of the organization spends a lot of their time in meetings and writing emails to try and decide on things.
I have much more sympathies for 'extreme programming' than for 'agile'. Mostly because I see many more clueless people use 'agile', while 'extreme programming' seems to be a term that time forgot and eg http://www.extremeprogramming.org/ is still in a state of relative innocence. The website was apparently last updated in 2009.
My experience is that invariably results in "development by veto". Each prototype they say that's not what I want, give me something else (that I'll fail to describe just like the last time) and I'll tell you that is wrong too after you've worked on it for a few weeks.
Occasionally, you'll randomly get something they accept - but only for a few weeks until they come across some missing capability for some other thing they never told you about.
> My experience is that invariably results in "development by veto".
Yes, I wasn't entirely serious.
Though you can get pretty far by doing some roleplay, where you pretend to be the computer/system (perhaps put up paper screen to make it easier to roleplay, and pass messages written on paper) and have the expert interact.
You still have to actually listen to the complaints. "That's not what I want" does not mean try again, it means they have no interest in what you are trying to offer in even the most basic sense. The lesson from that type of complaint is that you are barking up the wrong tree. Time to move on to something else.
When you are solving a real problem, you will still receive complaints, but they will be much more constructive.
COBOL dates back to 1959, much earlier than 4GLs, and the cited 1992/1999 articles make the point that 4GLs were poised to replace the likes of COBOL and FORTRAN when in fact those dinosaurs, or rather nautili since still living, turned out to outlive 4GLs except SQL (when counted as 4GL).
> In some ways similar to analysts/data folks using SQL to communicate with databases.
But SQL has the exact same problem. Except for very trivial scenarios, you can't just be an expert and plop your expertise into a SQL query. You have to learn how to use SQL to use SQL.
As with any other tool one has to learn it to effectively use it. Some find the learning curve not worth it and stick with Excel which is OK. But the thing is even Excel has to be learned to make full use of its potential.
Keep in mind that the context is around domain experts being able to transcribe their domain knowledge into a machine-understandable language without concern for the intricacies of the machine it is executed on. That is where COBOL and SQL are said to have failed to live up to the hype, of which I'd agree. SQL is not a particularly good abstraction. Even for relatively trivial tasks, you still need to understand how computers work. EXPLAIN is the bread and butter of SQL users.
Ultimately every abstraction is leaky. There will never be a solution where you never need to understand how computers work under all circumstances. But my impression is that you can go a lot further in Excel before the stuff going on behind the scenes starts to get in your way? From what I have seen, Excel itself is more likely to get in your way before not knowing how computers work does.
> COBOL's promise was ... we wouldn't need programmers anymore..average person doesn't know how to explain & solve a problem
COBOL wasn't intended to be used by an "average" person but rather those with deep domain knowledge. They would know the business processes so well that they could transcribe it in COBOL with little or no need to learn how the computers worked. In some ways similar to analysts/data folks using SQL to communicate with databases.
While at it let me share a few more aspects of the top of my head.
COBOL and 4GLs in general were primarily intended to be used to build business applications; payroll, banking, HRMS, inventory management and so on. Even within that emphasis was more towards batch processing operations to reduce the burden on people doing routine bulk operations like reconciliation.
COBOL harks back to the times when there was no dedicated DBMS software. Which is why you see so much focus on how files are organised and the extensive verbs around files which somewhat resemble SQL today.