For complex, specialized systems what I'm often seeing is that essentially every decent large system gets split into a de-facto domain specific language (no matter if it's with language features or through some specialized libraries or webservices or whatever API) and the implementation of the business logic in that DSL. And, in general, the implementation of that DSL is something like ten times smaller than the amount of code using that DSL.
The maintainability of the code is mostly determined not by how good the underlying language is but how good is that DSL. To take an example that's been widely discussed in HN, let's think about machine learning - the look, structure and maintainability of your code is highly influenced by the choice of your framework/"DSL" e.g. pytorch vs tensorflow1.x vs straight numpy vs Caffe will have major differences despite being in the "same language".
And in that regard, the features of the core language matter only in regard of whether they facilitate making a good DSL. Of course, some languages fail because their choices result in the "core DSL" i.e. standard libraries that everybody uses being fragmented and confusing; Go has it quite good IMHO for the standard library; but for every specific domain it also matters if the DSL for that domain is good, whether it's an ORM layer or a OpenGL interface or a deep learning framework or an physics simulation library - whether the language abstractions mean that these DSL's (often being interfaces/wrappers to some third-party code written in another language) are going to be good and convenient, or whether the language structures limit the DSL so it becomes unwieldy to use.
> For complex, specialized systems what I'm often seeing is that essentially every decent large system gets split into a de-facto domain specific language
Every time I've seen this, it's been a disaster. Custom DSLs for business logic are, 100% of the time, a red flag of a development process gone off the rails.
(We may work in different industries.)
Go prevents you from doing this at a very low level. This is a huge advantage.
If you're writing a React or Django or Laravel or Rails app, then what you're doing is essentially work within that DSL which is quite different from the underlying Javascript/Python/PHP/Ruby language. The same goes for data analysis frameworks such as Pandas, or deep learning such as Tensorflow or Keras - if you're working in that most of your code and calls are going to be within that API/DSL, not "pure" Python. And I'd definitely consider all of the abovementioned as successes and a good thing - if Go would prevent people from writing their Web-CMS systems in a good Web-CMS framework/DSL, or writing their machine learning systems in a good ML framework/DSL, then that'd be a serious limitation.
And it's worth noting that many (all?) of these particular systems were actually initially made as custom DSLs for internal use within a single company - Rails for 37signals/Basecamp, Pandas for AQR Capital Management, Django for Lawrence Journal-World, Tensorflow by Google Brain, etc. They were the exact thing that you describe - a custom DSL invented for the business logic that the particular company needed repeatedly. And they were not a disaster but the right thing to do in their case - mostly because they had the ability to make a somewhat good DSL though in all those cases it was an 'okayish' DSL initially and became good only through years of substantial changes. Yes, for every such case there are many bad internal DSLs. "Not Invented Here" is a common problem. You need to ensure that you're using good DSLs, and good DSLs generally gain wide adoption within that domain instead of every company building their own shoddy DSL.
I've seen a bunch of nightmarish financial systems with horrible DSLs. However, even in that case the DSL problem was that this DSL was horrible not that a DSL shouldn't have been used - any fix or rewrite would generally require replacing it with a better DSL/framework/whole-encompassing-structural-API/whatever, instead of sticking to the core programming language (which would just result in another emergent "DSL" of some core libraries/classes, which would be lousy compared to something you intentionally design to be usable).
I would say most of the examples you mentioned are actually examples of well implemented libraries, with well named methods and abstractions, rather than DSLs. If you want to imply that a good large codebase is similarly structured into libraries of independent components rather than one big class hierarchy then I would agree.
I'm probably trying to make a point that when writing code to a large domain-specific library or framework, at a certain level of intensity of coupling it de facto becomes a DSL. You're writing code using "primitive" data types, data structures and functions specific to that DSL instead of the data types of your core language; the control flow and data flow is not the same as in your core language but determined by that framework.
The differences between programming languages are smaller than the differences between DLS's implemented in the same language. Code in Rails (Ruby) and Django (Python) has more in common than code for Django and Pandas. Writing Tensorflow code in Python is somewhat similar to writing TensorflowJS code in Javascript, but very dissimilar to writing Pytorch code that does the same thing.
They're not really independent libraries, they're frameworks that suggest, influence and sometimes even mandate most of the infrastructure around your code and the structure of your code in a much more stringent manner than the core language does. There's a fundamental difference between making a few calls to a black box library versus having the content and style of most of your code being dominated by the traits of that "library" - the large frameworks overwhelm the small(ish) core languages. So I believe it's justifiable to consider the frameworks I mentioned as DSLs.
Replace "DSL" with "idiom" and I totally agree with you. It's often possible to look at a code snippet and recognize what codebase it's from, even if it's doing something pretty generic. There are explicit local APIs, implicit local APIs (often associated with local data structures or stores), and local habits such as looping over items in a particular way. Managing memory, scheduling work, logging, and so on all tend to follow local rules. Those rules together are the moral equivalent of a DSL.
So you're right, for programming in the large it's very important how well the language supports creating good reusable APIs and data structures. I think that's why so many people make such a big deal about generics in Go, though personally I think it's a poor example and the complaints are tiresome. I think I'd focus more on whether the language has things like macros and lambdas/continuations that let non-API idioms be expressed consistently and conveniently, as though they're part of the language (even though I think creating a true DSL is usually a bad idea). Not sure Go meets this standard TBH.
What would you need to do refactoring operations on such code?
Given that there are classes, arguments are specialized over classes, slots have types, ...
Didn't know about those type annotations in lisp, but i suppose what you mean is that an ide could parse all the code, infer the type of all the function / variables etc (just like the lisp interpreter would) and provide you with refactoring facilities ?
That's a good thing. But then comes the git merge (or any other automatic code altering tool). How do you know something hasn't been messed up in the process ? (unless you wrote unit test for every argument of every function)
The maintainability of the code is mostly determined not by how good the underlying language is but how good is that DSL. To take an example that's been widely discussed in HN, let's think about machine learning - the look, structure and maintainability of your code is highly influenced by the choice of your framework/"DSL" e.g. pytorch vs tensorflow1.x vs straight numpy vs Caffe will have major differences despite being in the "same language".
And in that regard, the features of the core language matter only in regard of whether they facilitate making a good DSL. Of course, some languages fail because their choices result in the "core DSL" i.e. standard libraries that everybody uses being fragmented and confusing; Go has it quite good IMHO for the standard library; but for every specific domain it also matters if the DSL for that domain is good, whether it's an ORM layer or a OpenGL interface or a deep learning framework or an physics simulation library - whether the language abstractions mean that these DSL's (often being interfaces/wrappers to some third-party code written in another language) are going to be good and convenient, or whether the language structures limit the DSL so it becomes unwieldy to use.