I believe pretty much anything Paul Biggar has to say about dynamic language interpreters / compilers. His talk at StackOverflow DevDays London about dynamic languages was a great eye-opener for me and a fantastic experience.
I was at the same talk, but I got the impression that because his perspective comes from trying to do static analysis and optimization, he doesn't fully appreciate some of the capabilities of dynamic languages. The overall effect was a static typer missing the point.
But as he says in this post, PHP code doesn't necessarily take advantage of these capabilities in practice.
Yeah, PHP isn't Python or Ruby, which are properly dynamic. I think those languages are too far gone in their particular dynamic direction to be saved. But I'm convinced you can build a language that feels dynamic, but whose meta-programming is largely compile-time. Even the funky Python I see (metaclasses and decorator) could largely be compiled if the language rules were slightly tweaked.
I'm interested: which capabilities don't I appreciate? (BTW, my background is very much being a static-typer, missing the point, and slowly realizing it).
First off, I'm pretty much a static typer myself, so I'm not necessarily the best advocate for the dynamic case. But by the same token, I can argue against some of the problems of static typing.
First up: I'm talking about object-oriented approaches, with polymorphism and dynamic dispatch being the core abstraction tool.
If you try to introduce more static typing into a system with polymorphism, you're trying to push type decorations so that they can follow where the expressions go a little further, so that you can reason more. Parametric polymorphism, or generic types and methods (hence forth called generics to avoid confusion with OO polymorphism), are a key tool. They let you better reason about values of which types cross method boundaries. With generic methods, you can usually infer the type parameters. With generic types, the type arguments can serve as extra documentation. A Java ArrayList<E> seems clearly superior to ArrayList at first glance.
But all sorts of problems start coming up when you mix polymorphism with generics. Type inference isn't so simple anymore, as ad-hoc choices need to be made. Instantiations of generic types have no necessary subtyping relationship. So you have to reintroduce dynamicism in the back door via base classes or interfaces etc. that use a top type (java.lang.Object or whatever) to re-unify types; or you dabble in variance. And you want to do more with your arguments of type parameter types, so you bring in type parameter constraints, or type classes, or other means of generically handling values.
So perhaps you have use-site variance or declaration-site variance combined with type classes combined with polymorphism. Suddenly using or declaring generic types has gotten quite a bit harder. Getting type information to flow into the right places starts requiring confusing tricks, particularly when you want e.g. the methods of your ancestor to return a type related to the concrete instantiation type, for example something like this in C# syntax:
// so you want a GetThis method that returns a correctly
// typed thing for the current instance
class Base<T> where T: Base<T> { public T GetThis() { ??? } }
class Desc: Base<Desc> {}
When the static typing advocate has tangled himself up in a knot like this (or deeper, e.g. when you're passing other type arguments around across multiple classes and hierarchies, just to get the damn types right), the dynamic typer shakes his head, and says "don't you see - it's all just objects!".
One of the paradoxes is that declaratively specifying, at compile-time, all the types that may reach different parts of your program at runtime, is harder than writing the program to build the object graph at runtime in the first place, and treating the occasional runtime type error as a bug that will be eliminated.
Now to the other side of the story, meta-programming. I agree that you can do lots of meta-programming at compile-time, but there are two problems, as I see it. One: compile-time is too early to be making decisions. What do you do when you can't afford to restart the process, but you still need to change the running code and how it works? This isn't a theoretical concern; I've architected such a server system in the past. It could keep on handling requests relating to client sessions associated with a previous versions of the server-side behaviour, but new client sessions would use the latest version of the server-side behaviour. Rolling over the server farm from one version to the next didn't require tricks with load balancers and restarting app servers, etc.; it was all built into the system.
(Actually, the architecture was really interesting, and I should write it up in more detail one day. Another key fallout of this approach was that the session data, which was (usually) stored on disk and was a bit like a Lisp or Smalltalk image only typically 20K in size, had a pointer (a URL) inside it telling the server where to get the behaviour for this session. This meant that problem sessions could be post-mortemed quite effectively: the old session could be resurrected, stepped through, etc.)
And the second problem of compile-time metaprogramming: the formalisms you use to munge with the types, particularly static types which the compiler will later infer further things from, may in practice be quite different to the imperative constructs programmers are used to using to implement behaviour. When they want to add a method to a class, they want to do it using the same tools (or close) to that which they'd use for adding an item to a list or hash table.
Perhaps the meta-programming is implemented as a plugin to the compiler, or a macro system, where the code is modifying the compiler's own definitions of the types etc. But now we've introduced an abstraction level into the chain of source code -> executable text that most programmers are not familiar with, and may be intellectually ill equipped to deal with - not because they're stupid or incompetent, but rather because they simply don't need to know. Consider e.g. hygienic macros and the problems of which symbol refers to which level of abstraction, the compilation level, or the runtime level; the distinction between the level you're at when you're quoting and unquoting, in Scheme terms. Consider the confusion of this poor fellow whom I helped out:
When you consider the tangled mess of static typing you can get into when you mix polymorphism with generics, and then add in the ability to munge with code and types such that the compiler sees the modified code / types, and hopefully you can see it's easy to add more complexity than the benefit you get out.
