Off topic, but I have some free time over the next year and a half and thought a fun intellectual project would be to work through the Lisp books (SICP, AIP, Theory of Metaobject Protocols, Let over Lambda, etc...).
Would this actually be worth it if the metrics are intellectual curiosity and general engineering skill improvements with a focus on the long-term?
Edit: If it's not worth it, what would be a similarly interesting alternative?
A bit late to the party. I recommend SICP, On Lisp & Art of Metaobject Protocol. The last you can skip to some degree because its quite advanced so unless you want to stay in the Lisp world, not as useful IMO (but lovely from an intellectual perspective, its a masterpiece). SICP definitely IMO.
On Lisp is by Paul Graham and pretty good too, but again its kinda more Lisp focused. Actually try PAIP by Norvig, I heard that is really good and it should be more useful for non lispers (even though its a lisp based book, it teaches lots of good programming concepts)
I would say do one of these but not all. Then find something in school or work that is hard and worth learning and go deep in that area for a time. A year and a half is a long time so keep reevaluating :)
One of them would be good, but you wouldn't learn much from two I think. Lisp isn't that semantically interesting anyway, it just has the most persistent/pretentious advocates.
Pages 4 and 5 illustrate the central conceit: Instead of a set of fixed operations (like allocating a class instance or accessing properties/slots/fields of a class in a fixed way), you can reprogram how your classes behave (in CLOS, at both compile and runtime). By way of analogy, consider a GPU implementing a fixed-function pipeline versus having shader units. The former gets the job done, but only offers so many ways to control what gets rendered, the latter is far more capable in the longterm (at the cost of added complexity).
Trying to get this to fit into a comment so I'm being handwavy a bit.
Consider the Person class that they describe. Because it's sparse (not every field is populated), instead of having a large fixed size for all thousand fields (or however many) they change out the underlying representation to use a hash table. Now, anyone can represent their data using a map. However, just storing your data in a map doesn't let you continue to use the other language features that are built around classes and objects. In particular, take C++, if you stored your data in a map you'd lose this bit of handy syntax:
Person p = ...
p.age = ...
Well, if age is stored in a map inside the person class, there is no age field to assign to. With a MOP, however, you can still have this more compact data representation format (since the data is sparse), but still have all the normal language syntax and semantics around the object (like accessing fields using the dot syntax). In Common Lisp you might use (or any other slot accessor):
(with-slots (age height) person
...)
And the change in representation doesn't change how the slot values are accessed (both for reading and for writing).
Now, changing the underlying data representation is useful, but a relatively small thing. Consider something a bit more useful: Ada's protected objects. A protected object in Ada adds, basically, mutexes and condition variables via language syntax to control access to data shared across tasks (and other things, this is just the main utility of them). It's a special set of syntax added to provide this ability, in principle you can use MOP to extend CLOS to provide this behavior. Change your meta-class from standard-class to protected-class. Users still access objects (from their perspective) exactly like any other, but all the mutex and condition variable stuffs can be automatically baked in (no need to add extra logic in your methods).
Now, this does introduce potential runtime complexity which eats into performance. And that's addressed further into the linked paper. It's not totally insurmountable, but it does have to be considered (any time you add indirection or potential indirection you have to decide if it's worth the cost).
> With a MOP, however, you can still have this more compact data representation format (since the data is sparse), but still have all the normal language syntax and semantics around the object (like accessing fields using the dot syntax)
just sounds like the JavaScript object model, and this:
> Users still access objects (from their perspective) exactly like any other, but all the mutex and condition variable stuffs can be automatically baked in (no need to add extra logic in your methods).
kinda sounds like computed properties/getters+setters (or simply "Properties" in C#), which many languages have at this point. I assume I'm missing or misunderstanding something?
Yes, you missed the point where you as the programmer can choose the semantics of the language as you go, rather than the language designer doing it ahead of time. Instead of having a language designer pick one spot in a large space of possible designs, the Metaobject Protocol gives the language user the ability to pick their own spot within that space.
You can have sparse objects in the one project where they are needed, while not bothering in projects where they aren’t. You can have version controlled objects, or database–backed objects, or objects which implement inheritance differently. If you can think of it, you can probably do it.
Piumarta's papers are totally underrated! There is only one thing I didn't understand from that paper (as a barely-used-c programmer), which is what does it mean to access some_array[-1]?
His COLAs paper seems like it's really interesting as well, but I can only understand maybe 30% of it.
He proposes placing a value's vtable pointer in the memory word before the value's base address. Now you can pass the value around to APIs that expect, say, a C-struct. But the vtable is "secretly" attached to it so you can call methods on it from your language implementation.
I guess I just don't know C well enough. How can it be legal to put anything at a memory address before a given array? Does the compiler magically manage this? (Also if you know of any posts about this technique I would really appreciate it)
That still sounds like getters/setters, then (which you could use to implement properties-as-a-hash-map, for example). Maybe the ergonomics are different - we're still speaking pretty abstractly here so it's hard to tell - but it sounds like the same basic idea:
class TimePeriod
{
private double _seconds;
public double Hours
{
get { return _seconds / 3600; }
set {
if (value < 0 || value > 24)
throw new ArgumentOutOfRangeException(
$"{nameof(value)} must be between 0 and 24.");
_seconds = value * 3600;
}
}
}
What about only providing methods named after fields for outside use? That way you can extend that class to provide either field-based access or anything else you like. E.g. in scala there is no syntactic difference between method and field access so code will compile with both person.age and person.age(), while implementation can be both “sparse and dense”.
I was writing a reply that got deleted (browser refreshed on me when I left the computer alone too long), so I'm going to focus on the second part because I'm not rewriting it all. Also, know that I'm new to MOP programming (which is part of how I came across this paper) so my example is high level, but (as far as I presently understand) correct, just not detailed.
So for my particular example, the difference would be (in my phrasing) automation. If someone took the time to implement everything, then you could do something like this:
(defclass protected-class (standard-class))
(defmethod initialize-instance :after ((class protected-class) ...)
...) ;; adds accessor logic with locks
(defclass my-protected-class ()
((a-number :protected)) ;; maybe not explicit, but for here it is
(:metaclass protected-class))
This would be more like (but not quite the same as) decorators in Python. In C# with computed properties you could do something like (been a few years, treat as pseudocode):
class MyProtectedClass {
private int _a_number;
public int ANumber {
get => _a_number;
set {
lock;
_a_number = value;
unlock;
}
}
}
Ok, so it's protected. But how do you do this without manually inserting this locking logic throughout? (Again, been a bit for C#, it has some good reflection libraries so you probably actually can do this similar to CL, but it won't be as straightforward or accessible to the programmers using the language). In Java (also been a few years, so don't treat as current best practice), you used to add synchronized to methods and get the automatic locking (why I did :protected above, to keep it in line with Java). But note that this is something the Java creators added, there was no way (within Java) for you to extend the language to do this yourself (or no easy way, again their was reflection capabilities from the start).
And it's important to remember, I meant that as just an example. There are other, conceivable, characteristics you may want to add for some reason (hopefully a good reason). With a MOP-ish facility, you can extend your language to do these things, but without it you can't (or can't easily, consider C where there is zero reflection so you have to have an extra-language facility to provide extensions).
i.e. you could write a single get/set pair that works for every property on an object, intercepting operations and doing arbitrary logic, including passing values on to the default behavior if you want to
Basically (I don't understand this satisfactorily either) : represent meta concepts of the object system inside the object system itself.
As always, the best way to make sense of it is with an example. One example is meta classes in python. Classes, those 'blueprints' that we create objects in their image? they are themselves but objects, their class is called 'type', it's a special kind class, a meta class (a class whose objects are themselves classes).
You can inherit this type and create your own meta class, and that lets you do various cool things. For one thing, object creation in python is done by calling the object' s class as if it's a function. Now this is a method call (__call__), where do methods reside? in the class off course. So for classes, the __call__ reside in the meta class. When python says you can inherit this class, you listen very carefully. This means you can inject arbitary code during the instance creation process. Now every class that is an instance of your meta class can email you whenever somebody wants to create an object from it. It's like the interpreter is giving you a hook to something incredibly intimate and private to the VM, exposing it from within the interpreted language itself (and not as a C API for example)
meta classes is a kind of MOP for python because it represents intimate details about an object system ('hey, what exactly happens when we ask a class for an object?') inside that object system itself ('it calls this particular method, which you are more than welcome to override and augment as you please without leaving the comfort of the language').
The idea is to let users of a language, here CLOS, to be able to intercept meta-messages like instance creation, method call, property access, etc to provide their own implementations.
It's like the doesNotUnderstand in SmallTalk, method_missing in Ruby or more recently .Net Dynamic Language Runtime or invokedynamic in Java.
That’s a bad idea though, and the past 30+ years of OOP has laid-bare all of the problems associated with not having compile-time object type safety. Just because you can do something doesn’t mean you should…
Outside of prototyping and quick-hacks, there is no legitimate reason to use `dynamic` in C#/.NET: instead, the moment your program receives or handles a “late-bound” (i.e. untyped) object your program should verify its actual type and then access it only through a wrapper type (be it an interface, struct, or class) which is how you can enforce guarantees about that object. This also applies to reflection (so have all of your `GetMethod` calls during the wrapper’s ctor and use Action/Func wrappers over MethodInfo).
> That’s a bad idea though, and the past 30+ years of OOP has laid-bare all of the problems associated with not having compile-time object type safety
Hyperbole much?
The past 30 years has shown me that a language with a joke of a compile type system system continues to be The Language to write operating systems that still rule the world and run most of the chips we use in every day life: C. Until recently, Linux built in said crappy type system was glued together with completely unchecked shell scripts. The great and abominable typless wonder that rules and glues mankind to the internet is JavaScript. Visual Basic dominated the “business scripting” world. The language that reins in scientific computing, is typeless Python. The language that participated in Apples comeback with a fraction of the manpower being spent elsewhere: ObjectiveC.
I write a lot of Swift and Kotlin. I have as many bugs as I did when I wrote Objective C.
The thing I have learned with types is that it’s an 80/20 thing. I’m not an anti-typer. I don’t believe in relying on any one single paradigm to improve the quality and robustness of my systems. Do tests help? Yes. Can some level of type annotation/checking be good? Yes. But taken to an extreme, for me they follow the axiom:
“Every institution perishes due to an excess of its own first principle.”
Maybe tooling is finally getting there. And we’ll finally transition to a more compile time safe world. What I have observed though is that when any system/language becomes too rigid/constrained, people will swing to more @dangerous” solutions to have the freedom to create what they want.
Even between program this occurs. Numerous inter program formats and schemas have come and gone. What dominates today? JSON.
> Just because you can do a thing doesn’t mean you should…
Just because you don’t appreciate a thing, doesn’t mean others shouldn’t do it.
Sure, it's usually a bad idea to muck around with the MOP or use metaclasses in Python or whatnot. But every once in a while it really is the best solution to the problem, so you're glad you can do it at all.
> It's like the [...] .NET Dynamic Language Runtime or invokedynamic in Java.
My answer concerns the "DLR", aka `dynamic` type in C#/.NET, which is untyped.
The GP is not correct when describing the DLR/`dynamic` as to "be able to intercept meta-messages like instance creation, method call, property access, etc to provide their own"...
...C#/.NET does not allow for this: you cannot intercept methods or otherwise have Smalltalk-style message-passing in .NET. Using `dyanmic` does allow you to intercept methods and define behaviour but only with `dynamic` objects managed by your own `IDynamicMetaObjectProvider`, which I concede you could use to implement this, but that would force you to type everything as `dynamic` which also means you lose static type safety.
(pre-.NET Core, the CLR had hardcoded magic behaviour for `Proxy` types, which is how .NET Remoting worked with any object type: including sealed types, but this was removed in .NET Core and its replacement in `DispatchProxy` falls far short: it only works with interfaces (hence vtables), not static/non-virtual methods).
--------------
What surprises me about .NET's CLR is that despite the (frustrating) simplicity of its type-system, that applications still don't have any real opportunities to extend the CLR's type-system to suit themselves. This includes anonymous refinement types via flow-analysis (TypeScript gets this right), the lack of support for real ADTs (F#'s tagged-unions exist in their own hierarchy: you can't have a tagged-union of arbitrary types, which basically makes them the same as Java's `enum` classes).
The oversimplified view is that it's basically an API like Python's "dunder methods," but both more powerful and (iiuc) able to be compiled to reasonably fast code.
Raku (formerly Perl 6) is built on top of a metaobject protocol:
> Raku is built on a metaobject layer. That means that there are objects (the metaobjects) that control how various object-oriented constructs (such as classes, roles, methods, attributes or enums) behave.
It’s useful to remember that 1993 was near the peak of the OOP hype in the industry. There was every incentive to present solutions that extend the class hierarchy paradigm rather than replace it.
Metaobject protocols: Why we want them and what else they can do (1993) [pdf] - https://news.ycombinator.com/item?id=12641917 - Oct 2016 (1 comment)
Richard P. Gabriel's Review of "The Art of Metaobject Protocol" - https://news.ycombinator.com/item?id=1306177 - April 2010 (1 comment)
More in comments:
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...