I had this question for awhile too, but I realized after hacking on the Python interpreter that it breaks the execution model:
- Python compiles a single module to bytecode at a time (.py to .pyc)
- It interleaves execution and compilation -- imports are dynamic, you can put arbitrary computation between them, and conditionally import, etc.
If you had structs, then you would need to resolve names to offsets at compile time -- e.g p.x is offset 0, p.y is offset 4. That's a form of type information. Types can be defined in one module and used in another, and Python really has no infrastructure for that.
Nor do any other dynamic languages like Ruby, Perl, JavaScript, Lua, or PHP, as far as I can tell. They are all based on hash tables because deferring name lookup until runtime means that I can start executing "now" rather than looking all over the program for types. It probably helps a bit with the REPL too.
The need for type information in a translation unit is also what gives you C's header file mess, so it's not a trivial problem.
Actually, Python does have that :) It's called __slots__:
The presence of __slots__ does several things. First, it restricts the valid set of attribute names on an object to exactly those names listed. Second, since the attributes are now fixed, it is no longer necessary to store attributes in an instance dictionary, so the __dict__ attribute is removed. Instead, the attributes can be stored in predetermined locations within an array. Thus, every slot attribute is actually a descriptor object that knows how to set/get each attribute using an array index. Underneath the covers, the implementation of this feature is done entirely in C and is highly efficient.
It still involves a dictionary lookup to access the member doesn't it? It's just that the dictionary is located at the class level instead of the instance level, to fetch the offset of the member within the instance.
The goal of using __slots__ is to save memory; instead of using a .__dict__ mapping on the instance, the class has descriptors objects for each and every attribute
So python still has to look at the class for each attribute access on an instance of Foo (to find the descriptor). Any unknown attribute (say, Foo.ham) will still result in Python looking through the class MRO to search for that attribute, and that includes dictionary searches.
If I have time I'll write an example program to test this. Since I've been curious for awhile.
I've actually forked the Python interpreter to make my highly compatible bash shell faster and smaller, while not completely rewriting it [1].
-----
Even if __slots__ did what you think it does, it doesn't completely solve the problem.
A lookup like "return os.path.exists(foo)" can't use slots, because it's a module and not a class.
In principle, local variable lookups are also slower in Python than say Dalvik (a JVM without a JIT). LOAD_FAST still involves a dict lookup (as opposed to the even slower LOAD_ATTR). It just happens at the beginning of the function once, not on say every iteration of a loop inside the function.
In contrast, a Java compiler will resolve the attribute access at compile time.
Also, if slots did what you think it does, it would be 10x faster than normal attribute lookup, but it's not (according to the benchmark above, and at least one other one I saw.)
EDIT: My last two points might seem like a tangent, since they're not really related to structs. But my overall point is that Python doesn't do very much at compile time, so some of that work has to be done at runtime. slots could easily be faster than it is; it's mainly a memory optimization.
So python still has to look at the class for each attribute access on an instance of Foo (to find the descriptor). Any unknown attribute (say, Foo.ham) will still result in Python looking through the class MRO to search for that attribute, and that includes dictionary searches.
Yes, it has to lookup, but the lookup is cached (see typeobject.c).
Where in typeobject.c? I looked, and there's a lot of stuff related to slots, but I don't see any caching.
Also I did a speed test on my machine. My results are the same as the ones I've found through Google: __slots__ is not much faster. It's maybe 20% faster, whereas if it was really a dict lookup vs. index lookup I would expect it to be an order magnitude faster, or at least 5 times faster.
To be clear, I'm claiming that every time you do "p.x" with a class with __slots__, at least one dictionary lookup occurs. It's not like p.x in C or Java.
I could be wrong, but I've never heard the claim you're making, and I don't see any evidence it's true.
Hm I did a test with an coverage-instrumented Python binary, and you are right.
I counted the number of times lookdict_string in dictobject.c is called -- call that number D. And then I made N attribute accesses like 'p.x'.
Without slots, if N goes up, then D goes up. With slots, as N goes up, D stays constant.
Thanks for the information -- I'll look into this more.
(However, as mentioned, slots are not more than 20-30% faster on my machine, in this limited benchmark. I feel like this is a double mystery. Or maybe it's just that interpreter loop overhead or some other overhead is the bottleneck, and not dict access.)
I'm talking to myself at this point, but I think the problem with the caching is that every attribute lookup p.x has to consult the type, not just the object. That's not true in Java.
So you avoid the dictionary lookup, but you're still following extra pointers.
> C header file mess is caused by its designers ignoring the work outside AT&T and not bothering to implement a module system.
To be fair to them, they wanted to create a language whose compiler did fit into the limited hardware they had.
It's not their fault that UNIX had so much success and the world largely went there ignoring most of the progress made in programming tools and languages during the 70s.
That success was mostly caused by AT&T not being able to sell UNIX (at least during the first years of UNIX), so they just made the source code available to everyone willing to pay for a symbolic license, which was a tiny fraction of what vendors were charging for their own systems.
An OS and systems programming language available almost for free, including source code, versus what a commercial mainframe, its OS and SDK would cost without source code included, sure recipe for success.
modules are unloved children, I recently used scheme48 which has a very nice module system (funny considering how old, but anyway) even though other don't have one..
> and Python really has no infrastructure for that.
Nor do any other dynamic languages like Ruby, Perl, JavaScript, Lua, or PHP, as far as I can tell.
This is not correct.
Common Lisp easily supports structs, and is a dynamic language. No need to specify types for structs, although you can optionally do it in CL if you want.
Here's a textbook example (straight from the CLTL2 book...)
Here a struct is defined (only one line needed), and then the struct is used.
;; structure definition
(defstruct door knob-color width material)
;; Automatically creates the following function:
;; make-door
;; And accesors (can be used for getting and setting)
;; door-width
;; door-knob-color
;; door-width
;; door-material
;; ----------------------
;; 'instancing' the structure into "my-door"
(setq my-door
(make-door :knob-color 'red :width 5.0))
;; accessing
(door-width my-door) => 5.0
;; setting
(setf (door-width my-door) 43.7)
;; accessing
(door-width my-door) => 43.7
(door-knob-color my-door) => red
I don't doubt that CL could do it -- if there's any dynamic language that can support efficient C-style attribute lookup for structs, it's Lisp.
(Although I honestly doubt that idiomatic Lisp does this. Idiomatic Lisp might be even slower than idiomatic Python, because I've seen people use alists as structs rather than dictionaries.)
Either way, your example doesn't really address what I'm talking about. Python has a struct module too, and it has namedtuple, etc. My claim is that Python doesn't have C-style structs, where attribute names are resolved to indexes AT COMPILE TIME, because it breaks the model of the bytecode compiler and dynamic module system.
Lisp is flexible about compile time vs. runtime, so I won't be surprised if it can do that. But again there's no evidence in your example that it does that.
CL implementations offer the indexed access, but include the data in the struct only for few data types.
Any good Common Lisp implementation will use fixed indexes. The Common Lisp compiler can inline the accessor and will be using fixed indexes.
What Lisp compiles usually not do is inline the data in the struct. Thus the struct slots will contain a pointer to the slot value.
Exceptions: small integers, characters, ... Those will fit into the slot and will be stored directly in the struct.
That's the point of structures in Common Lisp.
Below is an example in Clozure Common Lisp on a ARM processor. As you can see, the compiler can inline the accessor and uses fixed indexes to access the slots...
It doesn't have to be exactly like C in that it's just a pack of binary data with offsets. It can be like named tuples without the immutability and [index] access.
- Python compiles a single module to bytecode at a time (.py to .pyc)
- It interleaves execution and compilation -- imports are dynamic, you can put arbitrary computation between them, and conditionally import, etc.
If you had structs, then you would need to resolve names to offsets at compile time -- e.g p.x is offset 0, p.y is offset 4. That's a form of type information. Types can be defined in one module and used in another, and Python really has no infrastructure for that.
Nor do any other dynamic languages like Ruby, Perl, JavaScript, Lua, or PHP, as far as I can tell. They are all based on hash tables because deferring name lookup until runtime means that I can start executing "now" rather than looking all over the program for types. It probably helps a bit with the REPL too.
The need for type information in a translation unit is also what gives you C's header file mess, so it's not a trivial problem.