Metatables are "hard" (Re: Question about __index and classes)

[Mark Hamburg <mark@...>]

This thread plays a bit to a message I started a while ago and never finished. Metatables are "hard" -- particularly for people coming from backgrounds in OOP languages. This message is not about potential changes to Lua but rather is an attempt to understand why people get confused so often.

Some metatables entries like __add, __mul, etc are pretty easy for people to grasp. Their implementation is as fallbacks when the VM tries to add or multiply two values and discovers that they aren't both numbers, but they feel like fairly straightforward definitions of how the "type" being constructed via the metatable behaves when adding or multiplying. In an OOP language, these would correspond to add or multiply methods or perhaps some other form of operator overloading. So, metatable entries seem just like Lua's particular construct for building types just like in other languages.

The stumbling blocks start to appear when one encounters the difference between overrides and fallbacks. Overrides always win. Fallbacks only get invoked when there isn't something else to do. Most languages offering these sort of facilities offer overrides. Lua offers fallbacks. For example, why doesn't __len work on ordinary tables -- something changing in 5.2, I think? -- because __len is already defined for ordinary tables. Lua doesn't go looking for a __len metamethod when taking the length of a table because it doesn't need to.

Then we get into __index and __newindex on tables. These only get looked to if the table doesn't contain the key. They are fallbacks for that case rather than overrides for reading or writing tables. As a result, building a structure that catches all reads or writes requires the use of proxy tables and those in turn cause other complications like pairs not working. What's more not containing the key is essentially equivalent to the value being nil, so if you need to build a structure with slots that can contain nil, the structures you need to build to make __index and __newindex work grow more complicated. What's more with both __index and __newindex able to reference tables instead of functions, it is easy to build things that mostly work -- except perhaps for nil values -- and that smell a lot like inheritance in traditional OOP languages.

So, what I think happens for people coming in from other languages is that metamethod fallbacks look like overrides but aren't and the semantics for some metamethods and the techniques for using them successfully get relatively complicated. Furthermore, afflicting both people coming from other languages and people starting programming with Lua, there are cases where it's easy to build solutions that mostly work but the fully "correct" solution will be more complicated, more subtle, and more expensive.

All that said, there is almost certainly a performance tradeoff. Doing dynamic dispatch for every operation could give one a simpler semantics, but could it run nearly as fast? If Lua's semantics continue to evolve -- as 5.2 seems to show they will -- this is probably a worthwhile area to keep pushing on. If __len can become an override instead of a fallback, what else can change or be introduced without slamming performance? But as I said when I started out, this message is not a call for change, it's a speculation as to why Lua is at times hard for people to grasp.

Mark