Re: Desired Lua Features

[Paige DePol <lual@...>]

Dibyendu Majumdar <mobile@majumdar.org.uk> wrote:

> Okay thank you for the explanation. In the case of Ravi, having knowledge
> of types at compile time is useful for JIT code generation. I assume above
> approach is more a runtime check? I assume also that this approach works
> with up-values?

Yes, it works everywhere, including upvalues! Though, I did forget about
upvalues initially, so things broke quite spectacularly the first time I
tried storing an Index data type in an upvalue!

All types are also known at compile time as well, OP_TYPELOCK was created to
actually lock stack slots to ensure that the values assigned to them would
not violate the expected types. If no lock is done then the type is assumed
to be 'Varies', or untyped.

OP_TYPECHEK was added as there are times where you may have an object
sitting in an untyped variable so you need to typecast it in order to use
it. Also, the underlying field in lua_TValue used for type casting is also
used directly for flagging the current object isa scope when performing
super calls.

Everything works quite well now, however. Tables nested inside of an Index
data type don't need to be extracted either, which works out quite nicely. I
also have an optimisation that can be enabled which creates 'pseudo-indices'
to allow direct indexing of nested Index data types. It is able to stuff 4
levels deep into a single int value, at which time it extracts the value and
keeps indexing, including pseudo-indexing should the structures be nested
deeply. If the value to finally be extracted is a table, then it can just
leave that inside the Index data type and access it directly.


> I found adding support for a statically checked structured type was
> virtually impossible in Lua because then you would need to have type
> naming system - and handling recursive types add their own complexity. Do
> you handle such types at run time or compile time? Based on an idea by
> Roberto I decided to focus instead on making table access faster (this is
> one of the tricks used in LuaJIT too I think - i.e. make table access so
> fast that the cost is negligible).

I have a full type naming system, along with a preprocessor to declare both
@struct and @object types. Also, @class, which is for declaring the public
properties and methods of a class, but nothing private. Also note that I
have a number of other things parsed by the preprocessor including constants,
enumerations, macros, and inline functions.

Yes, this type of layout means that the source code must be parsed twice,
once for any @declartions and then again for actual code. This also gives
the option for packages to have a header file to declare things directly,
then the code is parsed on the next pass.

To support this setup I have also added @import which first parses any
header files and then also loads the code at compile time. To support the
standard Lua 'require' feature there is also @require, which parses the
source at compile time for @declarations only and then replaces @require
with a normal 'require' so the code will be loaded at runtime instead.

So far one of my design goals is to implement as much as is possible at
compile time, which is why the parser is now 233% larger than the vanilla
Lua one. I've tried to ensure that only the absolute minimum is present in
the runtime to support the features I've added. As the runtime has only
grown by 15%[1] so far I think I am succeeding in that!

I agree making tables fast is a great goal, and one that may preclude the
work I am doing. However, with an Index data type the values are just
available immediately without any hash table or metamethod lookups required.
The fields for the Index are stored directly in the VM instruction which
makes it quite speedy in terms of direct access, also with my pseudo-index
optimisation it is even faster to access things 4 levels deep at one go.

Right now I have basic recursion checking, if a data type contains itself
then the initialisation stops at that level and the recursive type value is
left as 'nil', otherwise you would just enter into an infinite loop until
memory ran out. I suspect there are other cases I should be aware of and
should be checking for... hence your next suggestion!

I have also been using the Lua Test Suite in full testing mode to ensure 
that nothing breaks, while adding to the test suite for my changes.


> Btw I would recommend putting your code up in github and doing development
> in the open.

Right now I am in the cathedral mode[2] of development, however, I will
probably change to the bazaar mode eventually. I am currently working on
finishing off my object model so I can create some tests around it, then
I will release SP1 (Sneak Peek 1) of what is to become the Lua++ codebase.

Well, actually, right now I am working on getting a website online with
my patch collection, after which I will return to hacking on Lua++! :)

~Paige

[1] All size values are comparing Lua with Lua++ using -Os optimisation.

[2] https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar