What's next for Ravi in 2018?

[Dibyendu Majumdar <mobile@...>]

Hi,

It is that time of the year when one looks back at the year gone past
and plans for the next year. So firstly I would like to summarize the
changes I made to Ravi in the last year and whether they were
successful or not.

I started an initiative to generate JIT code using C, rather than LLVM
IR. The goal was to a) make it easier to maintain the JIT code
generator, and b) target other JIT engines. This work is partially
done. In terms of results - I tried using NanoJIT backend to handle
machine code generation but the results were not satisfactory as
NanoJIT's lack of optimizations meant that the generated code was no
better than the interpreter in performance. This work was useful in
other ways however, as looking at the generated C code made me realize
that it may be feasible to write a hand-coded VM in assembly.

Above realization led to the current initiative to write a VM in
assembly using dynasm. This is a very slow process due to the effort
required to debug and resolve issues. I anticipate this work to
proceed in 2018 but am not sure whether it will be completed or not.

In 2017 I also implemented additional type annotations in Ravi. I am
particularly pleased that my approach (compared to other approaches)
keeps full backwards compatibility with Lua, and even uses Lua's
single pass compilation approach.

The other changes in 2017 were mostly bug fixes and LLVM upgrades,
including the migration to LLVM ORC api for JIT compilation.

The question that comes up first in my mind as I look to 2018 - is it
worth it? Should I continue with this project or not?

I realized in 2017 that actually a JIT compiler for Lua was not really
needed for my own use case. I now think that because LuaJIT is
available people tend to think that they need a JIT compiler - but
when you look deeper into use cases for Lua, it is not really clear
that a JIT engine is actually needed. Arguably some of the use cases
where the JIT is used could now easily migrate to other languages like
Go. And the spectacular success of Python shows that interpreter speed
is not the primary factor in achieving popularity and success.

Maintaining any project that is dependent upon an upstream project is
fraught with issues. I have to take great pains to ensure that my fork
does not diverge too much from Lua so that I can merge changes easily
enough. But already this is becoming hard as merging upstream changes
needs a thorough impact analysis on the JIT backend. For this reason I
am not looking forward to merging the GC changes in upcoming Lua 5.4.

If I had sufficient time I would like to create the equivalent of
Python's numeric libraries for Ravi (maybe by starting with Torch).
But reality is that there isn't that time available, and it would not
be of much use anyway as the world is unlikely to switch from Python
to Lua, nor are torch users likely to switch from LuaJIT.

For now, I plan to continue to work on Ravi in 2018, mainly on the
assembly VM.  Other than that I will try to merge the GC changes in
5.4 to Ravi, and aim for a 1.0 release of Ravi.

Feedback welcome!

Happy new year all!

Regards
Dibyendu