Re: Lua distros again

[Hisham <h@...>]

On 13 February 2018 at 09:41, Pierre Chapuis <catwell@archlinux.us> wrote:
> Well, I'd be happy to know the source for that and how many years
> "some" is, because I don't think that was ever the case for most
> people's definition of "scripting". Lua has always had very strong
> niches (games, embedded...), more and more recently (with e.g.
> Software Defined Networking and Machine Learning) but I can't
> think of a time where it was much higher in mindshare than now.

Can we even still put Machine Learning in that list, with the demise of Torch?

>> I fear we are steadily losing terrain. I haven't hard data to back up
>> this feeling, so I may well stand to be corrected (and I'd like to be,
>> really), but I see lots of news about Python gaining ever more users
>> among scientists and non-programmers, whereas nothing about
>> Lua outside  very specific "circles".
>>
>> Mind, I don't think Lua user base is shrinking, on the contrary, I think
>> it is increasing, but much, much more slowly than other languages.
>> This means that a widespread diffusion of Lua (as was the intention of
>> their authors, at least years ago) is not going to happen, IMO.
>
> Scientific people is a specific case IMO, related to ML frameworks.
> I have been pretty much involved in this so how is how I see things:
>
> - Once upon a time people were using various things including
>   Matlab, R, Python (Scipy / Numpy), C++, etc.
> - Deep Learning happened, and the market split between three
>   major frameworks: Caffe (Java), Theano (Python) and Torch7 (LuaJIT).
> - Important companies like Facebook and Twitter chose Torch7 so Lua
>   grew in popularity.
> - Some people at Google were not happy because Lua is not one of
>   their blessed languages, but even their own team was using it
>   nevertheless (AlphaGo).
> - They pushed for new C++ / Python frameworks with TensorFlow
>   and Keras.
> - Meanwhile some people who preferred Python to Lua made
>   PyTorch which got a lot of adoption too.

Small clarification: these "some people" are the same people who made
Torch in the first place. PyTorch is developed by the original Torch
team and the Lua version is no longer being developed (it's in
"maintenance mode").

> So tl;dr Lua had a short-lived bubble in that space because of one
> framework, but now it's not the only one on the stage anymore so
> Lua usage is going back down.

And when I contacted members of the Torch team to learn more about
this, the reasoning I was given for abandoning Lua was twofold:

* most of the high-performance execution in this field happens in the
GPU anyway; using LuaJIT is not such a big win as opposed to Python
* Python's more mature ecosystem with the gazillion libraries for
visualization etc already available

(This is my recollection of the exchange, please don't quote them on this.)

> Add to that the (again) chicken-and-egg problem that few people
> use Lua hence few people *teach* Lua at universities (as opposed
> to Python which is on basically every curriculum now)...

Funnily enough, that includes PUC-Rio, which has Python in its
standard curriculum but not Lua. (Granted, Lua is offered as an
elective course every now and then, but Python is CS101).

>> "The first argument to string.pack, string.packsize, and string.unpack
>> is a format string, which describes the layout of the structure being
>> created or read."
>>
>> What structure? Not even an example is given! And Lua hasn't got any
>> "structure" datatype. Yes, I know that is some kind of inclusion of
>> Roberto's "struct" library. Of course a C programmer or someone using
>> protocols could discover this, but another wink to C programmers and
>> more unfriendliness toward a novice Lua programmer that is not already a
>> C programmer. Another very specialized thing (not really a "general
>> mechanism") that seems well fit for an external library that gets in the
>> language, instead. Moreover it is a fairly low-level mechanism. Another
>> thing that doesn't appeal to "higher level" programmers. Why?
>
> Well, to parse or generate any kind of real world data? Network
> protocols, file formats...

Lorenzo's examples do paint an interesting picture that I haven't
visualized as clearly before: the evolution of the Lua standard
library does seem to be biased towards the needs of embedded systems.
Which is a good bet, market-wise. There will always be a niche for
resource-constrained embedded systems where the computing budget is as
tight as possible: some applications that required a custom IC decades
ago may be doable today with a RPi running a full Linux distro, but
some new applications that were unthinkable back then are now doable
with a penny-sized computer where CPU and memory are still at a
premium. From credit-card sized, to penny-sized, to
grain-of-sand-sized, the point will still stand.

But it's understandable that many of us share this sort of frustration
about the language's popularity, because it is a great design as far
as languages go. It's the same feeling as turning on the radio and
hearing whatever is #1 at the moment instead of our favorite music
genres. Some people go "eww", some go "why world, why??" and some feel
good about themselves because they consider themselves the hipsters
with better taste than the mainstream. :)

-- Hisham