Re: Lua distros again

[Pierre Chapuis <catwell@...>]

On Tue, Feb 13, 2018, at 11:46, Lorenzo Donati wrote:

> Therefore he really downloads libraries that he cannot (by lack of 
> expertise or time) roll by himself. LFS is a clear example. It is 
> symptomatic that no string or table library comes first.

In other words, users download libraries written in C that deal with
platform-specific stuff, not small Lua snippets like in the Node.JS
ecosystem.

> This, IMHO, is /the/ big problem with Lua ecosystem. The most common 
> libraries are actually not published because they are often half-baked 
> solutions that are too author-specific and not worth maintaining in a 
> public way (who wants to support his own string library if it hasn't 
> enough traction from the community?)

Well, what do you expect from a "string library"? Because there is
a standard string library that ships with the language...

> It should seem strange that more general-purpose libraries are /not/ the 
> most downloaded ("penlight", "stdlib"), as if people didn't usually use 
> those functionalities.

I use Penlight, but mostly as a debugging tool (especially `pretty`), not
something I ship with my projects.

> After years of following Lua-l I've seen this discussion coming up 
> several times. And every time died off or gave rise to initiatives that 
> lasted just for a short while or didn't gain enough traction to become 
> de-facto standards. Every time.
> 
> Sadly I think this is a direct consequence of Lua team not wanting to 
> endorse any specific initiative.
> 
> I understand their decision of not wanting to create and maintain a "PUC 
> standard" library collection, and I also understand their attitude 
> toward backwards compatibility. That's ok. It's Lua team's way.
> 
> But there is a big "but": not endorsing, say, a charter of committed 
> people that took responsibility for creating a basic set of standard 
> library was and is a big mistake for the future of Lua, IMO.

I understand the sentiment, I thought the same 5-10 years ago,
I don't anymore. Maybe it's a chichen-and-egg project but I don't
think it is easy to "bless" a project that hasn't emerged as a clear
winner from the community itself. Moreover, even if the Lua team
(basically Roberto in this case I guess) did that, what effect would
it have? I mean, LuaJIT is still not implementing Lua 5.3, is it?

It makes sense that the community 

> I still remember some years ago when Lua was considered (I can't recall 
> exactly the source) one of the three major scripting language, together 
> with Python and Javascript.

Well, I'd be happy to know the source for that and how many years
"some" is, because I don't think that was ever the case for most
people's definition of "scripting". Lua has always had very strong
niches (games, embedded...), more and more recently (with e.g.
Software Defined Networking and Machine Learning) but I can't
think of a time where it was much higher in mindshare than now.

> I fear we are steadily losing terrain. I haven't hard data to back up 
> this feeling, so I may well stand to be corrected (and I'd like to be, 
> really), but I see lots of news about Python gaining ever more users 
> among scientists and non-programmers, whereas nothing about
> Lua outside  very specific "circles".
>
> Mind, I don't think Lua user base is shrinking, on the contrary, I think 
> it is increasing, but much, much more slowly than other languages.
> This means that a widespread diffusion of Lua (as was the intention of 
> their authors, at least years ago) is not going to happen, IMO.

Scientific people is a specific case IMO, related to ML frameworks.
I have been pretty much involved in this so how is how I see things:

- Once upon a time people were using various things including
  Matlab, R, Python (Scipy / Numpy), C++, etc.
- Deep Learning happened, and the market split between three
  major frameworks: Caffe (Java), Theano (Python) and Torch7 (LuaJIT).
- Important companies like Facebook and Twitter chose Torch7 so Lua
  grew in popularity.
- Some people at Google were not happy because Lua is not one of
  their blessed languages, but even their own team was using it
  nevertheless (AlphaGo).
- They pushed for new C++ / Python frameworks with TensorFlow
  and Keras.
- Meanwhile some people who preferred Python to Lua made
  PyTorch which got a lot of adoption too.

So tl;dr Lua had a short-lived bubble in that space because of one
framework, but now it's not the only one on the stage anymore so
Lua usage is going back down.

Add to that the (again) chicken-and-egg problem that few people
use Lua hence few people *teach* Lua at universities (as opposed
to Python which is on basically every curriculum now)...

> Even when thinking about embedding a language in an application, where 
> Lua should be first choice, I hear people talking about Python and using 
> Python (urgh!)! Sometimes because they don't even know Lua, and 
> sometimes because they need better supported libraries, so they rule-out 
> Lua!

Have you already embedded Python, packaged it and shipped it to people?
Because I have done it and I can tell you using most of the C-based modules
on the package index is much, much harder than in Lua.

> I didn't even understand the rationale behind the latest changes in the 
> language. I did appreciate the addition of "goto" in 5.2 and bitwise 
> operators in 5.3, but, really, what's the use of the utf8 library in a 
> language that doesn't support unicode natively (relying on underlying C 
> reading files in text mode - I know that in Linux that doesn't matter- 
> but Windows is not a marginal system)? The minimalistic mantra of 
> keeping unneeded stuff out of the language wasn't applied here. (why?) 
> Utf8 was an ideal candidate for an external library.

I think UTF-8 is a very good candidate for an internal library, but I agree
that it didn't go far enough in supporting Windows (but that has already
been discussed in other threads).

> Lua team stripped the math library of world-standard hyperbolic 
> functions because they were little used (?!?) and they were a simple 
> forwarding to C libs. "OK, if you are a C programmer you can re-add 
> them, or you can implement them using exp". Sadly who usually needs the 
> optimized C versions that map to math processor instructions 
> (scientists) is typically /not/ a C programmer, whereas a C programmer 
> rarely uses hyperbolic functions!

Well, scientific stuff is a very good candidate for an external library...
as it is the case in Python by the way. We don't have SciPy / NumPy
in Lua but is that the core team's fault?

> Then also the string.pack/unpack/packsize, things. Why? That may make 
> sense to some, but is it really a "general mechanism"? For what? The 
> manual not even explains what is going on:
> 
> "The first argument to string.pack, string.packsize, and string.unpack 
> is a format string, which describes the layout of the structure being 
> created or read."
> 
> What structure? Not even an example is given! And Lua hasn't got any 
> "structure" datatype. Yes, I know that is some kind of inclusion of 
> Roberto's "struct" library. Of course a C programmer or someone using 
> protocols could discover this, but another wink to C programmers and 
> more unfriendliness toward a novice Lua programmer that is not already a 
> C programmer. Another very specialized thing (not really a "general 
> mechanism") that seems well fit for an external library that gets in the 
> language, instead. Moreover it is a fairly low-level mechanism. Another 
> thing that doesn't appeal to "higher level" programmers. Why?

Well, to parse or generate any kind of real world data? Network
protocols, file formats... Again, look at Python: the exact same library
is called "struct" and is included in the standard library. Moreover
both "struct" and "pack" were some of the most used libraries
in former Lua versions.

> Judging by lua-l alone there are more attempts at creating 
> customized/patched/jitted Lua variants than to create a set of 
> standardized libraries having the consensus of the community!

There *is* a standard library. You just don't like what's in it.

Again, I am not saying you are wrong. I used to think like you,
and now I don't anymore. I am extremely happy with the additions
in 5.3 (64 bit integers, binary operators and string.{un,}pack)
which cater to the "networks and systems" niche. The only
thing I still miss is a much better defined standard way to deal
with sequences (i.e. "I have gotten a table from somewhere,
is it a list or a map?"), but otherwise I think the community can
do the job, and is doing it judging from the increase in number of
modules on LuaRocks.

That being said, on the community end, we are still bad at
making the core libraries and tools we all need. We have several
package managers, network libraries, crypto libraries... and most
of them do not cover all platforms and use cases. And that's the
curse of Lua not being a generic programming language: we don't
have people paid to (as part of their job) maintain libraries which
do one of those things well *in all cases*. Writing a (C-based) library
that works well on, say, only Windows is easy and requires few
resources (a Windows machine). But when you want to support
Linux, MacOS, iOS, Android etc as well, you incur costs in hardware
and time that can't be justified in many cases.

I think that's the problem we have to solve, and I don't know how
we can do it. But I don't mind that everyone has their 150 lines
table / string library anymore.

-- 
Pierre Chapuis