Re: modules, require, magic

[Hisham <hisham.hm@...>]

I'll try to condense some replies into one, in order to avoid
spreading out this thread much further.

On Mon, Oct 17, 2011 at 7:11 PM, Luiz Henrique de Figueiredo
<lhf@tecgraf.puc-rio.br> wrote:
>> My opinion is that modules are important enough so that there should
>> be a "blessed" way of writing modules, and preferrably a somewhat
>> obvious one. There has to be a policy.
>
> The recommended way of writing modules is the trivial way: a module should
> return a table with its data and functions. It'd be nice if it also did
> not pollute the global environment. In Lua, this is done trivially with:
>
> local M={}
> M.version="1.0.2"
> function M.f() ... end
> ...
> return M
>
> This way works fine for both 5.1 and 5.2

In fact, that works fine for 5.0 as well. But if it was not enough for
encouraging reusable code back then (and module() was created in hope
to "remedy this situation", according to the HOPL paper quoted below),
then why would it be for 5.2?

On Tue, Oct 18, 2011 at 3:53 AM, David Manura <dm.lua@math2.org> wrote:
> On Mon, Oct 17, 2011 at 1:00 PM, Sam Roberts <vieuxtech@gmail.com> wrote:
>> I like module(), in particular, I like the behaviour that it sets up
>> globally reachable names for modules, and I like that I can call it
>> like module(...,package.seeall), and while containing no internal
>> reference to its actual name, it will build itself into the global
>> namespace depending on where its deployed in the filesystem.
>
> I understand other pro-module users would not agree on those points.
> Petite Abeille and Hisham said (or suggested) it's ok to get rid of
> luaL_pushmodule, and Hisham argued the benefits of explicitly writing
> out the name to module.
>
> Personally, I've thought that if you do want to design the module
> system to automatically set up global names, this responsibility
> belongs more inside a function loading the module (like require) than
> a function defining the module (like the module function).

Correct. I think it's good to stop auto-declaring globals (if Sam needs
a globally reachable name for a module (for inspection purposes or
something similar, there's package.loaded["modname"]) and I don't like
the module(...) idiom.

>> The require() function defines a namespace, a namespace that is global
>> to the lua state, inherited from the filesystem, and in which
>> conflicts must be avoided
>
> Mark Hamburg had argued that the main problem is not conflicts but
> rather hidden dependencies:
>  http://lua-users.org/lists/lua-l/2006-04/msg00547.html
> Improving visibility, I believe, follows in the same vein as some of
> Hisham's recent comments concerning passing an explicit module name
> and other things.

Yes.

On Tue, Oct 18, 2011 at 4:37 AM, Mark Hamburg <mark@grubmah.com> wrote:
> Let's run through what module does.
>
> 1. It handles constructing module tables including the fields _NAME, _M,
> and _PACKAGE. This is useful if one cares about these fields, but I note that
> they don't appear in the pre-defined modules like table and string.
>
> 2. It handles constructing submodules. I might question whether it makes
> sense to have socket.http available when socket is potentially just a stub
> table, but that's a more complicated matter.
>
> 3. It provides early registration for the module. This helps with mutually
> recursive requires, but it also means that require can return an incomplete
> module.
>
> 4. It adds the module to the global namespace. This, in my opinion, is a
> bad thing because it creates hidden dependencies -- i.e., code can use a
> module that it never required simply because some other code required
> it earlier and it became available in the global namespace.
>
> 5. It mucks with the environment to make it "easy" to export functions.
> But then to compensate for this, it offers package.seeall which results in
> a module that reveals a lot of globals in its table which have nothing to
> do with the module -- i.e., it pollutes the API for the module.
>
> I think it would be useful to hear which of these behaviors is deemed so
> valuable that people are fighting for it.

I'm indifferent to #1, I think #2 is nice for consistency but not
critical, never been bitten by any problems in #3, and agree that #4
is a bad thing.

I believe the _intention_ behind #5 is why module() was created: a
function so that you can declare a module and then what you code in
that file is the module. This is analogous to what we see in other
languages.

For that reason, I think #5 produces the clearest programming idiom.
Distinguishing private and public functions with "local function" and
"function" is very logical, if you think of "module" abstractly, as a
language feature.

The way it is implemented, with package.seeall, etc, is unfortunate.
Maybe that's what Mike Pall means when he says that module "has not
held up to its promise". But the promise was good: module() was
created with the goal of filling and existing need, and that's why
even with its flaws, it was a success.

To add to David Manura's statistics, here is a list of packages in the
LuaRocks repository that include Lua code (I unpacked all sources,
grepped and inspected the results by hand). The vast majority uses
module() (and are not Kepler modules, to get that out of the way).

abelhas: YES
alien: YES
alt-getopt: YES
bencode: YES
cgilua: YES
colors: YES
concurrentlua: YES
config: YES
copas: YES
cosmo: YES
cue: YES
dado: YES
diff: YES
dkjason: no
fbclient: YES
flu: no
htk: YES
json4lua: YES
lanes: YES
leg: YES
loop: YES
lpeg: no (YES, up to 0.9)
lrexlib-pcre: YES
lsqlite3: no
ltcltk: YES
lua-aplicado: no
lua-apr: no
luabitop: no
lua-coat: YES
lua-codegen: YES
luacov: YES
lua-discout: YES
luadoc: YES
lua-espeak: YES
lua-ev: YES
luaexpat: YES
luafft: no (used to)
lua-geoip: no
luagraph: YES
luahaml: YES
luahtml: YES
luaidl: YES
lua-imlub2: YES
luajson: YES
lua-llthreads: no
lualogging: YES
luamacro: no
luanotify: YES
lua-nucleo: no
luapod: YES
luaposix: YES
luapsql: YES
luasec: YES
luasoap: YES
luasocket: YES
luasofia: YES
luasolidstate: YES
lua-testlongstring: YES
lua-testmore: YES
luatexts: no
lua-tinycdb: YES
luaxml: YES
lua-xmlreader: no
lua-yajl: YES
lua-zmq: YES
lua-zmq-threads: YES
luchia: YES
luma: YES
lunary-optim: YES
lunatest: YES
lunit: YES
luse: YES
lxsh: no
markdown: no
math-evol: no
math-rungekutta: no
math-walshtransform: no
md5: YES
midi: no
midialsa: no
nixio: YES
oauth: YES
objectlua: YES
oil: YES
orbit: YES
penlight: YES
petrodoc: no
recaptcha: YES
redis-lua: YES
remdebug: YES
rings: YES
sha2: YES
shake: YES
simulua: YES
sociallua: YES
sputnik: YES
stdlib: YES
tamale: YES
tango-complete: YES
tango-copas: YES
telescope: YES
tethys: YES
tlua: YES
twitter: YES
validate-args: YES
vararg: YES
wsapi: YES
wsapi-fcgi: YES
xavante: YES
xssfilter: YES

In terms of the package.seeall issue, what I would consider
"intuitive" (for my intuition at least) would be to have only globals
declared by the module accessible through it, but have available in
its environment the same base library one sees at the beginning of a
Lua program.

> Oh. And on the documentation front, the "module( ... )" idiom obviously
> fails to document the module name and comments are another good
> way to write documentation.

Yes (and I dislike "module(...)"), but self-documented code is better.
Languages have high-level constructs for that reason, or else we'd
only use goto and instead of structured programming we'd just comment
our loops. :)

>From what I've seen of the non-module() users, some create modules
declaring a table called "Modname", some "M", some "_M" (wasn't there
some recommendation against using names with underscores followed by
capitals?). Some declaring those tables in the end, some on top, some
as a big table declaration spanning the whole file... some pollute the
global namespace, some don't. I've seen some polluting the global
namespace with names different from that of the module. Was it
require("luagl") that created a global table GL? I don't remember
precisely, but I've seen things like that. Without a module() function
that enforces some proper behaviors it's likely we'll start to see
things like that more and more. If we don't like the current behavior
of module(), let's fix it, but please don't throw out the baby with
the bath water.

Discussing this offlist with Fabio he also mentioned another benefit
of module()-style modules: that one can easily inspect the code
mechanically for global declarations and infer what's exported, while
module()-less-modules make this kind of static analysis a lot more
fragile.

Petite Abeille wrote:
> Paradoxical, the most articulated proponent of module is Roberto
> himself, before he had a change of heart and, sadly,  turned to the
> dark side:
>
> "Despite our “mechanisms, not policy” rule — which we have found
> valuable in guiding the evolution of Lua — we should have provided
> a precise set of policies for modules and packages earlier. The lack
> of a common policy for building modules and installing packages
> prevents different groups from sharing code and discourages the
> development of a community code base. Lua 5.1 provides a set of
> policies for modules and packages that we hope will remedy this
> situation." [1]
>
> "Usually, Lua does not set policies. Instead, Lua provides mechanisms
> that are powerful enough for groups of developers to implement the
> policies that best suit them. However, this approach does not work
> well for modules. One of the main goals of a module system is to allow
> different groups to share code. The lack of a common policy impedes
> this sharing." [2]
>
> The young and idealistic Roberto will be fondly  remembered by the many mourning little modules he is leaving
> behind.

PA's remarks are often a bit too dry for my taste, but this one made
me smile. :)

But apart from the humor, indeed, the first quote is right on the
mark. If "we should have provided a precise set of policies for
modules and packages earlier", then getting rid of module() altogether
is a step backwards.

As you can see from the modules list above, the community code base is
still in its infancy compared to repositories from other languages.
The Lua community is still trying to catch up with the lack of module
policies that held it back in the past. A loose recommendation to
create modules such-and-such way in Lua 5.0 days was clearly not
enough, hence the creation of module(), and once a more precise policy
promoted by a standard library function was offered, it was widely
adopted. If that function has policy problems in its implementation,
please fix the function instead of just getting rid of it and
returning things back to their previous state, which "prevents
different groups from sharing code and discourages the development of
a community code base".

To sum it all up: I agree with the concerns voiced against module() by
the anti-module camp. At least some of the pro's I've mentioned (or at
least the motivation to have a module declaration function) were
agreed with by the anti-module proponents. I'd like us to try to meet
halfway, and perhaps to show the Lua authors that the intentions they
had when they wrote those paragraphs above were indeed valid.

In terms of "fixing module()", I think Fabio's implementation is right
on the money:

http://article.gmane.org/gmane.comp.lang.lua.general/84283

It shows that a sane implementation of module() can be done with
standard Lua mechanisms. Do any of the anti-module proponents see any
problems with that design? Would you hate module() as much as you do
now if it behaved like that instead of the way it behaves in Lua 5.1?

-- Hisham
http://hisham.hm/ - http://luarocks.org/