Re: Make "package.loaded" weak

[Francisco Olarte <folarte@...>]

On Sun, Feb 19, 2017 at 3:24 AM, Martin <eden_martin_fuhrspam@gmx.de> wrote:
> I offer to make values of table "package.loaded" weak to free
> unreachable data at garbage collection cycle.

I do not think this is nice. In some of my code I explicitly do some
requires for the sole purpose of populating package.loaded, and
sometimes I populate it directly ( although I'm switching to
package.preload which is cleaner ).

> My usage case is many (over 350) small (about 30 lines) files which
> is lua code which typically returns table or function. Conceptual
> problem is that result of each of that loaded files occupies
> "package.loaded" forever. I think it is not very clean from abstract
> point of view.

Your problem seems to be in the design of your code. If you do not
know the required lifetime of the result of the code, return a
function which calculates it.

> >From the other side I understand that after making "package.loaded"
> weak, multiple loads of the same code may occur. For my usage case
> this is not a problem, but you guys may have serious objections
> which I'd like to hear.

I would not like lua recompiling my modules each time I require them.
I think your very peculiar case is better solved by nil-ing
package.loaded when needed.

> Here is artificial example.

Certainly artificial and purposely built to try to make a point.

> Suppose we have code chunk that creates some big table and uses it
> in work:
>   local do_work =
>     function()
>       local result = {}
>       for i = 1, 1e6 do result[i] = {} end
>       return result
>     end

FALSE, it creates (uses it in work ) AND returns a big table. Using it
to work will be something like an array used in sieving for primes,
which gets discarded when the primes list is returned.

>   print(collectgarbage('count'))
>   do_work()

So you make a function which sole purpose is calculating a big array
and then discard its result?

>   collectgarbage()
>
>   print(collectgarbage('count'))
>
> It does not create memory problems yet.



> But now we wish to move do_work() to separate module
>   --[[ "garbage_making_module" ]]--
>   local do_work =
>     function()
>       local result = {}
>       for i = 1, 1e6 do
>         result[i] = {}
>       end
>       return result
>     end
>
>   return do_work()

BAD dessign, you shall return do_work, so caller chooses calling time
( and wheter to keep the result as in the example above )

>   --[[ main ]]--
>   print(collectgarbage('count'))
>   do
>     local garbage = require('garbage_making_module')

And here do require(...)(), call require return value.

>   end
>   collectgarbage()
>   print(collectgarbage('count'))

And the result would be the same.

> Now no memory is freed!

Neither would be if you had stored do_work() result above. I do not
think this code proves any potential problem with require. You may
have that pattern in some code due to technical debt, or impositions
from above or whichever, but I do not think modifying package.loaded
is the way to go. Where I forced to use some untouchable modules
writen as above I would probably do a weak_require wrapper to load it
without caching, seemd easy to do by just requiring and erasing from
p.l before return or just using package.searchers.

Summarising, I do not think making package.loaded weak is a good idea,
and I think your example does not merit it. Maybe I could be convinced
by a better example.

Francisco Olarte.