Re: (not) handling new programming idioms with grace

[Hisham <h@...>]

On 20 July 2018 at 13:33, Roberto Ierusalimschy <roberto@inf.puc-rio.br> wrote:
>> > I remain curious. Has this been tested and found too slow before?
>>
>> Yes. I can't provide a reference right now but relying on
>> collectgarbage() to force release of non-memory resources via __gc at
>> the necessary times has proven to be unusable for my workload pattern.
>> It incurred a significant performance hit due to unnecessary memory
>> traversal.
>
> I am still curious whether that could work as I suggested, calling
> __gc as a kind of emergency collection, only when some resource
> allocation fails. For instance, in the case of file handles in Linux,
> that would mean ~1 full GC for every 1000 streams created, not
> one GC per stream.

For LuaRocks traversing a directory that would have been okay, but for
any kind of resource being used by a server, doing a full GC every N
requests (whether N = 100, 1000 or 10000) would produce pretty
noticeable spikes in the performance graphs. We're talking about
significantly degraded performance (a full stop-the-world GC that
wouldn't ever otherwise happen given incremental/generational GC,
right?) on one out of every N requests. I imagine that running a full
GC during a game frame would also be frowned upon.

Emergency-collecting on io.open hides the problem in that one
particular case which happens to be the easy-to-understand go-to
example for advocating deterministic resource release, but it doesn't
sound like an approach that one would want to do on other, even more
limited resources, such as DB connection handles, or ones that should
always be released ASAP even if the resource isn't exhausted, such as
sockets — one might argue that sockets should always be explicitly
closed and never left to the GC, but then libraries such as LuaSocket
follow the example of Lua's io library and implement __gc for sockets,
as if to say that this is okay.

If the emergency collection became the standard behavior of io.open I
suspect that "beware that opening a file in Lua may trigger a full GC"
would become one of those dreaded folk-wisdom performance tips.

-- Hisham