Re: Let deprecate to-be-close variables as harmful concept

[Francisco Olarte <folarte@...>]

Hi Lars.
On Tue, 3 Oct 2023 at 14:16, Lars Müller <appgurulars@gmx.de> wrote:
> I've roughly seen four concepts for resource management:
> (0. No language support, do it yourself. Error-prone; often only code for the "happy path" is written (so errors that don't terminate the application result in resource leaks).)
> 1. "with" blocks. In Python this is a statement with a block scope. In languages like Lua, this can be implemented using closures (for example, in Lua 5.1 I use a with_open akin to Scheme's with-(input|output)-(from|to)-file).
> 2. "defer" statements, as found e.g. in Go or Zig, which run in reverse order when the function (Go) or scope (Zig) is left (normally or with an error).
> 3. SBRM (Scope-Bound Resource Management) like Lua's to-be-closed variables: Resources are released when the corresponding variables go out of scope.

Those are similar, they end up producing the same results.

> 4. Reference counting, which ensures that resources are released when they aren't needed any more, no matter how contrived the program logic. Works as long as there are no reference cycles among resources (which there practically almost never are). The special case of 0-1-reference counting can to a large extent be done by "ownership" models like Rust's, but those are very much out of scope for Lua.

To do this you need to manage the "pointer" object with some of the
above techniques. In lua you can make a ref-pointer class to point to
a ref-counted and manage it with close. In C++ you just use shared
ptr. In perl every reference is a shared ptr. The problems is if you
leak the pointer, by forgetting the lua <close>, stashing the perl ref
somewhere long lived or putting the c++ shared ptr in a heap allocated
object. In the end, you need some discipline for resource management.

Linux kernels use option 0, IIRC, and manage to have extremely low leaks.


> I think 3. is the best option for Lua here:

In the end, 1 is 3 with the scope defined by the with, and not sure of
defer, but it seems like 3 with scope defined by the defer-enclosing
function/block. Or 2/3 are 1 with an auto-generated with. Or 3 is 2
with an autogenerated defer and 1 can be implemented using 2. Thay are
all what I like to call syntactic sugar for try-finally.

4 is the odd one, as it reduces to manage a ref-countinf ptr using one
of the above.

> 1. Becomes awkward as the more resources you use, the deeper your nesting goes. The closure-based approach has the shortcomings of function usage (for example not allowing break).
> 2. This is a pretty decent solution; it is slightly more verbose however (another statement). I also don't like tying this to block- or function scope; this means in practice resources will live longer than they have to if the programmer (say, the function/block does some expensive computation after being done with the resource; the resource will only be released after that). It has the small advantage vs. to-be-closed variables that you need less boilerplate if you have cleanup code that naturally isn't tied to an object.

the expensive computation stuff happens also with 3 or 1, is just a
case of bad scoping.

> 3. The best solution for Lua, IMO. Very concise - just add "<close>" after the resource variable name. Definitely much better than the error-prone "local f = io.open(...); [...]; f:close()" antipattern, which won't clean up immediately if the "[...]" throws an error that is caught later on. Let's the compiler figure out for you when you last use a variable (and when it is thus safe to clean it up).

> 4. Not really an option for Lua, since Lua doesn't base its garbage collection on reference counting; not using RC is presumably also better for performance, and is simpler than having to use something like a RC + tracing/generational GC approach to collect reference cycles.

It needs work, but reference counting is good specially for heavy,
very shared objects with have non-trivial closing behaviour, like
database connections or connection pools. I use it in Lua and Java,
just takes a little discipline. First do a dtabase connection class
with an ( idempotent ) close() method. Then make _close() call close
in case you want to use it one-shot. Then make __gc call close() in
case you forget to close(). Then make a ref_count_holder class with a
.pointed and .ref_count members, and a close() method which, if
pointed is not nil, clears it and calls its close() method ( typically
with a local-var swap to avoid problems ). Add ref() method with
increments ref-count and deref which decrements and close()s. Then
make a ref_pointer_class with takes a ref_count_holder on creation,
stores it and calls ref(), give it a get() method which returns the
holders .pointed, give it a close() which clears the ref_count_holder
and calls deref() and make __close() and __gc() call close(). That can
be done in half-page and makes ref-counting work unless you do strange
things. Use it for classes like the above which do not contain loops,
many of the heavy classes are just trees, or its dependencies can be
made trees with judicious use of weak pointers. And with a little work
ref-counted can be used as a "base class" to save some indirections.

This allows me to get a db_connection() in a one-shot, store it in a
global and close() it at end. Or just let it __gc if lazy.
Or get it in a local <close> in the entry method of a complex
operation and pass it around.
Or store a ref-counted in some structure and acquire extra references
via a clone method in the ref-pointer which just returns a new
ref-pointer for the pointed and pass them around in complex things
which do not know exactly when every one is done.

It can be made to fail, but has its uses, and there are not many
alternatives when you have heavy objects passed around in complex
processes.

....

> The only main downside of to-be-closed variables I see at the moment is that it interferes with tail calls. I think the simplest solution would be to always close all to-be-closed variables before tail calling, preserving the simple rule that "return f(...)" is always a tail call.

this has a problem with
   local f <close> = io.open(...)
   return do_stuff_with_may_error(f,other_things)

which is an idiom, for me at least. But when this pattern is used I do
not normally rely on a tail call.

> In principle, reference counting is the cleanest solution in my opinion, but it would come at a performance cost and much more complexity than to-be-closed variables.

Reference counting, a particular case of shared ownership, can be
implemented in the language, like C++ does shared_ptr using plain
destruction.

I agree with you regarding <close>. Nice feature, light to use,
totally optional, solves a huge fraction of the problems easily.

Francisco Olarte.