Re: to-be-closed: the case for terminating exceptions

On Sun, Oct 23, 2022 at 9:42 AM John Belmonte <john@neggie.net> wrote:

I tried to implement this patch myself, with the intention of using it to continue the "Lua and structured concurrency" article.

After many hours, I wasn't able to find a surgical change that would do it. I'm hoping Roberto would provide some guidance on how it might be implemented.

The initial problem is that, once `callclosemethod()` is changed to expect a return value from `__close()`, it's quite hard to capture that value and restore the stack in the case where `__close()` yields. I thought it might be done in `finishpcallk()`, but couldn't find a way to detect that a `__close()` call had actually returned in that context.

On Thu, Oct 13, 2022 at 9:30 AM John Belmonte <john@neggie.net> wrote:
The ability to have __close() suppress an exception is useful -- more so than I thought.

I'll leave it to Roberto to judge whether this is practical, but as far as API, I think it would be:

"If __close() returns the specific value `false`, then the exception is terminated, and execution continues normally, following the exited scope (after __close of remaining to-be-closed variables)." Since existing __close() instances are likely to return nothing (`nil`), this behavior is backwards-compatible.

Some example use cases follow. Of course, pcall() is always a crude replacement, but the presumption is that maintaining return/break/goto within the code block, and avoiding multiple levels of nested lambdas, is highly desirable.

1. ignoring an exception

do
local scope <close> = open_move_on_if_error()
-- .. some code that may raise an error ..
end
-- .. continue here if there was an exception ..

If an application has some exception hierarchy, then `open_move_on_if_error()` can optionally accept a base exception type.

2. asserting an error in unit testing

do
local scope <close> = open_assert_error()
-- .. some code that is expected to raise an error ..
end

An assert error is raised out of the do/end block if no exception was encountered. Here, too, there's the option to pass a base exception type into `open_assert_error()`.

3. cancel scopes

It's very useful to declare a scope around arbitrary sequences of asynchronous code, such that execution of the block can be cancelled at any coroutine resume point. There are numerous uses:

3.a timeouts

do
local scope <close> = open_move_on_after_seconds(5)
-- .. any code that includes transitive yields ..
end
-- .. continue here if timeout reached

At each yield, the scheduler will check if the deadline has been exceeded, and raise an exception, which propagates a Cancelled exception up through the task hierarchy, finally terminated in __close(). (Assumes the recently proposed structured concurrency framework.)

3.b cancellation by event

do
local scope <close> = open_move_on_when(event.await)
-- .. any code that includes transitive yields ..
end
-- .. continue here if event.await() returns

3.c manual cancellation

do
local scope <close> = open_cancel_scope()
-- .. any code that includes transitive yields ..
if (some_condition) then
scope.cancel() -- request cancel of this scope
end
end

Of course, the cancel() call may be encountered transitively within the scope of the block (nested function or task), as well as being invoked by some task outside the hierarchy, which happens to have a handle to the scope object.

Actually, the recently proposed nurseries fall into this use case. When a nursery is cancelled, the body of the nursery itself needs to exit promptly, despite being blocked on sleep or I/O.

As to-be-closed stands, with no way to terminate exceptions, the next installment of "Structured concurrency and Lua" may be the end of the road for now. It will introduce cancel scopes, but explain that the nursery implementation is incomplete, and cancel scopes aren't implemented at all -- pending some solution to terminating exceptions.