Re: error() or nil, 'error msg'

[Sean Conner <sean@...>]

It was thus said that the Great William Ahern once stated:
> On Thu, Sep 11, 2014 at 03:38:23PM -0400, Sean Conner wrote:
> > It was thus said that the Great William Ahern once stated:
> > > On Thu, Sep 11, 2014 at 09:50:29AM -0400, Sean Conner wrote:
> > > > It was thus said that the Great Andrew Starks once stated:
> > > <snip>
> > > > > Are there interesting differences between how you think about / deal with
> > > > > error handling in C/C++ vs. Lua?  Accepting that the mechanisms are
> > > > > different, do you do more "try to recover" code in C?
> > > > 
> > > >   I don't program in C++, so I don't use exceptions.  In regard to error
> > > > handling, I tend to handle errors in Lua like I do in C.  And I don't really
> > > > do "recover" code in C.  [2][3]
> > > > 
> > > 
> > > What do you mean by recover? Surely you handle errors like EMFILE and ENOMEM
> > > without exiting the process.
> > 
> >   How would *you* handle ENOMEM?  
> 
> The same way I handle EMFILE. Unwind to a steady state.
> 
> If I have 5,000 open connections, and connection 5,001 hits ENOMEM (maybe
> the client sent me a huge JSON file that, after parsing, exploded to a
> gigabyte), why would I sabotage the other 5,000 connections?
> 
> FWIW, I also disable swap on my servers in addition to disabling overcommit.
> I'm not going to leave quality of service to chance. In many cases excessive
> swapping is even worse than the process aborting. If I don't have the
> resources to perform a particular task, why would I let that kill every
> other task?

  If it's Linux, then disabling swap does *NOT* prevent overcommiting
memory.  You might want to read

	https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

for a bit more information.  Also, try this program:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
    printf("Before\n");
    int i = 0;
    while (malloc(1 << 30)) {
        printf("Allocated %d GB\n", ++i);
    }
    printf("After\n");

    return 0;
}

  On a 64-bit Linux system, it "allocates" 131,071 gigabytes (128 terabytes)
of memory before failing.  I *wish* I had that much memory.

> >   I wrote a big on this a few years ago:
> > 
> > 	http://boston.conman.org/2009/12/01.2
> 
> Your example is something of a strawman because even the original is broken
> in several ways. For one thing, it leaks the listen object. So you weren't
> bothering to try to write useful code.

  And there's a follow-up to that post that describes that very bug:

	http://boston.conman.org/2009/12/02.1

  It's even linked to in the original post.

> Also, you're using a global epoll descriptor. 

  It's an example.  A full example would swamp the point I was trying to
make with a sea of (in my opinion at the time) unnecessary detail.  Guess I
was wrong with that one.

> I point that out because, yes,
> in many cases it can make sense to bail on recoverable errors (e.g.
> command-line programs, forked server models where you have one process per
> session, etc), but rarely in _library_ code, and library code shouldn't be
> using global objects. Context matters. I only object to the attitude that
> applications should bail on ENOMEM no matter the context. 

  The original post I wrote: "better exit the program as quickly and cleanly
as possible because something bad, like ENOMEM (insuffient kernel memory)
just happened ...."  I wrote "*like* ENOMEM," presenting it as an example. 
If you have a better example, by all means, please tell me.  Because ENOMEM
can pop up in quite a few places (fork(), for example, or open()).

> It takes skill to
> deal with ENOMEM efficiently, and that skill will never be acquired if one
> takes the attitude from the outset that it's not practical to do so. Nobody
> ever said programming was easy--dealing with ENOMEM sometimes requirings
> using patterns you wouldn't otherwise, but eventually you grow to appreciate
> them.
> 
> In any event, your example represents the worst case. In even a large
> application such an example would most likely be one of the most complex. As
> I said before, using RAII patterns you confine and isolate logic like that
> to only a handful of places. Then if you strive to write failure-free
> interfaces as much as possible--such as using data structures that don't use
> dynamic allocation--you reduce even further the amount of error checking
> sprinkled around your code.
> 
> >   And depending upon the context, ENOMEM may be a case 3 (per the post) or
> > case 4.  I generally just stop processing if I get ENOMEM since I usually
> > have no idea how to handle the situation.
> 
> How do you handle EMFILE?

  I don't explicitly check for EMFILE (I've never actually encountered that
error to be truthful).  But looking over the code I have written, what will
happen is that the error (most likely from accept()) is logged and the code
continues on.  The client that attempted to connect didn't get a connection.

  The only other place I can see it happening is in recursing down a file
system using opendir(), and depending upon the context, I may just exit the
program (if it's a command line program).  If it's important to handle huge
file trees, I might have to reconsider the approach I take.  

  It really depends on the goals of the program.

> 2) Sometimes I choose to use a red-black tree rather than an array. For
>    example, in my JSON C library with path-based querying, I use trees to
>    represent arrays. Each JSON object already has the tree node as a member,
>    which means once I allocate the JSON object structure, I know I won't
>    need any more dynamic allocation. I also allocate string nodes with one
>    allocation. Using such simple strategies I do at least 3x fewer
>    allocations than typical, naive JSON libraries. Which means it's 3x
>    easier for me to handle the ENOMEM case. And not coincidentally my JSON
>    library is faster than most, even though it's much more sophisticated
>    from a feature perspective, and I never really invested much effort into
>    performance.

  For my greylist server [1] I pre-allocate everything up front.  For my DNS
encoder/decoder library [2] the caller passes in the memory to use for
encoding/decoding the packet.  My syslog daemon [3] is primarily written in
Lua, so it handles the memory for me.  My Motorola 6809 emulator [4][5] is a
library, and I leave any memory issues up to the calling application to
handle.  

  It seems I try to avoid memory allocation errors.

  -spc

[1]	https://github.com/spc476/x-grey

[2]	https://github.com/spc476/SPCDNS

[3]	https://github.com/spc476/syslogintr

[4]	https://github.com/spc476/mc6809

[5]	I've even written a Lua wrapper for this emulator, although it's not
	included in the distribution.