Re: 5.4.0 Makefile patch for FreeBSD Was: Re: [ANN] Lua 5.4.0 (work1) now available

[William Ahern <william@...>]

On Wed, Mar 14, 2018 at 11:15:35PM -0700, Russell Haley wrote:
> On Wed, Mar 14, 2018 at 12:38 AM, Andrew Gierth <andrew@tao11.riddles.org.uk
> > wrote:
> 
> > >>>>> "Luiz" == Luiz Henrique de Figueiredo <lhf@tecgraf.puc-rio.br>
> > writes:
> >
> >  Luiz> All feedback welcome. Thanks.
> >
> > Building on freebsd doesn't work without adding CC=cc (since gcc is not
> > installed by default and if you install it from ports, it's usually
> > under a version-specific name such as gcc6), and also enabling readline
> > requires adding -I/usr/local/include and -L/usr/local/lib, or tweaking
> > things to use libedit instead.
> >
> > I've tried these, which all seem to work on my system (though obviously
> > the presence of a port build would make this somewhat moot):
> >
> > # with base system libedit:
> > freebsd:
> >         $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX
> > -DLUA_USE_READLINE -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit"
> >
> > # ... with readline from ports:
> > freebsd-readline:
> >         $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX
> > -DLUA_USE_READLINE -I/usr/local/include" SYSLIBS="-Wl,-E -L/usr/local/lib
> > -lreadline"
> >
> > # ... with editing disabled
> > freebsd-noedit:
> >         $(MAKE) $(ALL) CC="cc" SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E"
> >
> > and speaking of readline, it would still be good for lua.c to set
> > rl_readline_name and (maybe) rl_inhibit_completion as discussed
> > recently.
> >
> > Hi,
> 
> I'd be really pleased to see the Makefile patched for FreeBSD, and all the
> BSDs for 5.4. There were a few discussions last summer about what's
> actually required to build on FreeBSD, other BSDs, Mac and Solaris. I'd
> like to help out with the other BSDs, but I'm mostly interested in FreeBSD
> and Windows right now. On FreeBSD, an official patch would be great, but I
> could live with patching it myself in ports. Some items of interest to me:
> 
> - As Andrew nicely illustrated, libeditline is the standard readline
> package on FreeBSD.
> 
> - Andrew: luaconf.h indicates to me that LUA_USE_READLINE is not required
> when using LUA_USE_LINUX?  Also, -fPIC is required for Arm and was added to
> x86/amd64. Do you have any input on -fPIC? I understood why some time ago,
> but my understanding was pretty weak.
> 
> - There was a lengthy discussion (on my part. tee hee) which concluded that
> dlopen library (or whatever it's called) is only required on glibc. My less
> than stellar memory says that I concluded (potentially) LUA_USE_DLOPEN
> isn't needed on ANY of the BSDs or OSX. I also checked what Solaris
> documentation and found that dlopen was also not required there (not
> surprisingly).
> 
> Given the above, for a 5.4.0 port, I'd like to see something as close as
> possible to this:
> CC=cc
> ...
> $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_POSIX -DLUA_USE_READLINE -fPIC
> -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit"
> 
> OR
> 
> Perhaps the Makefile can be modified to have a single target (that FreeBSD,
> BSD, MACOSX point to) using a new define of LUA_USE_NON_GNU? That define
> would then be  LUA_USE_POSIX and LUA_USE_READLINE?
> 
> There are other combinations that could be better I'm sure so I'm asking if
> there is stomach for that kind of change?

[For the specific problem of overriding these variables jump to bottom. I
originally replied to respond with an alternative naming scheme that more
directly and more idiomatically addresses porting requirements, but now it's
too late to rearrange everything.]

There's a systematic and consistent way to make specifying and changing
these parameters simple for porters, expounding upon the autotools model and
the GNU style guide--both what they prescribe (which is much less than the
following) and how they're used in practice (falls just short of the
following). Using the above two dependencies as an examples

  READLINE_CPPFLAGS= # flags to locate readline headers + dependencies
  READLINE_CFLAGS=   # flags for buildings object linkable to readline
  READLINE_LDFLAGS=  # flags to locate readline library + dependencies
  READLINE_LDLIBS=   # flags to link readline + depdendecnies

  DL_CPPFLAGS=
  DL_CFLAGS=
  DL_LDFLAGS=
  DL_LDLIBS=

However you provide convenient defaults, by making each of them
independently specifiable as feature groups _and_ by compilation phase it
becomes trivial for porters to control a build without modifying the
Makefile in most common situations. (Whether porters recognize and
appreciate this ability before hacking and patching the Makefile is another
matter.) Some of the prefixes could be left out, especially for particular
features or behaviors, but I try not to deviate from the system as the whole
point is consistency and flexibility without foreseeing specific use cases.

In autotools projects the above feature-group flags are always consolidated
like so,

  ...
  CPPFLAGS =   # default application flags easily overrideable
  MYCPPFLAGS = # so you're not forced to clobber CPPFLAGS just to add a flag
  ALL_CPPFLAGS = $(READLINE_CPPFLAGS) $(DL_CPPFLAGS) $(CPPFLAGS)
                 $(MYCPPFLAGS)

This is all very verbose and repetitive, but it's _simple_, _consistent_,
_readable_, and most importantly immediately _understandable_ both by
porters and experienced unix programmers. IME there's much more cognitive
burden and time wasted deliberating how and where to avoid the repetition
than habitually following the pattern.

While not as prevalent, the most common variable to use for independently
specifying flags for generating shared libraries is SOFLAGS, which defaults
to -shared for everything but macOS.

  SOFLAGS=-shared # or -bundle -undefined dynamic_lookup for macOS

(SOFLAGS can also be used to specify ELF sonames. Some projects break out
SONAME as a separate variable and default SOFLAGS to "-shared
-Wl,-soname,$(SONAME)", but for a project like Lua this is probably just
past the point where the costs exceed the benefits, especially if LDFLAGS
and MYLDFLAGS are available for injecting ad hoc flags without clobbering
other defaults.)

You can also extrapolate the scheme to make it easier to control aspects of
how internal objects are built. So, for example, I'll often have

  PIC_CPPFLAGS =
  PIC_CFLAGS = -fPIC # -xcode=pic13 on Solaris, crazy stuff for AIX
  PIC_LDFLAGS =
  PIC_LDLIBS =

  WARN_CPPFLAGS =
  WARN_CFLAGS = -Wall -fsanitize=address
  WARN_LDFLAGS =
  WARN_LDLIBS =

to make it easy to change specific sets of flags without having to
copy+paste all the others. (Similar to the logic behind having, e.g., CFLAGS
and MYCFLAGS). The above names naturally suggest themselves. I began using
them before I learned that they are actually relatively common. No
coincidence and strong evidence that this naming scheme puts us on the right
track.

The WARN case is also a great example of why you might want to preserve the
ability to control all phases independently even if you can't conceive of a
situation where, e.g., WARN_LDFLAGS would be useful. On systems where a GCC
or clang compiler with sanitizer support must be installed manually
(separate from the native compiler) you may need to explicitly specify
linker paths and libraries for the sanitizer runtime. (IIRC, this was at
least true in the early days of those features.) The amount of pain even one
user facing such a problem might save dwarfs the time spent writing and
maintaining the additional flags.

With any abstraction it's easy to go off the rails and end up with an
overwrought solution. But I think in it's basic form it's pragmatic and
agreeable.

The biggest headache with the existing Lua build, though, is that there's no
way to override the default, per-platform flag sets without forgoing the
pre-defined flags altogether. There's no way for me to say, "build for
macosx, _except_ use this flag instead of the default macosx flag". The
least intrusive way to fix this this is, from the platform target rules, to
specify the default flags as environment variables and then use the standard
(POSIX-defined) -e option. Make is defined by POSIX to propogate command
line options and macro definitions to recursive make invocations by passing
them through the MAKEFLAGS environment variable (whether or not the rule
uses the $(MAKE) macro; make always exports SHELL and MAKEFLAGS by default).

For example, I can override the definition of CC when using the macosx
target by applying this diff,

-   $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline" CC=cc
+   CC=cc $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline"

and then doing `make MAKE="make -e" macosx CC=gcc`. A proper fix would
look something like this (untested)

-   $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline" CC=cc
+   CC=cc SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline" $(MAKE_E) $(ALL)

I haven't actually used the -e option before in real production builds, but
it works as expected on GNU make 3.81 (macOS) and OpenBSD make 6.2. At the
moment I can't test NetBSD make (aka bmake aka pmake, also used by FreeBSD)
or Solaris' default make, but I'd surprised if it didn't work correctly
there, either.

Now that I think about it, with the -e option I think I can finally adopt
Lua's clever method for conveniently aggregating default platform flags
using simple, portable constructs that don't confuse people (myself
included).

> I tested much of this this some time ago but will retest in case there is
> interest. I've put a message on Google+ for the openbsd group and will send
> some emails for input from others.
> 
> Thanks for listening.
> 
> Russ
> 
> 
> 
> Thoughts?
> 
> Russ
> 
> 
> --
> > Andrew.
> >
> >