David Manura

wiki

 local two
 local function one() two() end
 function two() one() end -- forward declared local (misleadingly looks like global)
 

 -- Lua 5.1, example without exceptions
 local function readfile(filename)
   local fh, err = io.open(filename)
   if not fh then return false, err end
   local data, err = fh:read'*a'
   -- note: in this case, the two fh:close()'s may be moved here, but in general that is not possible
   if not data then fh:close(); return false, err end
   fh:close()
   return data
 end
 -- Lua 5.1, example with exceptions, under suitable definitions of given functions.
 local function readfile(filename)
   return scoped(function(onexit)  -- based on pcall
     local fh = assert(io.open(filename)); onexit(function() fh:close() end)
     return assert(fh:read'*a')
   end)
 end
 
 -- proposal, example without exceptions
 local function readfile(filename)
   local fh, err = io.open(filename); if not fh then return false, err end
   defer fh:close()
   local data, err = fh:read'*a'; if not data then return false, err end
   return data
 end
   -- note: "local val, err = io.open(filename); if not val then return false, err end" is a common
   -- pattern and perhaps warrants a syntax like "local val = returnunless io.open(filename)".
 -- proposal, example with exceptions
 local function readfile(filename)
   local fh = assert(io.open(filename)); defer fh:close()
   return assert(fh:read'*a')
 end
 -- proposal, example catching exceptions
 do
   defer if class(err) == 'FileError' then
     print(err)
     err:suppress()
   end
   print(readfile("test.txt"))
 end
 
 -- alternate proposal - cleanup code by metamechanism
 local function readfile(filename)
   scoped fh = assert(io.open(filename)) -- note: fh:close() or getmetatable(fh).__close(fh) called on scope exit
   return assert(fh:read'*a')
 end
 

 function foo() error("fail", 2) end
 function bar() foo() end -- any way for bar not to increase the error level when invoking foo?
 foo()  -- error level ok
 bar()  -- error level wrong
 

 local two
 local function one() two() end
 function two() one() end -- forward declared local (misleadingly looks like a global declaration)
 

Wiki pages

There are Lua wiki pages and projects I've been involved in.

Code and Projects:

• Formal Lua modules (core):
  • [lua-file-slurp / file_slurp] - Easily read/write entire files or pipes from/to a string
  • [lua-require-any / requireany] - require any one of the listed modules
  • [lua-glob-pattern / globtopattern] - Convert file glob to Lua string pattern - FileGlob
  • [lua-find-bin / findbin] - Locate directory of original perl script
  • [lua-lib / lib] - Simple insertion of directories in package search paths
  • [lua-compat-env / compat_env] - Lua 5.1/5.2 environment compatibility functions (load/loadfile/getfenv/setfenv)
• Formal Lua modules (more):
  • [lua-bit-numberlua / bit.numberlua] - Bitwise operators in pure Lua using Lua numbers. Includes Lua 5.2 'bit32' and (LuaJIT) LuaBitOp 'bit' compatibility interfaces.
  • [lua-digest-crc32lua / digest.crc32lua] - CRC-32 checksum implemented in pure Lua - [dependencies: bit.numberlua (or native bit library)]
  • [lua-pythonic-optparse / pythonic.optparse] - Command-line argument parsing similar to Python's optparse library
  • [lua-compress-deflatelua / compress.deflatelua] - DEFLATE (RFC1951)/gunzip implemented in pure Lua - ModuleCompressDeflateLua - [dependencies: bit.numberlua (or native bit library), digest.crc32 (optional), and pythonic.optparse (optional)]
  • [lua-sourcepack / sourcepack] - Extract Lua files from other Lua files.
  • [luabalanced] - match delimited snippets of Lua code in a string - LuaBalanced
  • [lua-stringlua / stringlua] - Lua 5.1 string library reimplemented in Lua - StringLibraryInLua
  • [comprehension] - pure Lua implementation of List comprehensions via dynamic code gen - ListComprehensions
  • [lua-matrix / matrix] - Matrices and vectors of are real, complex, and symbolic elements, implemented as Lua tables - LuaMatrix (maintenance)
  • LuaCom - Lua interface to Microsoft's Component Object Model (COM) (maintenance)
  • [lua-mbuild] - Lua-implementation of a memoized build system (like fabricate, memoize.py, fbuild, etc.) LuaBuildSystems
  • [makebuild.lua] - rudimentary Lua makefile generator in < 100 lines of code. LuaBuildSystems
• LuaToCee - convert Lua to C code using C API calls
• InlineCee - simple module that allows C code to be embedded inline in Lua code.
• LuaDist - CMake-based ports system for Lua and non-Lua packages.
• LuaFish - Lua 5.1 parser, macro system, and static type checker in LPeg [1] (related to Metalua [2])
• LuaSearch - effort to standardize Lua module documenation via POD
• ModuleReview - effort to review/classify modules in terms of quality and usefulness for Lua distributions
• SlightlyLessSimpleLuaPreprocessor - enhanced version of the enhanced version of the Lua preprocessor
• LuaProxyDll - avoid DLL dependency issue
• DetectingUndefinedVariables - checkglobals function is an alternative to etc/strict.lua and luac -p -l - | lua test/globals.lua.
• LuaGenPlusPlus - A C++ header file to generate efficient sequences of Lua C API calls at compile-time via template metaprogramming techniques.
• LuaInterpreterInLua - reimplements the Lua 5.1 interpreter (lua.c) in Lua.
• CommandLineParsing - command line parsing (getopt)
• LuaPatch - Lua implementation of the patch utility
• ExPattern - regular-expression like support implemented in terms of Lua patterns and code generation
• SerialCommunication - serial communications in Lua/alien
• SourceOptimizer -- utility for optimizing Lua source, inlining functions
• StringQuery - string pattern matching and replacement library inspired somewhat by jQuery
• LuaAnnotate - C code comments externally stored but displayed inline.

General Topics, Patterns, Tricks, and Design:

• LuaNews - A summary of the latest happenings on the Lua mail list and elsewhere in the Lua world.
• LuaStyleGuide - every language has a style guide(s)
• LuaModuleFunctionCritiqued - please don't use the module function
• ModuleExecutionProposal - proposal+patch to add a new Lua command-line switch to load a function with a given package name and execute it as a script
• ModuleDefinition - various ways to define modules
• ModuleVersioning - module versioning discussion
• LoadLibrary - discussion of issues with Win32 LoadLibrary module loading
• ApplicationBinaryInterface - notes on binary interface (ABI) compatibility in Lua modules
• BuildingModules - (section on avoiding/mixing C runtimes)
• LuaLocales - notes on locales in Lua
• NamedParameters - named parameter support in Lua
• MetaLuaRecipes - Metalua syntax extensions
• ResourceAcquisitionIsInitialization - RAII is poorly supported in Lua
• StatementsInExpressions - relating to Lua statements cannot be used inside expressions without anonymous functions
• ExpressionsAsStatements - Lua expressions are not statements, though we can simulate the effect.
• VarargTheSecondClassCitizen - relating to varargs not being first class objects limits expression
• MutableFunctions - demonstration that closures can implement the semantics of Lua tables without using any Lua tables (functional programming)
• FunctionalTuples - a code generation approach to implementing tuples of length N
• LuaVersusPerl - Lua and Perl have different applications
• LunaticPython - call Python from Lua
• ErrorHandlingBetweenLuaAndCplusplus - issues in mixing Lua with C++ rather than C
• SourcePreprocessing - transforming source code before execution
• LuaDataFormats - Lua as a data representation language
• CppMetaprogramming - metaprogramming C++ not with templates or macros but with Lua.
• MakingLuaLikeRuby - Rubyesque Lua.
• GeneralizedPairsAndIpairs - why can't a table pretend to be another table?
• SandBoxes - issues relating to sandboxing: running untrusted Lua code in a restricted Lua environment
• LuaDesignPatterns - Lua design patterns
• LuaHacks - novel things you shouldn't do but could
• DecoratorsAndDocstrings - associate metadata or help with an object
• CustomOperators - simulating support for custom operators
• ImmutableObjects - topics of immutability in Lua
• StoringNilsInTables - topics relating to nil not being explicitly stored in tables

• ShortAnonymousFunctions - making function() ... end less verbose
• LuaTypeChecking - approaches to type checking in Lua, a dynamically typed language
• ReadOnlyTables - preventing writes to Lua tables.
• StringInterpolation - placing variables inline to strings, like Perl "x = $x"
• ColonForMethodCall - concerning the distinction in Lua of o:f() v.s. o.f()
• ObjectProperties - getters/setters that look like table field accesses
• LuaCarp - Perl Carp implemented in Lua (handle error level transparently)
• AssociativityOfConcatenation - discussions related to concatenation being right associative
• TernaryOperator - addressing the lack of a C-like ternary conditional operator (a?b:c) in Lua
• TableScope - solutions (and patch) to allow table constructs to enclose a scope
• CnumberPatch - fast implementation of getter/setter handler functions for locals - Lua patch.
• SimdExperiment - experimental patch to Lua VM to support a type of Single Instruction Multiple Data (SIMD)
• PortableNetworkGraphicsParser - partial PNG graphics file parser (example)
• TheEssenceOfLoadingCode - explains the relationship between the various ways of loading external code in Lua.
• ExpressionTemplatesInLua - a Lua coding technique similar to C++ expression templates
• OutputLuaTableToHtmlFile - Lua table to HTML/CSS with expand/contract and xref links
• RuntimeSyntax - automates the run-time simulation of what resembles syntax extensions
• MethodChainingWrapper - cooperatively add custom methods to built-in types like strings and functions, particularly when using method chaining
• TablePreallocation - methods to preallocate tables from Lua
• SleepFunction - pause for N seconds
• GettingVariablesFromValues - pass-by-reference and the like for Lua
• StringTrim - comparison of string trim operations
• LuaInspect - Lua code analysis and HTML code browsing
• PcallAndCoroutines - implement pcall in terms of coroutines
• LocalsVsGlobals - Comparison between local and global variables
• ProgramAnalysis - static code analysis or dynamic program analysis) of Lua code
• LuaImplementationVerification - validating correctness of Lua implementations
• FileSystemOperations - file system operations libraries for Lua
• LightUserData - explains the purpose and use of light userdata
• LuaVersionCompatibility - ensuring that Lua code remains compatible across multiple versions and implementations
• UniversallyUniqueIdentifiers - various ways to created universally unique identifiers (UUID's) in Lua
• PlatformDetection - detecting the platform from Lua.
• TypeIntrospection - determining types of objects at runtime
• GotoStatement - goto statement
• MiningLuaCode - content analysis of Lua source code repos
• MechanismNotPolicy - comments on Lua's "mechanism not policy" design rule
• HammingWeight - test of hamming weight implementations

Comments/Annotations on Other Documents:

• LuaReferenceManualComments
• ProgrammingInLuaComments
• LuaSource
• WikiStructure
• LuaFiveTwo/LuaFiveThree

Lua 5.3 Wishlist

Here are few things bug me, though I'm not necessarily sure what to do about all of them.

• Tracebacks might use some reform.
  • Refactoring f(...) to local ok, err = pcall(f, ...); if not ok then error(err) end or assert(pcall(f, ...)) has side-effects in traceback behavior. LuaList:2012-02/msg00858.html
  • other older more general comments: LuaList:2010-08/msg00481.html
  • Why truncate paths in tracebacks? LuaList:2006-03/msg00101.html
  • The level parameter used in the error function lacks maintainability since a function might be called from different levels and those levels might be refactored. Refactoring f() to (function() return f() end)() has side-effects in terms of level behavior, with no option to suppress the change in stack level. LuaCarp was a very early effort to improve this. Also see [6][7]. At least in 5.2 work, getfenv/setfenv are mostly removed and setfenv behavior in module is deprecated, so it's moving in the right direction.
  • The assert function is arguably misnamed: [8]. Also, the fail expression is always evaluated.
• I have qualms about metamethods. Despite their uses, they make semantics more complicated.
  • Proliferation of raw v.s. nonraw functions and semantics. Also, where's "rawtostring"? [LuaFiveTwo].
  • I don't get why the change was made in 5.2 table.* functions to partly honor metamethods. [9][10][11]
  • __index (unlike the proposed __getindex) doesn't fully intercept table accesses (hence proxy tables).
  • There were proposals to allow 'a:b()' not to depend on 'a.b' (e.g. __mcall or __mindex metamethod) [12]
• NamedParameters as tables seem heavy. What effect does it have on LuaJit optimizations? Converting a function from non-named to named parameters is a fundamental change to its interface, not to be taken lightly.
• Questions remain about versioning. ModuleVersioning/LuaVersionCompatibility. 5.2 is a breaking change and not fully supported in LuaJit. Other languages like Perl 6 have statements like "use v6;" to indicate the code is Perl6 rather than Perl 5. _ENV = require '_G53'?
• Lua syntax is not always necessarily ideal as a DSL. Consider representing something like PHP, where control structures are embedded inside tree nodes, but StatementsInExpressions are not seamless in Lua. Lua commas can also clutter [13]. Compare to Lisp s-expressions or Ocaml.
• Cleanup on scope exit (ResourceAcquisitionIsInitialization). The prototypical example involves file handles, like in file_slurp readfile [3]. One of the simpler solutions is a Google Go "defer" or D language "scope(exit).
• local ok, err = f(); if not ok then return ok, err end is a common idiom. Should it be made shorter?
• StringInterpolation for local variables has no really nice solution. Some type of compile-time preprocessing step that expands things like printf "${x},${y},${x}" into printf("${x},${y},${x}", x, y, x) that is then evaluated at runtime might be a good compromise.
• Forwardly declared local functions look instead like global definitions. Commenting or semantic editors may help some though:

local two
local function one() two() end
function two() one() end -- forward declared local (misleadingly looks like global)

• package.path and package.cpath are strings rather than arrays. LuaList:2010-06/msg00214.html . This led to the "lib" module [4].
• Debugging info lacks character positions (only has line numbers): LuaList:2010-08/msg00273.html
• luac with multiple files could be generalized: LuaList:2010-11/msg00898.html
• Lack of comments in the Lua source [14] (LuaSource) can make it more difficult to penetrate than otherwise. See LuaAnnotate though.
• for _,v in ipairs(t) do is ugly, and it's less efficient than for i=1,#t do local v=t[i] do. The too forms do noth have the same behavior either in terms of __len.

Lua 5.2 Wishlist

Caveat: The below list may not be entirely updated to reflect the final 5.2.0 release and may need cleaned up more.

These changes I want are included in the 5.2 work releases (LuaFiveTwo):

• Module system reforms - Make the module system more consistent - see LuaModuleFunctionCritiqued. 5.2 now deprecates the module function and its ilk. Some wanted to improve rather than dispose of the module functions, but there was not consensus on precisely what that would be.
• Environment reforms - I support the _ENV proposal because of the simplifications it makes to lexical analysis (LuaInspect, LuaFish/luaanalyze) and compilation (LuaToCee). I think this is related to a more general problem though involving stack levels (see traceback points above). I agree though that _ENV is ugly and probably should not be used by most user code.
• C function improvements -- Unlike lua_cpcall, avoid allocation in C functions [15] (for performance and avoiding complications in error handling) and support multiple arguments and return values in protected calls. In LuaFiveTwo, lua_cpcall is eliminated and replaced with a lua_pushcfunction that no longer allocates, and multiple arguments and return values are supported via lua_pcall.
• Lua function caching - There is now less need to manually hoist function instantiation out of inner loops since functions are now cached. However, this feature might not be implemented to its full potential[16].
• BitwiseOperators standard library. 5.2 implements this now. I had thought the Lua community had already converged on Lua BitOp (supported by LuaJIT and superseding lbitlib). See BitwiseOperators/[5] Both the 5.2 and LuaJIT bit libraries are probably fine for most purposes, and it can be possible to write code that is compatible with both: `local band = bit.band or bit32.band` provided you avoid incompatible cases. Should get/set bitfield operations be added? [17]
• __len metamethod for tables (as well as __pairs -- GeneralizedPairsAndIpairs). This causes me some hesitation though because of the questions it opens. Should we have a rawlen (similar to lua_rawlen in the work version)? Why should pairs() observe metamethods but not next()? Lua tables remain not truly virtualizable (LuaVirtualization). Should "table" functions recognize metamethods?--probably not [18].
• Expose all Lua operations to C API - Extend C API to support missing operators [19]. LuaFiveTwo now supports lua_compare and lua_arith. My main motivation for this is to simplify my LuaToCee output. Also it makes things orthogonal and is relatively easy for Lua to implement. Some things may remain lacking like tail calls in C functions [20].
• Hex escapes in strings are sometimes nice (avoiding conversion to decimal), particularly when reading files or communications streams whose specification includes bytes in hex format or whose individual bits have significance. Sure, you can use a helper function, hex 'cdef' or binary '0101', but why need that.
• Some ask for ContinueProposal. I tend to find breaking out of nested loops more common. Here's my proposal that generalizes nested break+continue into a single construct: LuaList:2010-11/msg00556.html . See also GotoStatement.
• Minor odds and ends:
  • closing a pipe returns exit status
  • Protect against triggering undefined behavior in C libraries from Lua standard libraries [21] like os.date. IMO, the standard library (except maybe debug) should never crash on bad input.
  • Rename lua_typerror to lua_typeerror [22]. Actually, the function is now deprecated instead.
  • "package.loaders" misnomer [23][24] (package.loaders contains searchers not loaders).
• Other things I don't often run into but I guess are nice: more freedom to yield (including PcallAndCoroutines), xpcall accepting arguments like pcall, non-string error messages in Lua interpreter, emphemeron tables (improves weak table garbage collection).
• `luac -l -l` is currently undocumented (done in 5.2.0)
• Treat unrecognized escape sequences as errors? LuaList:2010-10/msg00449.html
• Overcome implementation limits in number of locals [25] and concatenation AssociativityOfConcatenation is not done, but LuaFiveTwo is increasing "Maximum number of constants per chunk increased to 2^26 from 2^18" though.

Here's other older ideas not addressed in 5.2 and which I haven't decided to move them up to the 5.3 wishlist:

• Some type of macro mechanism. See MetaProgramming.
• Add efficient tests for existence of operators on objects [26][27]
• File handle virtualization - Sometimes you can pass a custom table that looks like a file handle to functions that expect a file handle, but this doesn't work all the time (e.g. if the function tests with io.type).
• Maybe add versions of lua_pushstring and lua_newuserdata that don't longjump. May be useful for mixing C++ exception code and C API longjump code.
• Small source code changes to reduce patching complexity? LuaList:2010-11/msg00899.html
• Reference manual formatting suggestions: LuaList:2010-10/msg00350.html
• Generalize Lua table constructor notation [28]. This also provides list comprehension syntax.
• Manipulations with ... are somewhat awkward unless you copy it in a table (not as efficient) and handle nils correctly. See also VarargTheSecondClassCitizen. Something like the proposed #... and ...[i] would help, although that particular syntax has problems. The dilemma is summarized by Roberto here: LuaList:2010-06/msg00434.html. Some small improvements are likely possible though.
• Abstracted liolib.c [29] - Many people embed Lua in systems without STDIO (e.g. GUI applications) and patch this file. Lua 5.2 provides some abstraction for this (e.g. luai_writestring) but it does not cover liolib.c.
• Clearer syntax errors [30][31][32]. Some improvement is possible. Really good error/warning reporting might require an external linter.
• ModuleExecutionProposal - execute Lua command-line scripts located via LUA_PATH
• On DetectingUndefinedVariables, my current opinion is that it is best handled by the text editor. I find runtime analysis (strict.lua) inferior to static bytecode analysis (globals.lua) inferior to static source analysis (LuaInspect). Furthermore globals.lua and the (hybrid) checkglobals don't work as well in 5.2 due to _ENV changes.
• ShortAnonymousFunctions - It's often said that the Lua syntax for anonymous functions can be verbose for tiny functions used in function programming (map/filter) and functions used as blocks (e.g. simulated switch or try/catch statements). The workarounds in ShortAnonymousFunctions are not very satisfying. It doesn't seem like discussions on the user group are seriously going anywhere here though.
• Allow pragmas in source to inject debugging info into the bytecode. [33] (useful for preprocessors of Lua source). Metalua currently uses its own bytecode generator because of such limitations. There is a workaround but it is not fully general: LuaList:2010-06/msg00311.html . I suspect, however, that rather than injecting debug info, it may be better to just rewrite the message handler.
• More efficiently (O(1)) preallocate or subsequently clear arrays. This may involve both Lua and C API functions like lua_createtable [34]. See TablePreallocation for the preallocation (does not discuss clearing).
• Lua has the dichotomy of statements and expressions (StatementsInExpressions/ExpressionsAsStatements). The loophole in this are anonymous functions, but this involves a heap allocation, unless it triggers 5.2 function caching, bug that's not always triggered, and syntax can be verbose (ShortAnonymousFunctions). This can be useful in metaprogramming, and Metalua recognizes this (see "Statements in expressions" in the Metalua Manual) and extends the Lua parser/compiler to allow it while requiring no change to the Lua VM. See also the "let" statements mentioned below on this page and "Stored Expressions" in LuaDesignPatterns. The SourceOptimizer provides an automated way to inline anonymous functions into standard Lua 5.1 source code, but the transformation wasn't straightforward.
• The highly dynamic nature of Lua limits its performance. True, Lua is fast, but it could be faster. For example, x*x is generally faster than x^2 since Lua cannot reduce the latter to the former at compile time (think: x might have a __pow metamethod). Macro processing could alleviate this (e.g. "transforming "y = SQUARE(x+y) + 1" into "y = (let t1 = x+y in t*t) + 1"). See also LuaJIT and LuaToCee. Global, or at least whole-file, flow analysis can help here.
• Some other things, not necessary related to Lua core development:
  • Lua needs more standardization of quality control of libraries covering non-ANSI C functionality. ExtensionProposal and LuaRocks may help here. Related concerns involve documentation, testing, and packaging.
  • Strength access to existing library frameworks in other platforms like .NET, Java, and Python. Lua acquires the vast resources of these other developer communities.