Re: [OFF] 5 years to find a typo in a config script

[Tom Sutcliffe <tomsci@...>]

> On 2 Sep 2018, at 2:17 pm, Enrico Colombini <erix@erix.it> wrote:
> 
> On 02-Sep-18 13:14, Egor Skriptunoff wrote:
>> The conclusions from this story:
>> 1) Typos are hard-to-find bugs.
> 
> True. A couple of weeks ago I spent half an hour trying to understand why a C function I was testing did not work as expected.
> It turned out I accidentally assigned a value to the wrong local variable.
> 
>> 2) IMO, Lua have to somehow solve the problem of absence of warnings about misspelled identifiers.
> 
> I heard a mysterious sect performs dark rituals involving a little-known, exoteric technique revealed only to a few selected believers. It's called "testing".
> 
> Seriously, having used my share of static and non-static languages over many years, I didn't notice much difference in the frequence of this sort of problems. But, again, I may be part of the above-mentioned sect.

For one of the Lua-based DSL projects I work on, I added a validator which requires that every line of code in your script gets hit when you throw the test data at it, and that the calculated results are as expected for each set of inputs. (The test data is also defined in the script). It makes heavy use of the debug library and line hooks as you'd expect, and has been surprisingly useful to detect not only bugs in the scripts, but also in the modules they rely on ("if all my scripts still validate as expected, the other libraries probably aren't too broken") and highlighting unhandled or untested code ("how exactly *would* I get to the situation where I would hit this line of code?"). If the validator doesn't pass, the build fails meaning no-one ever merges broken scripts assuming they built it at least once beforehand, and if they don't even build their code before merging then I put my Build Team hat on and give them a polite explanation of why I've reverted all their commits.

Obviously it's only practical in a tightly-controlled DSL environment, but it's proved quite a nice hybrid of static analyser and test harness for thousands of lines of conditional-heavy scripts which examine a lot of global state. And requiring every line to be hit is only a rough approximation of testing every possible code path and combination of input data, but it works pretty well.

I really should add strict.lua-esque global variable checks too.

Cheers,

Tom