Re: clang memory sanitizer report unsigned int overflow

[David Sicilia <dpsicilia@...>]

For reference that is this line marked with **:

   |  unsigned int luaS_hash (const char *str, size_t l, unsigned int seed) {
   |    unsigned int h = seed ^ cast_uint(l);
   |    for (; l > 0; l--)
**|       h ^= ((h<<5) + (h>>2) + cast_byte(str[l - 1]));
   |    return h;
   |  }

Though not sure which part of that would be causing an add overflow because h is an unsigned int which I believe should not trigger UB when it overflows.

David

On Thu, Jan 5, 2023 at 7:42 AM Jo Ember <usergoodvery@gmail.com> wrote:

Hi,

I recompiled lua-5.4.4 with memory sanitize instrumentation on linux 5.15.0-56-generic Ubuntu 22.04.1 LTS as a static library and linked to my application and I got the error below triggered from luaL_newstate().

Process 626150 stopped
* thread #1, name = 'ufsrv', stop reason = Unsigned integer overflow
    frame #0: 0x0000555555ce4950 ufsrv`__ubsan_on_report
ufsrv`__ubsan_on_report:
->  0x555555ce4950 <+0>: retq
    0x555555ce4951:      int3
    0x555555ce4952:      int3
    0x555555ce4953:      int3
(lldb) bt
* thread #1, name = 'ufsrv', stop reason = Unsigned integer overflow
  * frame #0: 0x0000555555ce4950 ufsrv`__ubsan_on_report
    frame #1: 0x0000555555cdf706 ufsrv`__ubsan::Diag::~Diag() + 214
    frame #2: 0x0000555555ce15f1 ufsrv`void handleIntegerOverflowImpl<__ubsan::Value>(__ubsan::OverflowData*, unsigned long, char const*, __ubsan::Value, __ubsan::ReportOptions) + 497
    frame #3: 0x0000555555ce13ec ufsrv`__ubsan_handle_add_overflow + 60
    frame #4: 0x000055555724847e ufsrv`luaS_hash(str="\b", l=15, seed=1672921169) at lstring.c:46:18
    frame #5: 0x00005555572439b8 ufsrv`luai_makeseed(L=0x000071b000000008) at lstate.c:79:10
    frame #6: 0x00005555572404d5 ufsrv`lua_newstate(f=(ufsrv`l_alloc at lauxlib.c:1011), ud=0x0000000000000000) at lstate.c:375:13
    frame #7: 0x0000555557370723 ufsrv`luaL_newstate at lauxlib.c:1089:18