Re: Parse 12 bit numbers from binary file

[Andrew Gierth <andrew@...>]

>>>>> "Russell" == Russell Haley <russ.haley@gmail.com> writes:

 Russell> c1, c2, index = string.unpack('<h<b', str, index)
 Russell> samp1 = c1 >> 0x04
 Russell> samp2 = c1 >> 0x0c & c2

 >> That & there is clearly wrong, and you're putting the bits in the
 >> wrong place too. And yes, the signedness is wrong - you should
 >> probably read only unsigned values and convert to signed after
 >> messing with the bits. But first you'd need to find some data in
 >> this format to test against...

My solution (not really tested, because I have nothing to compare with)
would be along these lines:

local f = assert(io.open(arg[1],"r"))
local dat = f:read("*all")
f:close()

-- Given a 12-bit two's-complement integer value, sign-extend it to the
-- full width of a Lua integer, whatever that is. This works because
-- toggling the sign bit moves the negative values into order before the
-- positive ones, such that one can then just subtract an offset to get
-- the correct values.

local function sext12(v)
    return ((v & 0xFFF) ~ 0x800) - 0x800
end

-- Precompute lookup tables for byte 2 (much faster than doing the
-- bitops each time).

local t1, t2 = {}, {}
for b = 0,255 do
    t1[b] = sext12((b << 8) & 0xF00)
    t2[b] = sext12((b << 4) & 0xF00)
end

local sbyte = string.byte

for idx = 0, (#dat // 3) - 1 do
    local b1,b2,b3 = sbyte(dat, idx*3 + 1, idx*3 + 3)
    local v1 = t1[b2] | b1
    local v2 = t2[b2] | b3
    print(idx, v1, v2)
end

-- 
Andrew.