• Subject: Re: Parse 12 bit numbers from binary file
• From: Russell Haley <russ.haley@...>
• Date: Thu, 4 Jul 2019 18:23:14 -0700

On Thu, Jul 4, 2019 at 1:05 PM Andrew Gierth <andrew@tao11.riddles.org.uk> wrote:
>>>>> "Russell" == Russell Haley <russ.haley@gmail.com> writes:

Russell> c1, c2, index = string.unpack('<h<b', str, index)
Russell> samp1 = c1 >> 0x04
Russell> samp2 = c1 >> 0x0c & c2

>> That & there is clearly wrong, and you're putting the bits in the
>> wrong place too. And yes, the signedness is wrong - you should
>> probably read only unsigned values and convert to signed after
>> messing with the bits. But first you'd need to find some data in
>> this format to test against...

My solution (not really tested, because I have nothing to compare with)
would be along these lines:

local f = assert(io.open(arg[1],"r"))
f:close()

-- Given a 12-bit two's-complement integer value, sign-extend it to the
-- full width of a Lua integer, whatever that is. This works because
-- toggling the sign bit moves the negative values into order before the
-- positive ones, such that one can then just subtract an offset to get
-- the correct values.

local function sext12(v)
return ((v & 0xFFF) ~ 0x800) - 0x800
end

-- Precompute lookup tables for byte 2 (much faster than doing the
-- bitops each time).

local t1, t2 = {}, {}
for b = 0,255 do
t1[b] = sext12((b << 8) & 0xF00)
t2[b] = sext12((b << 4) & 0xF00)
end

local sbyte = string.byte

for idx = 0, (#dat // 3) - 1 do
local b1,b2,b3 = sbyte(dat, idx*3 + 1, idx*3 + 3)
local v1 = t1[b2] | b1
local v2 = t2[b2] | b3
print(idx, v1, v2)
end

--
Andrew.
You never cease to flabbergast me Mr. Gierth. Thank you.

Russ