Using fixed string buffers to avoid extensive allocs for small strings

[Flyer31 Test <flyer31@...>]

Hi,
I am currently designing a Lua32 application on a very
memory-restricted system with ca. 256kB ROM and 128kB RAM for some IoT
application.

ROM space is very well sufficient for me for Lua, I need only about 100kB.

But RAM alloc space should be limited to something in the range 30kB,
and this for me somehow seems to be impossible if I allow Lua its
"normal" extensive string allocation, which it does extensively even
for smallest strings. E. g. when I want to re-program my Lua user
program, the complete Lua ASCII program file will split into small
segments of 4-8 bytes, and if I do this with strings (just concat ..
and string.sub), then quite fast the 30kB memory limit will be hit...
. I tried to invoke the garbage collector in regular intervals (I use
yielding, so e. g. after every yield cycle...), then the garbage
collector invocation typically will fail after the 2nd invocation with
the some "memory problem" error message... (my alloc function refuses
any alloc attempt above 50 or 60kB).

So now I wanted to do a cute small library with fixed string buffers,
which I call strbuf. This buffers use a limited buffer size, in case
of any overflow they will mark the string with a final "~" sign, this
should be fine ... . Typically the strings in my programs anyway are
limited to 100 chars / typical line length, but this already would be
extreme ... most strings are much shorter (e. g. "myriads of strings"
with 4-8 chars as described above).

I want to use the following programming style for this in Lua (e. g.
defining two such buffers Tmp and InBuf, and then playing around with
them a bit):

Tmp= strbuf.new(100)
InBuf= strbuf.new( 100)

    Tmp[1]= 'hallo'
    InBuf[1]= Tmp
    InBuf[1]= Tmp[1]


This works all very nicely with a very simple strbuf lib which uses a
metatable and supports __index / __newindex:

    Tmp[1]= 'hallo' invokes __newindex for Tmp and then fills the Tmp-Buffer
    InBuf[1]= Tmp invokes __newindex for InBuf and copies Tmp-Buffer
to InBuf-Buffer
    InBuf[1]= Tmp[1] invokes __index for Tmp, copies to a
(user-hidden) global strbuf element
                                              __STRBUF, and returns
this __STRBUF, then it will inoke
                                              __newindex for InBuf and
gets __STRBUF (it is important here to
                                              use __STRBUF, otherwise
__index of Tmp would have to return a
                                              string, but this then is
again will result in stupid allocs...).

I can extend this quite easily also to further cute applications, e. g.
Tmp[n]= InBuf[m]      (to overwrite char n... of Tmp with char m... of InBuf)
                                  (also negative n/m allowed, then
counting from string end...)
index_of_f = Tmp['f']     to get the index of some substring (e. g. 'f')
1== Tmp['f']                   to check whether Tmp starts with 'f'
Tmp[n]= mul                  to "multiply" the n'th char of Tmp (or
delete if mul negative)

... and later also of course things like Tmp==InBuf, or Tmp[5]==InBuf,
or Tmp..Inbuf, or Tmp[m]..Inbuf[n] ... (or some additional helper
functions like strbuf.format or strbuf.scan...)

... this really looks very cute to me, I am happy with this, and I was
very impressed how fast this could be accomplished with this metatable
functionality of Lua, thank you for this...

Just there is one very dangerous and nerving error source for the user now:

If the user by some accident writes
Tmp='hallo'

instead of
Tmp[1]='hallo'

Because in this case then __newindex will NOT invoke, but instead by
Tmp varialble will somehow "nervingly" will be converted to a
string... .

Do you have some smart idea do avoid this, or block this type
conversion of my nice strbuf variables by Lua error?

Nice would be, that any such strbuf element (generated by strbuf.new,
and thus having this metatable strbuf...) should NOT be allowed to
"change type" any more, orto be re-assigned to anything else (but of
course Tmp[1]=... should still be possible and invoke __index).

Super-great of course would be, if my c software could be notified by
Tmp='hallo' (e. g. if there would be some meta function "__new" or
"__reassign" or so ... but this I did not find unfortunately... I am
quite sure the Lua machine will do this Tmp='hallo' typically extremly
fast without checking any metatables of Tmp?).