Re: 60kB RAM: Compare indexed string handling to Lua heap handling

[Flyer31 Test <flyer31@...>]

Hi Benjamin,
thank you for answer.

I am not sure whether I understand it correctly - you mean that the
final "run code" should not be present inside the controller, but at
every run event should be loaded from the interface?

(this would not be possible - the serial interface is considered here
to be only some sort of "debug+programming" interface. After end of
debug/programmning phase, the controller has to work "on its own",
with serial interface dis-connected again).

... the programming of course I could also do in many other ways,
mainly NOT driven by a Lua code, so  not "bothering the Lua string
handling system" too much ... I just did it like this to somehow also
test whether this brings the Lua memory alloc system to a limit or
not... .

But do not misunderstand this as some "general bad thinking" about the
"Lua string handling system / Lua memory system". In fact I think it
is really very great and impressive. Completely contrary: I was really
impressed that Lua does this for loop 3 times faster than my "indexed
string buffers". And of course I think much more than 99% of Lua
applications will run on systems which have abundant RAM in the
Megabyte or even Gigabyte range:

Just I think that for such vey RAM constrained systems, the extensive
use of malloc gets dangerous ... resp. it would be nice to have an
alternative method to skip extensive malloc usage for strings (like my
strbuf library - in my software if I want I can use Lua string lib and
my string buffers both together, this does not interfere ... just I
then would strongly recommend to users to use the strbuf things for
"extensive" string handling)... usually in my Controller software I am
used to do ALL with static memory, not using malloc at all ... . But
after looking more into the details of Lua, it is of course really
very clear that malloc / dynamic memory here is a very crucial point
for such an interpreter language approach... .

On Wed, Nov 24, 2021 at 11:06 PM Benjamin Riggs
<riggs@entropicengineering.com> wrote:
>
> Have you considered loading the script source straight into the interpreter from serial, and then dumping the compiler bytecode to flash? It should eliminate your extra step of chunking the source to store in flash. The VM will also detect (and reject) invalid source immediately, preventing you from having to deal with that error case later.
>
> This message composed at the mercy of thumbs and autocorrect.
>
> > On Nov 24, 2021, at 3:05 PM, Flyer31 Test <flyer31@googlemail.com> wrote:
> >
> > I just try some test code, to compare my indexed string functions to
> > the standard Lua string functions. Just for those people interested
> > here the results:
> >
> > This is my test project with a Cortex M4 STM32G4 controller running at
> > a lower speed of 16MHz, 512kB ROM, 128kB RAM, nice QFN48 housing, I
> > posted a blog 2 or 3 weeks ago already, where I described the basic
> > functionality for the indexed strings... .
> >
> > The main problem here is the 128kB RAM, so I preferred to limit the
> > heap available to Lua to 60kB - this heap is EXCLUSIVELY used by Lua
> > (I do NOT use dynamic memory anywhere else in my code...).
> >
> > My first main application is a Lua program, which just organizes the
> > "Chip Flash re-proramming with some new Lua user software". So for
> > this I send down a smaller Lua text file with 3kB size to the
> > controller via serial interface, and my Lua programming software
> > running on the controller will have to cut down these 3kB into
> > successive pieces of 8 Bytes to program the Flash piece by piece (the
> > 8 byte is the "programming page size" of this STM32G4 controller).
> >
> > Using the Lua string functions with concat and string.sub did not work
> > here ... the heap usage of Lua will run to the limit of 60kB fairly
> > fast... and then I get severe problems with my restricted RAM memory
> > of course... .
> >
> > With my new indexed functions I am very happy now, it runs very
> > nicely, and I think also the writing style is really nice... .
> >
> > For comparison I used the following Lua code (V5.4.3, my indexed
> > string buffers all start with SB..., and the library name is "sb"):
> >
> > SBtmp= sb.setup( 100)
> > SBstr= sb.setup( 100)
> >
> > Sstr= 'Hello Lua - please have a look at this 100 bytes long string
> > and cut it in very many small tiny parts'
> > Ttmp= { }
> >
> >    SBstr= Sstr
> >
> >    for i= 1, #Sstr do
> >      for j= 1, #Sstr-i do
> >        --Ttmp[j]= Sstr:sub(i, i+j)
> >        SBtmp= SBstr[i+0.001*j]
> >        sys.sleep( 0)
> >      end
> >    end
> >
> > (The sys.sleep(0) is an internal function of myself to allow
> > yielding/resume handling ... otherwise my count hook would terminate
> > such a ca. 5000 long loop with an error message after 1000 Lua
> > cycles...).
> >
> > This is the version using my indexed buffers.
> >
> > If I want to test the Lua string handling, then I have to flip the line comment:
> >
> >    for i= 1, #Sstr do
> >      for j= 1, #Sstr-i do
> >        Ttmp[j]= Sstr:sub(i, i+j)
> >        --SBtmp= SBstr[i+0.001*j]
> >        sys.sleep( 0)
> >      end
> >    end
> >
> > As expected, here the Lua string handling will run into 60kB heap
> > usage very immediately.
> >
> > And my indexed string buffers keep at the 21kB heap usage, which is
> > somehow the "Lua startup" heap requirement for my Lua program here...
> > .
> >
> > So I am very happy with this.
> >
> > By the way the speed of Lua string handling is factor 3 faster than my
> > indexed string buffers - this surprised me quite a bit... . (my code
> > needs 4.6sec for these 5000 allocations..., Lua needs only 1.8sec).
> > But speed is not the target here ... I mainly need memory security,
> > perfectly stable running and the "user programming flexibility" with I
> > get by Lua.
> >
> > ... just info for all who are interested in this ... and thanks very
> > much to the Lua people that I may use this terrific Lua for this, I am
> > really very excited, that all runs now so extremely nicely, after I
> > introduced my indexed string handling without heap requirements.