Re: How to collect as much as possible with collectgarbage() ?

[Jerome Vuarand <jerome.vuarand@...>]

2009/7/8 Ignacio Burgueño <ignaciob@inconcertcc.com>:
> Hi everyone. Maybe this topic has been discussed to death, but I don't fully
> understand how it is supposed to work.
>
> I need to read a huge file from a socket, write it to disk and then call a
> library which will perform some operations on the file. That library
> requires lots of memory, and I'm trying to release as much memory as
> possible before calling into the library (I already got some "not enough
> memory" errors and I'm trying to avoid that).
>
> Given this simple script:
>
> local function showMem(msg)
>   print( "Memory used: " .. math.floor(collectgarbage("count")) .. " Kb " ..
> (msg or ""))
> end
>
> local filename = [[c:\file_50mb.txt]]   -- a 50 megabyte file with random
> data
> local f = io.open(filename, "rb")
>
> showMem()
>
> local data = f:read("*a")
> f:close()
> f = nil
>
> showMem("before collecting")
> data = nil
> for i=1,20 do
>   collectgarbage()
>   showMem("after collecting")
> end
>
> I get the following: (under Windows XP SP3, Lua 5.1.4)
>
> Memory used: 230 Kb
> Memory used: 157204 Kb before collecting
> Memory used: 12997 Kb after collecting
> Memory used: 3397 Kb after collecting
> Memory used: 997 Kb after collecting
> Memory used: 397 Kb after collecting
> Memory used: 247 Kb after collecting
> Memory used: 210 Kb after collecting
> Memory used: 201 Kb after collecting
> Memory used: 198 Kb after collecting
> Memory used: 198 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
> Memory used: 197 Kb after collecting
>
>
>
> Calling collectgarbage does not free all possible memory, so I suppose that
> each time it is called, more stuff becomes collectable, until there is
> nothing more. Short of measuring the memory use and stopping when the used
> Kbytes don't change between two calls to collectgarbage() (ugly hack!) is
> there any other way to collect everything?

IIRC the number of cycles necessary depends on the depth of the data
to collect. For example
  t={}
  t=nil
will require one cycle, while
  t = { t = { t = { t = { t = { t = {} } } } } }
will require 6 cycles.

Your example quite surprised me, because data is a single string. But
I don't know what is the layout of the file userdata and its attached
values. So I split your test to see that:

Memory used: 30 Kb
Memory used: 194776 Kb before collecting file
Memory used: 62359 Kb after collecting file
Memory used: 52971 Kb after collecting file
Memory used: 50624 Kb after collecting file
Memory used: 50040 Kb after collecting file
Memory used: 49894 Kb after collecting file
Memory used: 49858 Kb after collecting file
Memory used: 49849 Kb after collecting file
Memory used: 49846 Kb after collecting file
Memory used: 49846 Kb after collecting file
Memory used: 49846 Kb after collecting file
Memory used: 49846 Kb before collecting data
Memory used: 30 Kb after collecting data
Memory used: 30 Kb after collecting data
Memory used: 30 Kb after collecting data
Memory used: 30 Kb after collecting data
Memory used: 30 Kb after collecting data
Memory used: 30 Kb after collecting data
Memory used: 30 Kb after collecting data
Memory used: 30 Kb after collecting data
Memory used: 30 Kb after collecting data
Memory used: 30 Kb after collecting data

As you can see data is collected in one cycle, while the file is
collected in 8 cycles. I don't think there is a way to predict that
number besides knowing exactly how data is structured. What you can do
is repeatedly collect until nothing is collected.

Here is my script:

local function showMem(msg)
  print( "Memory used: " .. math.floor(collectgarbage("count")) .. "
Kb " .. (msg or ""))
end

local filename = [[c:\file_50mb.txt]]   -- a 50 megabyte file with random data
local f = io.open(filename, "rb")

showMem()

local data = f:read("*a")
f:close()

showMem("before collecting file")
f = nil

for i=1,10 do
  collectgarbage()
  showMem("after collecting file")
end

showMem("before collecting data")
data = nil
for i=1,10 do
  collectgarbage()
  showMem("after collecting data")
end