Re: LuaJIT-2 on Mac OS X and FreeBSD x86-64

[Mike Pall <mikelu-1010@...>]

Tony Finch wrote:
> #define MMAP_REGION_END		((uintptr_t)0x80000000)
> 
> Is there any reason this is 2GB not 4GB? LuaJIT on 32 bit Linux is happy
> to get pointers from mmap() above 2GB. I have done some tests on FreeBSD
> x86-64 with a 4GB ceiling and it seems to work fine. (Sorry, haven't got a
> 64 bit Mac to hand right now.) This improvement allows me to allocate over
> 3.33 GB data (vs 2.75GB on 32 bit Linux).

Nope, this doesn't work. Address offsets in x86/x64 instructions
are signed. And the JIT compiler folds the addresses of fixed
objects (e.g. string constants) into offsets. This doesn't matter
for 32 bit, since the address space wraps around at 4GB. But this
means an object in the range 2GB-4GB may cause generated code to
point to an invalid address on x64.

Possible test case for this:
- Allocate lots of memory.
- Load a script from a string and hope everything gets allocated
  above 2GB.
- From within the script sum the bytes of a 100-char string
  constant in a loop.
- Optionally dump the generated code with -jdump.
- Observe the crash.

> It's probably worth increasing the mmap floor a bit so the
> malloc heap size and LuaJIT heap size are more equally balanced.

Don't worry. All current malloc() implementations continue to
allocate memory with mmap() if the brk() space runs out. One could
even do without the latter, but it has other advantages (avoids
repeated clearing of mapped pages by the kernel). A C program may
still allocate gigabytes of memory with malloc(), even if you
reduce the brk() space.

> Also, what is the reason for the hint + retry wrapper? AFAICT it just
> duplicates logic that already exists inside the kernel. I guess that the
> kernel prefers to return memory addressed above the program's data segment
> (as suggested in the x64 ABI) but on Mac OS X the LuaJIT build tells the
> linker to move the text and data segments above 4GB, so LuaJIT needs the
> hint to force it to allocate below 4GB. Is that right? I've attached a
> patch that simplifies the FreeBSD mmap wrapper a bit.

The retry case is needed if the kernel gives you some address
beyond the allowed range. Happens in reality, too.

The -image_base directive for the linker is optional and helps to
free up a bit more space below 2GB. It doesn't stop the OSX kernel
from allocating memory at unwanted addresses.

> The pre-GPLv3 gcc-4.2.1 docs say that only -mcmodel=small and
> -mcmodel=medium are supported, which restricts code to being in the bottom
> 2GB memory. More recent versions of GCC do support the large model
> which allows the program to sprawl all over the address space, though the
> larger pointers slow things down a bit.

I think you misunderstood the meaning of the memory models. The
code of the main executable is loaded below 2GB. But shared
libraries on x64 generally load above the 4GB mark and may be
spaced out by more than 2GB. The large model is only needed if the
code for a single module is larger than 2GB (rather uncommon).

> I tried fiddling with -Ttext=100000000 to move the executable but the
> linker complained of lack of space for the executable headers. (I'm not
> sure what logic led to that error message.) I tried frobbing the linker
> script to move the text segments but (as you might expect) the small model
> uses 32 bit relocations and checks that the results fit, which they don't
> if you try to place stuff above 4GB :-)
> 
> There are also PC-relative models defined by the x64 ABI. However if you
> try linking with --pic-executable it fails since crt1.o is not PIC.

PLT relocations on x64 are always 64 bit. Only intra-module
relocations need to be 32 bit in the small/medium memory models.
You may need to get more creative to move everything that's
needed. But I don't think it's that important to free up a couple
megabytes of address space.

--Mike