I wonder what is the use of passing a 10MB long string as the "name" of the library to load. As far as I know, all I/O APIs should have length limitations in the filesystem. Though path lengths longer that 256 bytes are now becoming common in virtualized OSes and with containers, but still limited (in Windows the 256 characters limit exists by default but has to be explicitly removed by a security token, but it's then up to the application to perform early validation. I don't see anyway any use for having a single file name (independently of the length of the full path) to be larger than 256 bytes.
So a simple validation would just look for a "/" or null byte in this limit of 256 bytes, allowing then dlopen() to create longer paths is needed (from the dynamic link path environment).
Note that GCC uses a special implementation of dlopen() in its library, which also has its own "strange" (but documented) bug as it uses an internal structure with addresses that are resolved incorrectly between compile time and run-time, and requiring some part of the code to be compiled in "position-independant code" mode.
The bug is also tied to the C library you use with GCC and may be fixed depending on versions of this library, because dlopen() is not an OS API but an implementation of the POSIX interface inherited from old versions of SunOS. The implementation of this library is in charge to perform lookups of paths, and for this it may create local strings not allocated on the heap but possibly using alloca() on the stack, assuming that these buffers would not be overly large to fit with the dlopen() parameter or with the environment strings listing the search path.
Anyway dlopen() has mostly been tested with common library names that are short such as "libm.so".
And using random library names with dlopen() can be a security risk. library names specified in user scripts should then have a minimal validation.
Note that it is not possible to use the pointer directly as the implementation needs to derive full path names from it for the search...
