> I am not pushing the strings and using luaL_checkoption, but rather a checkoption function that takes the name as const char * argument instead of getting it from the Lua stack.
That means writing even more code than I originally thought. I think you could make your life easier by not using strtok_r for parsing (if you insist on it).
> I did this before, but then I came accross a situation where additional parameters are specified after the flags:
You could move all of the flag parameters after all the other parameters. The really troublesome case would be one when your (underlying) function is variadic and has flag parameters, then you would need to come up with some clever scheme or use tables.
You could create helper function just for flags that you use. For example, say you have flag group FOO and flag group BAR. The you could create variadic functions foo() and bar(), and use them as follows:
whatever(a, ,b ,c, foo('A', 'B'), d, e, bar('X', 'Y', 'Z'), f) -- eq. FOO_A | FOO_B, BAR_X | BAR_Y | BAR_Z
This, I think, is more readable than tables.