lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]


Hello Lua Users!

Announcing another (slightly faster) JSON parser for Lua: lua-simdjson!
* https://github.com/FourierTransformer/lua-simdjson

Feel free to try it out:
* luarocks install --server=https://luarocks.org/dev lua-simdjson

It does require a Lua build system with a modern c++ compiler, and will only run on a 64 bit system.

lua-simdjson, as the name implies, is a Lua binding around the incredibly fast simdjson library (https://simdjson.org/). Having worked on a fast lua csv parser (ftcsv), simdjson caught me eye, in it's unique way of parsing. It uses various architecture instructions to do multi-byte parsing of JSON. I noticed there weren't any bindings to Lua, and figured I could learn more about lua/c++ bindings. I spent some time on it and got something going! As this is my first Lua binding, I was hoping someone could take a look and offer feedback on how I implemented it, before doing a general release.

## API (up for discussion)
For now, there are a few methods for parsing json:
* local simdjson = require("simdjson") -- to initialize.
* 'simdjson.parse' and 'simdjson.parseFile' will parse the entire JSON document (either in memory or as a file on disk respectively) and return a normal Lua table.
* 'simdjson.open' and 'simdjson.openFile' will parse the JSON document, but keep it in memory as simdjson's internal structure. Then, you have to use a JSON pointer to access the internal elements (ex: "statuses/0/id").

The open methods end up being a bit quicker if you know what elements you want from the JSON body, and don't need large chunks of it. It could also be used to lazily return elements, but I ran into some issues with nesting (ex: myTable["deeply"]["nested"]), any help here would be appreciated.

I plan to keep it up-to-date with new releases of simdjson, and don't want to add too much onto it. However, having said that, there currently isn't an encoder with simdjson, and I feel like people might want that in a JSON library, so I may work on one in the future. If someone wants to help out, that would be great!

## License
It's licensed under the Apache-2.0 license, inline with the original simdjson.

Let me know if there are any questions!
--FourierTransformer