Re: Moai Cloud - Lua-based Web Services

[Tim Mensch <tim-lua-l@...>]

I looked at Tokyo/Kyoto Cabinet & Tyrant, but I swear I found something that eliminated it as an option, though I can't remember now what it was. :/ I thought it was that it took too much RAM, but it looks like 3MB of RAM is good for 1M records...and I have a 512MB VPS, so I think I can count on at least 32Mb of RAM for the database without breaking a sweat. And if I end up with more than 10M records, then I can afford a bigger server; the records would average about US$0.50 of value each to me, each, so I wouldn't be complaining at that point.

CouchDB does replication without almost any effort on my part, though, and that still wins for now, since my PRIMARY task is maintaining the client code.  At the point where I have enough load so horizontal replication of Nginx+CouchDB servers won't handle the load, I'll hire someone else to port to TokyoCabinet or MongoDB or Redis. It would be one of those "good problems to have." :)

Thanks for the advice,

Tim


On 7/12/2011 4:38 PM, David Hollander wrote:

>  Where I KNOW I'm not getting the speed is my connection to CouchDB; it's on the same host, 

If you just need a key-value store on the same host have you considered using Tokyo Cabinet ? [1]

It's the fastest on-disk persistant hashmap of non-fixed size, and has an excellent binding for Lua. It buffers very efficiently when writing to disk

so you can do 1million inserts per second, should blow CouchDB out of the water. In many situations, its best to just embed TokyoCabinet directly in your program and not worry about the occasional disk seek. Alternatively, you can isolate it in a fork using an RPC library, or use the Tokyo Tyrant server to access it asynchronously. TokyoTyrant allows you to write your queries as Lua functions embedded directly in the database server.

Currently, I use tokyocabinet to store serialized Lua into a hashmap, which is much faster to reload than using a JSON intermediary.

To make it nonblocking on disk seeks, I'm currently developing an async RPC format that lets me roll datastores by doing the following:

db = tokyocabinet.hdb()

db:open('posts.hdb', hdb.OCREAT+hdb.OWRITER)

data = "" -- creates a forked process

pagecache = cache(10)

-- define database method "tags"

data['tags'] = function(tagname)

  results={}

  for key, post in db:pairs() do

    post = loadstring('return '..post)()  -- load serialized lua table

    if post.tags:match(tagname) then

      table.insert(results, post)

    end

  end

  return results

end

-- define a handler that will call our database method

pagecache['tag/([^/]+)'] = function(tag_name)

   data.tags(tag_name, function(results)

        --do stuff with results table when database calls back

   end)

end

[1] http://fallabs.com/tokyocabinet/

On Tue, Jul 12, 2011 at 3:32 PM, Tim Mensch <tim-lua-l@bitgems.com> wrote:

On 7/12/2011 2:27 PM, Matt Towers wrote:
> On Jul 12, 2011, at 12:57 , Tim Mensch wrote:
>> But I was in a hurry and I couldn't find a ready-made MongoDB interface for Lua.
> Here ya go:  https://github.com/moai/luamongo

I guess I should have asked last week. :)

Everything is stable now, and there's no way I'm going to touch it, but
thanks for the link. If I have speed issues that I track to CloudDB,
I'll use this for sure -- and for my next game I'll go there first. :)

Tim