Build systems suck. Can Lua be the basis for a better build system?

[Spencer Parkin <spencer.parkin@...>]

Admittedly, my knowledge of build systems is limited, but I can already tell you that for as long as I've been working with them, (which has been as long as I've been programming, (which is a fairly long time.)), that I really, really hate build systems.

Let's start with dev-studio and its property pages.  Yuck.  Need I say more?

Next, the make utility.  After a great deal of effort, I finally created a make-file for one of my static library projects.  Here's a snippet...

define BUILD_OBJ_RULE =
$(1): $(2)
    $$(CPP) $$(FLAGS) -c $$< -o $$@
endef
$(foreach src, $(SRCS), $(eval $(call BUILD_OBJ_RULE, $(patsubst %.cpp, $(BUILD)/%.o, $(notdir $(src))), $(src))))

Yikes!  Yes, it's ugly and LISPY, but it's also probably embarassing, since no real programmer that actually knew what they were doing would write that.  Moving on!

How about jam?  To do in jam what was just done above in make, (though I really don't want to draw attention to that again), is so much easier to do.  Why?  Well, I think jam solves a lot of problems with build systems, but 1) it's a dead project, and 2) it's not easy to learn.  But remember, this is coming from a guy who really doesn't know what-the-heck he's talking about.

Alrighty.  Now that I've just discredited myself, I'm sure you'll all want to hear where all this is going.  (Ha!)  We are on a Lua mailing list after all.  ;) ;)

Surely there are other build systems out there that I don't know about, but assuming for the moment that they're not any better than the above mentioned build systems, how can we make a build system that just doesn't suck?  The ideal build system to me would have the following key characteristics.

1) Use an appropriate language for the build system.  Don't just make up some dumb language for the new build system.  If you want support for path manipulation, provide that in the form of a library for your language.  The language also doesn't need to have built-in knowledge of the fact that we're building a dep-graph either.  Provide dep-graph construction as an API.

2) Make it simple!  What is a build system?  It's just a darn dependency graph, isn't it?  Why make exposing the use of such a thing complicated?
3) Flexibility.  Maybe this system doesn't even care what a file system is.  Then you could probably use it for anything, not just building a project.  Being this abstract can make things more simple (or complicated), even if it's more work to make your make-file.  (I don't care that the jamfile to makefile line ratio is 1:5 or whatever, I just care that I can easily figure out how to build my projects.  I don't get excited about Perl one-liners if I don't know what the heck they do.)

4) It's got to be fast.  Be able to take advantage of multiple processors.
5) Scalability.  Obviously, being smart in your choice of algorithms is one means to this end.  If it doesn't scale, it sucks, which might mean lumping for c++ projects, but let's forget about that for now, leaving that as a detail needing solving by the user, not the build system.

So here's a bit of brain storming.  How about a build system broken down into two parts.  (I'm not claiming any originality here.  This may sound a bit like jam.)

1) Dependency graph generation, and

2) Dependency graph execution.

That's it!  Don't make it any harder than that!

Creation of the dep-graph can be a complex task for big projects, because some analysis of the project tree is needed to determine what the entire dep-graph is.  This can take a considerable amount of time.  Therefore, the build system should not proceed to the execution phase immediately upon generation of the dep-graph.  It should cache that sucker (the dep-graph) out to disk so that on a subsequent call to build the project (or any portion of it), it can simply hit that cache.  Invalidation of certain parts of that cache should somehow be detectable, and only a minimal/incremental rebuild of the dep-graph cache should ever be performed.

Enter graph execution.  The process that takes care of this is simply given a target (node of the dep-graph), and asked to up-date it.  It is not hard to conceive of an algorithm that will do this, and also get a bunch of processes going in parallel, (that can be safely run in parallel according to the dep-graph), to get the job done as quickly as possible.  Clearly any already-up-to-date target needs no processing, and any repeated request to build a given target, (assuming the first request succeeded), should be a no-op.

That's it, right?  What else does a build system have to be?  There are quite a few details to be worked out about handling a file-system, but does it have to be anymore complicated than all this?  I'm tempted to let any file-system details be left entirely to the user so that the build system doesn't even know what a file is.  Extension of the core build system could add support for all that.  The build system should be as small and light-weight as possible.

So now what about implementation?  A few ideas...

1) Use Lua as the make-file language.  By virtual of making this choice, the make-file language, (unlike the actual Makefile language), becomes immediately endowed with all the power of Lua, which is already light-years beyond what the Makefile language itself can do.  Plus, if you already happen to know Lua, (which may be a language of greater worth in learning than some weirdo jam language), then you're already half-way there in the task of learning the build system!  Hurray!

2) Write the dep-graph generator in Lua.  This may be a good choice since all the makefiles are just Lua files.
3) Write the dep-graph cache to disk as a Lua file.  It's just a table with a bunch of targets in it.

4) Write the dep-graph executor in C/C++.  Be as fast as possible!

I'd be surprised if anyone's bothered to read this far.  Sorry for the long e-mail.

Any thoughts?

(I'm so dumb staying up 'til 1 am.)