On Tue, Feb 8, 2011 at 3:41 AM, David Given
<dg@cowlark.com> wrote:
On 07/02/11 17:23, Anatol Pomozov wrote:
[...]
> There is an interesting build tool called Tup [1] that uses reverse
> algorithm. First it finds all files has been changed and then
> constructs subgraph that needs to be rebuild. It is a huge time-saver
> for incremental builds in large projects. See more info in this paper
> [2], and the result at this page [3]
That's very interesting --- I have actually been thinking a revisit to
Prime Mover; it works great for me but there are places where it needs a
redesign. One of its problems is that while it's hugely faster than
recursive make it's not great on truly huge source trees. Adopting some
of tup's design features ought to make things much snappier, as well as
allowing parallel builds.
Discussing how an "ideal" build system should look like is one of my favorite activities :)
I would split any build tool into 2 parts:
1) backend, the part that is responsible to dependencies graph manipulation and running commands
2) frontend, the user-visible part. This is the language that you describe Makefiles (or Tupfile or whatever filename you use).
The tup's backend is great if not ideal. Persistent graph, inotify for file changes, checking command dependencies by using LD_PRELOAD/ptrace. I played a little bit with Tup + Android source tree (more than 200K source files). On my computer Android null build takes 6 milliseconds with Tup. None of existing build tools known to me can reach this number. Current build system based on Make takes ~40 secs for null build.
Different story is with Tup's frontend. Tup uses hand-made language similar to Make. The language is great for small project without a lot of configuration. I tried to use it for Android. But more I used it for Android more I saw limitations of the existing language. Android build is very "customizable" and uses a lot of weird stuff as "for cc and cpp HOST compiler should use these CFLAGS, but if ARM_MODE variable is set use other CFLAGS", or "if module A is compiled with foo flag enabled, then module B and C must be compiled with the same flag".
And more I use Makefile-like syntax more I wanted to use some "real" language. Lua or Tcl are good candidates for embedding it to Tup.
There is actually patch that makes possible to include Lua fragments into Tupfiles
http://groups.google.com/group/tup-users/files. But I think it is be better if whole Tupfile can be implement in Lua in some sort of high-level abstractions. Such as (very rough example):
srcs = {'hello.c', 'world.c'}
if (configuration.debug) then
srcs += {'debug.c'}
end
c_library('some_name', srcs, shared=true, debug=false, cflags="-additional -cflags", out = {'device', 'host'})
java_library('some_java', glob('srcs/**/*.java'), dependencies = {'some_name.so'}, flags = '-Xlint')
Lua's ability to read and write data very
quickly should avoid the need for a transactional database like SQLite,
too...
Speed of Lua will be easily discarded by amount of files needed to parse. Huge projects (such as Chrome and Android) have thousands of Makefiles. With "parse them every time" strategy you'll waste a lot of cpu/io resources. For example - initial make phase (stat + parse Makefiles + building graph) for Android takes ~40 sec on my new 4 core cpu workstation.
Tup uses disk storage to avoid parsing/building graph every time, and it is a source of speed of Tup's incremental builds.
Resume:
I think there is more benefit in using existing Tup backend + adding a high-level language, rather than implementing build tool in Lua.
