File:  [NetBSD Developer Wiki] / wikisrc / projects / project / make-kqueue.mdwn
Revision 1.2: download - view: text, annotated - select for diffs
Fri May 27 06:42:28 2016 UTC (3 years, 5 months ago) by dholland
Branches: MAIN
CVS tags: HEAD
mention that the current overhead should be quantified; thanks soda@

[[!template id=project

title="Speed up BSD make with kqueue/inotify"

contact="""
[tech-toolchain](mailto:tech-toolchain@NetBSD.org)
"""

category="userland"
difficulty="hard"
duration="3-4 months"

description="""

Currently BSD make emits tons and tons of stat(2) calls in the course
of figuring out what to do, most notably (but not only) when matching
suffix rules.
This causes measurable amounts of overhead, especially when there are
a lot of files like in e.g. libc.
First step is to quantify this overhead so you can tell what you're
accomplishing.

Fixing this convincingly requires a multi-step approach: first, give
make an abstraction layer for directories it's working with.
This alone is a nontrivial undertaking because make is such a mess
inside.
This step should also involve sorting out various issues where files
in other directories (other than the current directory) sometimes
don't get checked for changes properly.

Then, implement a cache under that abstraction layer so make can
remember what it's learned instead of making the same stat calls over
and over again.
Also, teach make to scan a directory with readdir() instead of
populating the cache by making thousands of scattershot stat() calls,
and implement a simple heuristic to decide when to use the readdir
method.

Unfortunately, in general after running a program the cache has to be
flushed because we don't know what the program did.
The final step in this project is to make use of kqueue or inotify or
similar when running programs to synchronize make's directory cache so
it doesn't have to be flushed.

As an additional step one might also have make warn when a recipe
touches files it has been declared to touch... but note that while
this is desirable it is also somewhat problematic.

"""
]]

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb