Annotation of wikisrc/projects/project/make-kqueue.mdwn, revision 1.2

1.1       dholland    1: [[!template id=project
                      2: 
                      3: title="Speed up BSD make with kqueue/inotify"
                      4: 
                      5: contact="""
                      6: [tech-toolchain](mailto:tech-toolchain@NetBSD.org)
                      7: """
                      8: 
                      9: category="userland"
                     10: difficulty="hard"
                     11: duration="3-4 months"
                     12: 
                     13: description="""
                     14: 
                     15: Currently BSD make emits tons and tons of stat(2) calls in the course
                     16: of figuring out what to do, most notably (but not only) when matching
                     17: suffix rules.
                     18: This causes measurable amounts of overhead, especially when there are
                     19: a lot of files like in e.g. libc.
1.2     ! dholland   20: First step is to quantify this overhead so you can tell what you're
        !            21: accomplishing.
1.1       dholland   22: 
                     23: Fixing this convincingly requires a multi-step approach: first, give
                     24: make an abstraction layer for directories it's working with.
                     25: This alone is a nontrivial undertaking because make is such a mess
                     26: inside.
                     27: This step should also involve sorting out various issues where files
                     28: in other directories (other than the current directory) sometimes
                     29: don't get checked for changes properly.
                     30: 
                     31: Then, implement a cache under that abstraction layer so make can
                     32: remember what it's learned instead of making the same stat calls over
                     33: and over again.
                     34: Also, teach make to scan a directory with readdir() instead of
                     35: populating the cache by making thousands of scattershot stat() calls,
                     36: and implement a simple heuristic to decide when to use the readdir
                     37: method.
                     38: 
                     39: Unfortunately, in general after running a program the cache has to be
                     40: flushed because we don't know what the program did.
                     41: The final step in this project is to make use of kqueue or inotify or
                     42: similar when running programs to synchronize make's directory cache so
                     43: it doesn't have to be flushed.
                     44: 
                     45: As an additional step one might also have make warn when a recipe
                     46: touches files it has been declared to touch... but note that while
                     47: this is desirable it is also somewhat problematic.
                     48: 
                     49: """
                     50: ]]

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb