DTrace is a Dynamic Tracing framework developed by Sun and ported to NetBSD. It enables extensive instrumentation of the kernel and user space. See the [DTrace Community Page](http://dtrace.org) for more information.
Also see [DTrace Introduction](http://dtrace.org/guide/preface.html), Brendan Gregg's [DTrace one liners](http://www.brendangregg.com/DTrace/dtrace_oneliners.txt) and his notes for [DTrace on FreeBSD](https://wiki.freebsd.org/DTrace/).

# Current status

## Supported platforms

DTrace is a work-in-progress effort and it is for x86 systems and some arm boards.

* i386 and amd64
* earm* (evbarm and armv4 based ports)

## Supported providers

* DTrace: What to do when a script BEGINs, ENDs, ERRORs
* FBT: Function Boundary Tracing
* IO: Disk I/O
* Lockstat: Kernel Lock Statistics
* Proc: Process and thread related events
* Profile: Time based interrupt event source for Profiling
* SDT: Statically Defined Tracing
* Syscall: System Calls
* Syscall Linux (32bit & 64 bit): System calls via the Linux binary emulation layer
* VFS: Filesystem operations (confined to namecache events at time of writing - 8.99.22)

## TODO for netbsd-7

* Measure effect of `options KDTRACE_HOOKS` on system performance.
* Determine whether the profile module works and list it here.
* Integrate [[riz|users/riz]]'s syscall provider patch.

# How to use

##  Building DTrace 

You need the following options in your kernel: 
    
    options         KDTRACE_HOOKS   # kernel DTrace hooks
    options         MODULAR

Optionally:

    options         INSECURE   # permit modules to loaded from user space once system has gone multiuser and securelevel has been raised.

A Distribution needs to be built with the options `MKDTRACE=yes` and `MKCTF=yes`, this is taken care of automatically and doesn't need to be specified manually. The list of platforms it is applied to automatically is set in `src/share/mk/bsd.own.mk`

Set the system to load the solaris and dtrace related modules in `/etc/modules.conf`, for a list of available modules, see `/stand/$MACHINE/$VERSION/modules/`

For example, add the following to `/etc/modules.conf` (the file may not exist already on a system):
    
- `solaris`
- `dtrace`
- `dtrace_fbt`
- `dtrace_lockstat`
- `dtrace_profile`
- `dtrace_sdt`
- `dtrace_syscall`
- `dtrace_syscall_linux`
    
A `dtrace` device node is created automatically in `/dev/dtrace` when the modules are loaded into place.
    
List the dtrace probes 
    
    dtrace -l
    
       ID   PROVIDER            MODULE                          FUNCTION NAME
        1     dtrace                                                     BEGIN
        2     dtrace                                                     END
        3     dtrace                                                     ERROR
        4        fbt            netbsd             AcpiAcquireGlobalLock entry
        5        fbt            netbsd             AcpiAcquireGlobalLock return
        6        fbt            netbsd             AcpiAllocateRootTable entry
        7        fbt            netbsd                    AcpiAttachData entry
        . 
        .
    29129        fbt           solaris                   zfs_vop_getattr entry 
    29130        fbt           solaris                   zfs_vop_getattr return
    29131       proc                                                     create
    29132       proc                                                     exec 
        .
        .
    29140       proc                                                     lwp_start
    29141       proc                                                     lwp_exit


## Running hello world 

Put the following into the file hello.d:
    
    BEGIN
    {
        trace("Hello world");
        exit(0);
    }
    

Run the hello world script: 
    
    dtrace -s hello.d
    
    dtrace: script './hello.d' matched 1 probe
    CPU     ID                    FUNCTION:NAME
      0      1                           :BEGIN   Hello world

The same script could be executed as a one liner on the shell, using

    dtrace -n 'BEGIN { trace("Hello world"); exit(0); }'

## A more complex example

The following script traces the execution of a sleep operation
in the kernel. Put it in sleep.d:
    
    #pragma D option flowindent

    syscall::nanosleep:entry
    /execname == "sleep" && guard++ == 0/
    {
            self->traceme = 1;
    }

    fbt:::
    /self->traceme/
    {}

    syscall::nanosleep:return
    /self->traceme/
    {
            self->traceme = 0;
            exit(0);
    }

Start the script running:

    dtrace -s sleep.d

This will take a while as the script instruments every function in the
kernel. When it's ready, it will print a message like "dtrace: script
'sleep.d' matched 59268 probes".  Then execute a "sleep 2" in another
shell.

## Tools included in base

Starting with NetBSD-8, on builds where `MKDTRACE=yes` is set, scripts from
[Brendan Gregg's DTrace toolkit](https://github.com/opendtrace/toolkit/) are installed in base as standard.

At present, the following scripts are installed in `/usr/sbin`: 

- `dtruss` - An implementation of the truss utility in DTrace which traces the system calls
made by a process
- `execsnoop` - snoop on execution of processes as they occur
- `opensnoop` - snoop on openning of files as they occur
- `procsystime` -  print process system call time details.

## Troubleshooting

The Compact C Type Format (CTF) has a 2^15 limit on types which can overflow, this prevents DTrace from
working correctly.

Check the number of types using `ctfdump` e.g

    ctfdump -S /netbsd

Note the line which states `total number of types`, the value should by less than 32768.

If overflow is not an issue, `libdtrace(3)` can provide some insight into what is going on via an
environment variable. Define `DTRACE_DEBUG` before tracing.

     env DTRACE_DEBUG= execsnoop