Contents

  1. Introduction
  2. Example: OMAP 2420
    1. Default TISDP2420, 2251 kb
    2. Optimize for size with -Os, 2059 kb
    3. Remove debugging symbols, without -g, 2059 kb
    4. DIAGNOSTIC, 1867 kb
    5. VERBOSE_INIT_ARM, 1867 kb
    6. KTRACE, 1867 kb
    7. DDB, 1867 kb
    8. COMPAT_30, 1867 kb
    9. ksyms, 1510 kb
    10. PTRACE, 1510 kb
    11. FFS, 1510 kb
    12. MSDOSFS, 1509 kb
    13. PTYFS, 1509 kb
    14. bpfilter, 1445 kb
    15. INSECURE, 1445 kb
    16. INET6, 1316 kb
    17. MALLOC_NOINLINE, 1316 kb
    18. PIPE_SOCKETPAIR, 1316 kb
    19. VMSWAP, 1316 kb
    20. SYSVMSG, SYSVSEM, SYSVSHM, 1252 kb
    21. rnd, 1252 kb
    22. md, 1252 kb
    23. maxusers 2, 1252 kb
    24. MFS, 1124 kb
    25. COREDUMP, 1124 kb
  3. Kernel memory consumption
    1. top
    2. ps
    3. pmap
    4. vmstat
  4. References

Introduction

The NetBSD kernel is not big, but it can be made smaller. Generic size reduction steps are:

Here is an example which shows how these steps were applied to a NetBSD kernel for an ARM development board. As a result the kernel size got reduced from 2251 to 1124 kilobytes.

Since kernel executable size is not the only measure of kernel memory usage, some tools and their example output are also shown.

Example: OMAP 2420

The board receives the kernel via TFTP and the root filesystem is on NFS. Most important components on the board are the serial ports and the Ethernet adapter.

NetBSD current has a default kernel configuration for the board: sys/arch/evbarm/conf/TISDP2420. This file includes a number of other configuration files, but after compilation the resulting configuration is available as sys/arch/evbarm/compile/obj/TISDP2420/config_file.h.

The configuration file structure and most options are explained in options(4) manual page. In this example options were changed by adding a no options OPTION_NAME directive or an options OPTION_NAME directive to the end of the default configuration file. By adding these to end of the file, previous declarations from included options files were easily replaced without changing the included files.

Default TISDP2420, 2251 kb

Default kernel is build with -O2 optimization and stripped with objdump. Many debugging options are still enabled.

$ ls -l sys/arch/evbarm/compile/obj/TISDP2420/netbsd*
-rwxr-xr-x 1 test test  2614515 2008-06-23 11:32 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test  2305536 2008-06-23 11:32 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rwxr-xr-x 1 test test 13908834 2008-06-23 11:32 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.gdb
-rw-r--r-- 1 test test  3440335 2008-06-23 11:32 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


$ size sys/arch/evbarm/compile/obj/TISDP2420/netbsd
   text    data     bss     dec     hex filename
1902956  339456  214740 2457152  257e40 sys/arch/evbarm/compile/obj/TISDP2420/netbsd

Dmesg shows how much memory is available after boot, when the kernel executable been loaded to RAM and most of the RAM based data structures have been initialized:

# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80710DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255c000, paddr=0x80740DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x80760gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

Optimize for size with -Os, 2059 kb

Compiler can optimize for size when build variable DEFCOPTS is set to -Os instead of the default -O2.

-rwxr-xr-x 1 test test  2458323 2008-06-25 09:48 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test  2109012 2008-06-25 09:48 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rwxr-xr-x 1 test test 13390766 2008-06-25 09:48 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.gdb
-rw-r--r-- 1 test test  3446815 2008-06-25 09:48 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1764840  339540  214768 2319148  23632c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg | egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58220 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory
allocated at vaddr=0x8255d000, paddr=0x80710DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80730gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

Remove debugging symbols, without -g, 2059 kb

Build variable DEBUG contains the -g flag. Build without it. Results show that the kernel was already stripped of debug symbols by objdump, so this step is not usefull.

-rwxr-xr-x 1 test test 2502667 2008-06-25 09:58 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 2109012 2008-06-25 09:58 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3238409 2008-06-25 09:58 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1764840  339540  214768 2319148  23632c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58220 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80710DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80730gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

DIAGNOSTIC, 1867 kb

Kernel build without DIAGNOSTIC support.

-rwxr-xr-x 1 test test 2304678 2008-06-25 10:05 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1912404 2008-06-25 10:05 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3208060 2008-06-25 10:05 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1567336  339540  214768 2121644  205fac sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58408 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

VERBOSE_INIT_ARM, 1867 kb

Kernel build without verbose ARM specific boot messages. VERBOSE_INIT_ARM seems to depend on DDB support.

-rwxr-xr-x 1 test test 2304616 2008-06-25 10:22 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1912404 2008-06-25 10:22 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3207741 2008-06-25 10:22 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1565468  339540  214768 2119776  205860 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58408 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

KTRACE, 1867 kb

Kernel without KTRACE system call tracing support. KTRACE seems to depend on DDB kernel debugger support, since boot hangs without it.

-rwxr-xr-x 1 test test 2303792 2008-06-25 10:39 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1912372 2008-06-25 10:39 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3198575 2008-06-25 10:39 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1557776  339508  214704 2111988  2039f4 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58408 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

DDB, 1867 kb

Build kernel without DDB in kernel debugger.

-rwxr-xr-x 1 test test 2260235 2008-06-25 10:42 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1911944 2008-06-25 10:42 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3076696 2008-06-25 10:42 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1512444  339080  209440 2060964  1f72a4 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


total memory = 62464 KB
avail memory = 58416 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

COMPAT_30, 1867 kb

Kernel without NetBSD 3.0 compatibility.

-rwxr-xr-x 1 test test 2259424 2008-06-25 10:55 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1911944 2008-06-25 10:55 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3031468 2008-06-25 10:55 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1509828  339080  209440 2058348  1f686c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58416 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x806b0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x806e0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x80700gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

ksyms, 1510 kb

Kernel without /dev/ksyms support.

-rwxr-xr-x 1 test test 1925248 2008-06-25 10:59 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1546376 2008-06-25 10:59 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3023265 2008-06-25 10:59 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1503148   39048  209056 1751252  1ab8d4 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x8253a000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x82560000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82586000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

PTRACE, 1510 kb

Kernel without process tracing support.

-rwxr-xr-x 1 test test 1924948 2008-06-25 11:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1546376 2008-06-25 11:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3015392 2008-06-25 11:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1499312   39048  209056 1747416  1aa9d8 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x8253a000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x82560000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82586000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

FFS, 1510 kb

Kernel without fast filesystem support.

-rwxr-xr-x 1 test test 1924948 2008-06-25 11:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1546376 2008-06-25 11:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 3015329 2008-06-25 11:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1499248   39048  209056 1747352  1aa998 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x8253a000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x82560000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82586000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

MSDOSFS, 1509 kb

Kernel without FAT filesystem support.

-rwxr-xr-x 1 test test 1887407 2008-06-25 11:21 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1546152 2008-06-25 11:21 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2955657 2008-06-25 11:21 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1463132   38824  208672 1710628  1a1a24 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82538000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255e000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82584000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

PTYFS, 1509 kb

Kernel without pseudo TTY filesystem and pseudo-device pty.

-rwxr-xr-x 1 test test 1883462 2008-06-25 11:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1545916 2008-06-25 11:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2911049 2008-06-25 11:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1446512   38588  208536 1693636  19d7c4 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58764 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80660DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255c000, paddr=0x80680DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x806b0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

bpfilter, 1445 kb

Kernel without Berkeley packet filter pseudo-device support.

-rwxr-xr-x 1 test test 1848968 2008-06-25 11:46 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1480372 2008-06-25 11:46 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2896335 2008-06-25 11:46 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1435984   38580  208272 1682836  19ad94 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58828 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255c000, paddr=0x80670DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x806a0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

INSECURE, 1445 kb

Kernel without integrity protection.

-rwxr-xr-x 1 test test 1848968 2008-06-25 11:54 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1480372 2008-06-25 11:54 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2896335 2008-06-25 11:54 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1435984   38580  208272 1682836  19ad94 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58828 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255c000, paddr=0x80670DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x806a0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

INET6, 1316 kb

Kernel without IPv6 support.

-rwxr-xr-x 1 test test 1666318 2008-06-25 11:57 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1348532 2008-06-25 11:57 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2666559 2008-06-25 11:57 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1271320   37812  199760 1508892  17061c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58960 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80630DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80680gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

MALLOC_NOINLINE, 1316 kb

Kernel without inlined malloc functions.

-rwxr-xr-x 1 test test 1666318 2008-06-25 12:01 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1348532 2008-06-25 12:01 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2666559 2008-06-25 12:01 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1271320   37812  199760 1508892  17061c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58960 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80630DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255d000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80680gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

PIPE_SOCKETPAIR, 1316 kb

Kernel with smaller but slower pipe implementation.

-rwxr-xr-x 1 test test 1665096 2008-06-25 12:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1348468 2008-06-25 12:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2658886 2008-06-25 12:04 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1264788   37748  199760 1502296  16ec58 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58964 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80630DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255d000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000,  
paddr=0x80670gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

VMSWAP, 1316 kb

Kernel without swap support. Requires a build patch, but may still crash the kernel when uvm statistics are called from top, ps etc. programs.

--- a/sys/uvm/uvm_pdpolicy_clock.c
+++ b/sys/uvm/uvm_pdpolicy_clock.c
@@ -262,12 +262,13 @@ uvmpdpol_balancequeue(int swap_shortage)
                /*
                 * if there's a shortage of swap slots, try to free it.
                 */
-
+#if defined(VMSWAP)
                if (swap_shortage > 0 && (p->pqflags & PQ_SWAPBACKED) != 0) {
                        if (uvmpd_trydropswap(p)) {
                                swap_shortage--;
                        }
                }
+#endif /* defined(VMSWAP) */

                /*
                 * if there's a shortage of inactive pages, deactivate.


-rwxr-xr-x 1 test test 1662968 2008-06-25 12:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1348392 2008-06-25 12:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2642042 2008-06-25 12:09 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1248392   37672  199248 1485312  16aa00 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 58968 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80630DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80650DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80670gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

SYSVMSG, SYSVSEM, SYSVSHM, 1252 kb

Kernel without system V message queues, semaphores and shared memory.

-rwxr-xr-x 1 test test 1627140 2008-06-25 12:15 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1282748 2008-06-25 12:15 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2612942 2008-06-25 12:15 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1225204   37564  198904 1461672  164da8 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80620DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80640DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80660gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

rnd, 1252 kb

Kernel without /dev/random pseudo-device.

-rwxr-xr-x 1 test test 1625313 2008-06-25 12:25 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1282684 2008-06-25 12:25 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2599464 2008-06-25 12:25 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1219556   37500  197944 1455000  163398 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80620DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80640DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000, 
paddr=0x80660gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

md, 1252 kb

Kernel without support for memory devices like ramdisks.

-rwxr-xr-x 1 test test 1624021 2008-06-25 12:29 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1282588 2008-06-25 12:29 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2581393 2008-06-25 12:29 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1214024   37404  197752 1449180  161cdc sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82537000, paddr=0x80620DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x8255d000, paddr=0x80640DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82583000,  
paddr=0x80660gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

maxusers 2, 1252 kb

Kernel with maxusers set to two instead of the default 32.

-rwxr-xr-x 1 test test 1624021 2008-06-25 12:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1282588 2008-06-25 12:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2581393 2008-06-25 12:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1214024   37404  197752 1449180  161cdc sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59548 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824b6000, paddr=0x805a0DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x824dc000, paddr=0x805c0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82502000, 
paddr=0x805e0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

MFS, 1124 kb

Kernel without memory filesystem.

-rwxr-xr-x 1 test test 1483269 2008-06-25 12:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1151292 2008-06-25 12:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2437216 2008-06-25 12:37 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1106936   37180  197688 1341804  14796c sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59672 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824b3000, paddr=0x80580DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x824d9000, paddr=0x805a0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824ff000, 
paddr=0x805d0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

COREDUMP, 1124 kb

Kernel without core dump support.

-rwxr-xr-x 1 test test 1482841 2008-06-25 13:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd
-rwxr-xr-x 1 test test 1151292 2008-06-25 13:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.bin
-rw-r--r-- 1 test test 2428587 2008-06-25 13:34 sys/arch/evbarm/compile/obj/TISDP2420/netbsd.map


   text    data     bss     dec     hex filename
1102708   37180  197688 1337576  1468e8 sys/arch/evbarm/compile/obj/TISDP2420/netbsd


# dmesg|egrep "memory|gpm"
total memory = 62464 KB
avail memory = 59672 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824b3000, paddr=0x80580DISPC: omap2_lcd_alloc_fb(): memory 
allocated at vaddr=0x824d9000, paddr=0x805a0DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x824ff000, 
paddr=0x805d0gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0
gpmc0: CS#0 valid, addr 0x04000000, size  64MB
gpmc0: CS#1 valid, addr 0x08000000, size  16MB
sm0 at gpmc0 addr 0x08000300 intr 188

Kernel memory consumption

The kernel executable size is not the only measure of kernel memory consumption. A number of userspace tools are able to show how the kernel and it's executable threads use memory while the system is running. Following chapters list some example tool output on the ARM OMAP 2420 board.

Using these tools is easy, but interpreting the numbers seems to require some indepth knowledge of the NetBSD kernel's memory management system uvm. More details on the meaning of these numbers as well as examples of found problems and countermeasures like kernel configurations and sysctl setting would be appreciated.

top

top shows overall memory usage in the system and an interesting process called system.

# top
load averages:  0.14,  0.03,  0.01             09:20:54
4 processes:   3 sleeping, 1 on CPU
CPU states:     % user,     % nice,     % system,     % interrupt,     % idle
Memory: 1916K Act, 208K Wired, 996K Exec, 428K File, 52M Free
Swap:
  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
   51 root      43    0   976K  784K CPU        0:00 14.00%  0.68% top
    0 root     125    0     0K 1008K schedule   0:00  0.00%  0.00% [system]
   40 root      85    0   980K  956K wait       0:00  0.00%  0.00% sh
    1 root      85    0    32K  500K wait       0:00  0.00%  0.00% init


# dmesg|grep -i mem
total memory = 62464 KB
avail memory = 58032 KB
DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82536000, paddr=0x80710DISPC: omap2_lcd_alloc_fb(): memory  
allocated at vaddr=0x8255c000, paddr=0x80740DISPC: omap2_lcd_alloc_fb(): memory allocated at vaddr=0x82582000, 
paddr=0x80760gpmc0 at mainbus0 base 0x6800a000: General Purpose Memory Controller, rev 2.0


# top -t
load averages:  0.00,  0.00,  0.00          up 0 days,  1:54   11:14:37
23 threads:    5 idle, 17 sleeping, 1 on CPU
CPU states:     % user,     % nice,     % system,     % interrupt,     % idle
Memory: 3844K Act, 208K Wired, 1308K Exec, 2044K File, 50M Free
Swap:
  PID   LID USERNAME PRI STATE      TIME   WCPU    CPU COMMAND      NAME
    0     6 root     223 IDLE       0:00  0.00%  0.00% [system]     softser/0
    0     3 root     222 IDLE       0:00  0.00%  0.00% [system]     softnet/0
    0     4 root     221 IDLE       0:00  0.00%  0.00% [system]     softbio/0
    0     5 root     220 IDLE       0:00  0.00%  0.00% [system]     softclk/0
    0     2 root       0 IDLE       0:00  0.00%  0.00% [system]     idle/0
   77     1 root      43 CPU        0:00  0.00%  0.00% top          -
    0     7 root     127 xcall      0:00  0.00%  0.00% [system]     xcall/0
    0    25 root     126 pgdaemon   0:00  0.00%  0.00% [system]     pgdaemon
    0     1 root     125 schedule   0:00  0.00%  0.00% [system]     swapper
    0    28 root     125 vmem_reh   0:00  0.00%  0.00% [system]     vmem_rehash
    0    27 root     125 aiodoned   0:00  0.00%  0.00% [system]     aiodoned
    0     9 root     125 cachegc    0:00  0.00%  0.00% [system]     cachegc
    0     8 root     125 vrele      0:00  0.00%  0.00% [system]     vrele
    0    26 root     124 syncer     0:00  0.00%  0.00% [system]     ioflush
    0    11 root      96 iicintr    0:00  0.00%  0.00% [system]     iic0

ps

ps can show the system light weight processes/threads too with -s flag and lname displays a more sesible name:

# ps -awxs -o uid,pid,lid,nlwp,pri,ni,vsz,rss,command,lname
UID PID LID NLWP PRI NI VSZ RSS COMMAND LNAME
  0   0  28   20 125  0   0 976 [system vmem_rehash
  0   0  27   20 125  0   0 976 [system aiodoned
  0   0  26   20 124  0   0 976 [system ioflush
  0   0  25   20 126  0   0 976 [system pgdaemon
  0   0  24   20  96  0   0 976 [system nfsio
  0   0  23   20  96  0   0 976 [system nfsio
  0   0  22   20  96  0   0 976 [system nfsio
  0   0  21   20  96  0   0 976 [system nfsio
  0   0  12   20  96  0   0 976 [system iic1
  0   0  11   20  96  0   0 976 [system iic0
  0   0  10   20  96  0   0 976 [system pmfevent
  0   0   9   20 125  0   0 976 [system cachegc
  0   0   8   20 125  0   0 976 [system vrele
  0   0   7   20 127  0   0 976 [system xcall/0
  0   0   6   20 223  0   0 976 [system softser/0
  0   0   5   20 220  0   0 976 [system softclk/0
  0   0   4   20 221  0   0 976 [system softbio/0
  0   0   3   20 222  0   0 976 [system softnet/0
  0   0   2   20   0  0   0 976 [system idle/0
  0   0   1   20 125  0   0 976 [system swapper
  0   1   1    1  85  0  32 500 init    -
  0  40   1    1  85  0 980 968 -sh     -
  0  62   1    1  42  0 976 676 ps -awx -
  0  63   1    1  85  0 980 784 less -r -

pmap

Displaying process 0 memory map with pmap shows the kernel's memory map.

# pmap 0
80000000  17680K read/write/exec     [ anon ]
81144000      4K read/write/exec     [ anon ]
81145000     20K read/write/exec     [ anon ]
8114A000   7328K read/write/exec     [ kmem_map ]
81872000   4096K read/write/exec     [ pager_map ]
81C72000    304K read/write/exec     [ anon ]
81CBE000      4K read/write/exec     [ anon ]
81CBF000    580K read/write/exec     [ anon ]
81D50000   4096K read/write/exec     [ exec_map ]
82150000   1200K read/write/exec     [ phys_map ]
8227C000   2096K read/write/exec     [ mb_map ]
82488000     20K read/write/exec     [ anon ]
8248D000      8K read/write/exec     [ uvm_aobj ]
8248F000    524K read/write/exec     [ anon ]
82512000     48K read/write/exec     [ uvm_aobj ]
8251E000     32K read/write/exec     [ anon ]
82526000      8K read/write/exec     [ uvm_aobj ]
82528000     48K read/write/exec     [ anon ]
82534000      8K read/write/exec     [ uvm_aobj ]
82536000    456K read/write/exec     [ anon ]
825A8000     16K read/write/exec     [ uvm_aobj ]
825AC000     12K read/write/exec     [ anon ]
825AF000     48K read/write/exec     [ uvm_aobj ]
825BB000      4K read/write/exec     [ anon ]
825BC000     16K read/write/exec     [ uvm_aobj ]
825C0000     56K read/write/exec     [ anon ]
825CE000   8192K read/write/exec     [ ubc_pager ]
82DCE000     52K read/write/exec     [ anon ]
82DDB000      8K read/write/exec     [ uvm_aobj ]
82DDD000     20K read/write/exec     [ anon ]
82DE2000      8K read/write/exec     [ uvm_aobj ]
82DE4000      4K read/write/exec     [ anon ]
82DE6000      4K read/write/exec     [ anon ]
82DE9000      8K read/write/exec     [ uvm_aobj ]
82DEB000      8K read/write/exec     [ anon ]
82DED000      8K read/write/exec     [ uvm_aobj ]
82DEF000     16K read/write/exec     [ anon ]
82DF7000     16K read/write/exec     [ uvm_aobj ]
82DFB000     12K read/write/exec     [ anon ]
82DFE000      8K read/write/exec     [ uvm_aobj ]
82E00000    400K read/write/exec     [ anon ]
82E68000     12K read/write/exec     [ anon ]
82E70000    576K read/write/exec     [ anon ]
 total    48064K

vmstat

vmstat can display a lot of NetBSD memory management details and statistics, especially if the kernel is compiled with KMEMSTAT options. Output of vmstat -C -m on the ARM board.

References

Add a comment