Diff for /wikisrc/tutorials/tuning_netbsd_for_performance.mdwn between versions 1.1 and 1.2

version 1.1, 2011/11/25 00:19:18 version 1.2, 2012/02/05 07:14:36
Line 1 Line 1
 **Contents**  **Contents**
   
 [[!toc levels=3]]  [[!toc levels=3]]
   
 #  Tuning the kernel  #  Tuning the kernel 
   
 ##  Process and file descriptor limits  ##  Process and file descriptor limits 
   
 **Before reading:**  **Before reading:**
   
 _These are mostly only demonstative values on how to tune your system for different needs. They are not some kind of an ultimate optional values. This article mostly aims to provide a quick overview on the ways to fine tune your system settings and being aware of the limitations._  _These are mostly only demonstative values on how to tune your system for different needs. They are not some kind of an ultimate optional values. This article mostly aims to provide a quick overview on the ways to fine tune your system settings and being aware of the limitations._
   
 ###  maxusers  ###  maxusers 
   
 The name is a bit misleading, because it doesn't set the number of users on the system, but used in the formula to calculate maximal number of allowed processes.  The name is a bit misleading, because it doesn't set the number of users on the system, but used in the formula to calculate maximal number of allowed processes. 
   
 You can find it in your kernel configuration file, something like this:  You can find it in your kernel configuration file, something like this: 
          
     maxusers        32      maxusers        32
          
   
 This is the default value, so if we look at the formulae we get process limit values:  This is the default value, so if we look at the formulae we get process limit values: 
          
     /usr/src/sys/param.h:      /usr/src/sys/param.h:
        #define  NPROC    (20 + 16 * MAXUSERS)         #define  NPROC    (20 + 16 * MAXUSERS)
          
     /usr/src/sys/conf/param.c:      /usr/src/sys/conf/param.c:
        #define  MAXFILES (3 * (NPROC + MAXUSERS) + 80)         #define  MAXFILES (3 * (NPROC + MAXUSERS) + 80)
          
   
 So we got 532 for NPROC (maximal number of processes) and 1772 for MAXFILES (maximal number of open file descriptors).  So we got 532 for NPROC (maximal number of processes) and 1772 for MAXFILES (maximal number of open file descriptors). 
   
 Some say that the maxusers should be set to the amount of RAM in megabytes.    Some say that the maxusers should be set to the amount of RAM in megabytes.  
 For reference, FreeBSD sets is automaticaly by this formula, but limits it's maximum to 384.  For reference, FreeBSD sets is automaticaly by this formula, but limits it's maximum to 384. 
   
 Setting it to 64 is always a safe bet if you don't want too much experimenting. Just change it in your kernel configuration file:  Setting it to 64 is always a safe bet if you don't want too much experimenting. Just change it in your kernel configuration file: 
          
     maxusers        64      maxusers        64
          
   
 Compile the new kernel with build.sh or manualy, install the new kernel and reboot.  Compile the new kernel with build.sh or manualy, install the new kernel and reboot. 
   
 You can check your limits with sysctl:  You can check your limits with sysctl: 
   
 With maxusers 32  With maxusers 32 
          
     $ sysctl proc.curproc.rlimit.maxproc      $ sysctl proc.curproc.rlimit.maxproc
     proc.curproc.rlimit.maxproc.soft = 160      proc.curproc.rlimit.maxproc.soft = 160
     proc.curproc.rlimit.maxproc.hard = 532      proc.curproc.rlimit.maxproc.hard = 532
          
          
     $ sysctl proc.curproc.rlimit.descriptors      $ sysctl proc.curproc.rlimit.descriptors
     proc.curproc.rlimit.descriptors.soft = 64      proc.curproc.rlimit.descriptors.soft = 64
     proc.curproc.rlimit.descriptors.hard = 1772      proc.curproc.rlimit.descriptors.hard = 1772
          
   
 With maxusers 64  With maxusers 64 
   
 You can check your limits with sysctl:  You can check your limits with sysctl: 
          
     $ sysctl proc.curproc.rlimit.maxproc      $ sysctl proc.curproc.rlimit.maxproc
     proc.curproc.rlimit.maxproc.soft = 160      proc.curproc.rlimit.maxproc.soft = 160
     proc.curproc.rlimit.maxproc.hard = 1044      proc.curproc.rlimit.maxproc.hard = 1044
          
          
     $ sysctl proc.curproc.rlimit.descriptors      $ sysctl proc.curproc.rlimit.descriptors
     proc.curproc.rlimit.descriptors.soft = 64      proc.curproc.rlimit.descriptors.soft = 64
     proc.curproc.rlimit.descriptors.hard = 3404      proc.curproc.rlimit.descriptors.hard = 3404
          
   
 ###  login.conf  ###  login.conf 
   
 So you can change the hard limits now. Let's see the soft limits.  So you can change the hard limits now. Let's see the soft limits. 
   
 or with ulimit:  or with ulimit: 
          
     $ ulimit -a      $ ulimit -a
     core file size          (blocks, -c) unlimited      core file size          (blocks, -c) unlimited
     data seg size           (kbytes, -d) 131072      data seg size           (kbytes, -d) 131072
     file size               (blocks, -f) unlimited      file size               (blocks, -f) unlimited
     max locked memory       (kbytes, -l) 80920      max locked memory       (kbytes, -l) 80920
     max memory size         (kbytes, -m) 242760      max memory size         (kbytes, -m) 242760
     open files                      (-n) 64      open files                      (-n) 64
     pipe size            (512 bytes, -p) 1      pipe size            (512 bytes, -p) 1
     stack size              (kbytes, -s) 2048      stack size              (kbytes, -s) 2048
     cpu time               (seconds, -t) unlimited      cpu time               (seconds, -t) unlimited
     max user processes              (-u) 160      max user processes              (-u) 160
     virtual memory          (kbytes, -v) 133120      virtual memory          (kbytes, -v) 133120
          
   
      
 You can set it with the file /etc/login.conf:  You can set it with the file /etc/login.conf: 
          
     default:\      default:\
         :path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin /usr/pkg/bin /usr/pkg/sbin /usr/local/bin:\          :path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin /usr/pkg/bin /usr/pkg/sbin /usr/local/bin:\
         :umask=022:\          :umask=022:\
         :datasize-max=3072M:\          :datasize-max=3072M:\
         :datasize-cur=1024M:\          :datasize-cur=1024M:\
         :maxproc-max=1044:\          :maxproc-max=1044:\
         :maxproc-cur=512:\          :maxproc-cur=512:\
         :openfiles-cur=256:\          :openfiles-cur=256:\
         :stacksize-cur=8M:          :stacksize-cur=8M:
          
   
 Next time you start the sytem, all users belonging to the _default_ login group will have the following limits:  Next time you start the sytem, all users belonging to the _default_ login group will have the following limits: 
          
     $ ulimit -a      $ ulimit -a
     coredump(blocks)     unlimited      coredump(blocks)     unlimited
     data(KiB)            1048576      data(KiB)            1048576
     file(blocks)         unlimited      file(blocks)         unlimited
     lockedmem(KiB)       124528      lockedmem(KiB)       124528
     memory(KiB)          373584      memory(KiB)          373584
     nofiles(descriptors) 256      nofiles(descriptors) 256
     processes            512      processes            512
     stack(KiB)           8192      stack(KiB)           8192
     time(cpu-seconds)    unlimited      time(cpu-seconds)    unlimited
          
   
 You may set different limits for different user, thus different services:  You may set different limits for different user, thus different services: 
          
     database:\      database:\
        :ignorenologin:\         :ignorenologin:\
        :datasize=infinity:\         :datasize=infinity:\
        :maxproc=infinity:\         :maxproc=infinity:\
        :openfiles-cur=1024:\         :openfiles-cur=1024:\
        :stacksize-cur=48M:         :stacksize-cur=48M:
          
   
 You should run this command after editing your login.conf:  You should run this command after editing your login.conf: 
          
     $ cap_mkdb /etc/login.conf      $ cap_mkdb /etc/login.conf
          
   
 You can assign the newly created login class to the desired user by doing something like this:  You can assign the newly created login class to the desired user by doing something like this: 
          
     $ usermod -L database pgsql      $ usermod -L database pgsql
          
   
 Let's check our limits again with sysctl:  Let's check our limits again with sysctl: 
          
     $ sysctl proc.curproc.rlimit.maxproc      $ sysctl proc.curproc.rlimit.maxproc
     proc.curproc.rlimit.maxproc.soft = 512      proc.curproc.rlimit.maxproc.soft = 512
     proc.curproc.rlimit.maxproc.hard = 1044      proc.curproc.rlimit.maxproc.hard = 1044
          
     $ sysctl proc.curproc.rlimit.descriptors      $ sysctl proc.curproc.rlimit.descriptors
     proc.curproc.rlimit.descriptors.soft = 256      proc.curproc.rlimit.descriptors.soft = 256
     proc.curproc.rlimit.descriptors.hard = 3404      proc.curproc.rlimit.descriptors.hard = 3404
          
   
 Much reasonable for a modern system.  Much reasonable for a modern system. 
   
 ##  System V interprocess communication  ##  System V interprocess communication 
   
 Shared memory and semaphores are part of the System V IPC. Using and fine tuning shared memory and semaphores can give you increased performance on your NetBSD server.  Shared memory and semaphores are part of the System V IPC. Using and fine tuning shared memory and semaphores can give you increased performance on your NetBSD server. 
   
 You can check it's settings with sysctl:  You can check it's settings with sysctl: 
          
     $ sysctl kern.ipc      $ sysctl kern.ipc
     kern.ipc.sysvmsg = 1      kern.ipc.sysvmsg = 1
     kern.ipc.sysvsem = 1      kern.ipc.sysvsem = 1
     kern.ipc.sysvshm = 1      kern.ipc.sysvshm = 1
     kern.ipc.shmmax = 8388608      kern.ipc.shmmax = 8388608
     kern.ipc.shmmni = 128      kern.ipc.shmmni = 128
     kern.ipc.shmseg = 128      kern.ipc.shmseg = 128
     kern.ipc.shmmaxpgs = 2048      kern.ipc.shmmaxpgs = 2048
     kern.ipc.shm_use_phys = 0      kern.ipc.shm_use_phys = 0
     kern.ipc.msgmni = 40      kern.ipc.msgmni = 40
     kern.ipc.msgseg = 2048      kern.ipc.msgseg = 2048
     kern.ipc.semmni = 10      kern.ipc.semmni = 10
     kern.ipc.semmns = 60      kern.ipc.semmns = 60
     kern.ipc.semmnu = 30      kern.ipc.semmnu = 30
          
   
 As you can see, the default maximum size of shared memory segment (shmmax) is 8 megabytes by default, but for a postgresql server you will most likely need about 128 megabytes.  As you can see, the default maximum size of shared memory segment (shmmax) is 8 megabytes by default, but for a postgresql server you will most likely need about 128 megabytes. 
   
 Note, that you cannot set shmmax directly with syctl, but you need to set the value in pages size with _kern.ipc.shmmaxpgs_.  Note, that you cannot set shmmax directly with syctl, but you need to set the value in pages size with _kern.ipc.shmmaxpgs_. 
   
 The default PAGE_SIZE is 4096, so if you want to set it to 128M, you have to do:  The default PAGE_SIZE is 4096, so if you want to set it to 128M, you have to do: 
          
     grimnismal# sysctl -w kern.ipc.shmmaxpgs=32768      grimnismal# sysctl -w kern.ipc.shmmaxpgs=32768
     kern.ipc.shmmaxpgs: 4096 -> 32768      kern.ipc.shmmaxpgs: 4096 -> 32768
          
   
 So the formula is: 128*1024*1024/4096 = 32768  So the formula is: 128*1024*1024/4096 = 32768 
   
 **You can make any sysctl change permanent by setting it in /etc/sysctl.conf**  **You can make any sysctl change permanent by setting it in /etc/sysctl.conf**
   
 You can also get detailed information on System V interprocess communication (IPC) facilities on the system with the following command:  You can also get detailed information on System V interprocess communication (IPC) facilities on the system with the following command: 
          
          
     $ ipcs      $ ipcs
     IPC status from <running system> as of Mon Dec  3 18:52:00 2007      IPC status from <running system> as of Mon Dec  3 18:52:00 2007
          
     Message Queues:      Message Queues:
     T        ID     KEY        MODE       OWNER    GROUP      T        ID     KEY        MODE       OWNER    GROUP
          
     Shared Memory:      Shared Memory:
     T        ID     KEY        MODE       OWNER    GROUP      T        ID     KEY        MODE       OWNER    GROUP
     m     65536    5432001 --rw-------    pgsql    pgsql      m     65536    5432001 --rw-------    pgsql    pgsql
          
     Semaphores:      Semaphores:
     T        ID     KEY        MODE       OWNER    GROUP      T        ID     KEY        MODE       OWNER    GROUP
     s     65536    5432001 --rw-------    pgsql    pgsql      s     65536    5432001 --rw-------    pgsql    pgsql
     s     65537    5432002 --rw-------    pgsql    pgsql      s     65537    5432002 --rw-------    pgsql    pgsql
     s     65538    5432003 --rw-------    pgsql    pgsql      s     65538    5432003 --rw-------    pgsql    pgsql
          
   
      
 You can also force shared memory to stay in physical memory. This means that they will be never paged out to swap.    You can also force shared memory to stay in physical memory. This means that they will be never paged out to swap.  
 You may set this behaviour with the _kern.ipc.shm_use_phys_ sysctl.  You may set this behaviour with the _kern.ipc.shm_use_phys_ sysctl. 
   
 ##  TCP Performance  ##  TCP Performance 
   
 TCP uses what is called the “congestion window” to determine how many packets can be sent at one time. The larger the congestion window size, the higher the throughput. The maximum congestion window is related to the amount of buffer space that the kernel allocates for each socket.  TCP uses what is called the “congestion window” to determine how many packets can be sent at one time. The larger the congestion window size, the higher the throughput. The maximum congestion window is related to the amount of buffer space that the kernel allocates for each socket. 
   
 So on high bandwidth line the bottleneck could be the buffer sizes.  So on high bandwidth line the bottleneck could be the buffer sizes. 
   
 Here's the formula for a network link's throughput:  Here's the formula for a network link's throughput: 
          
     Throughput = buffer size / latency      Throughput = buffer size / latency
          
   
 So if we reorganise it a bit, we get the formula of the ideal buffer size:  So if we reorganise it a bit, we get the formula of the ideal buffer size: 
          
     buffer size = 2 * delay * bandwidth      buffer size = 2 * delay * bandwidth
          
   
 The delay is the network latency, which is most commonly known as "ping".  The delay is the network latency, which is most commonly known as "ping". 
   
 I think I don't have to introduce this tool:  I think I don't have to introduce this tool: 
          
     $ ping yahoo.com      $ ping yahoo.com
     PING yahoo.com (66.94.234.13): 56 data bytes      PING yahoo.com (66.94.234.13): 56 data bytes
     64 bytes from 66.94.234.13: icmp_seq=0 ttl=50 time=195.596 ms      64 bytes from 66.94.234.13: icmp_seq=0 ttl=50 time=195.596 ms
     64 bytes from 66.94.234.13: icmp_seq=1 ttl=50 time=188.883 ms      64 bytes from 66.94.234.13: icmp_seq=1 ttl=50 time=188.883 ms
     64 bytes from 66.94.234.13: icmp_seq=2 ttl=51 time=192.023 ms      64 bytes from 66.94.234.13: icmp_seq=2 ttl=51 time=192.023 ms
     ^C      ^C
     ----yahoo.com PING Statistics----      ----yahoo.com PING Statistics----
     3 packets transmitted, 3 packets received, 0.0% packet loss      3 packets transmitted, 3 packets received, 0.0% packet loss
     round-trip min/avg/max/stddev = 188.883/192.167/195.596/3.359 ms      round-trip min/avg/max/stddev = 188.883/192.167/195.596/3.359 ms
          
   
 However ping(1) will give you the round-trip of the network link -- which is the twice of delay -- so the final formula is the following:  However ping(1) will give you the round-trip of the network link -- which is the twice of delay -- so the final formula is the following: 
          
     buffer size = RTT * bandwidth      buffer size = RTT * bandwidth
          
   
 Fortunately, there is an automatic control for those buffers in NetBSD. It can be checked and and enabled with sysctl:  Fortunately, there is an automatic control for those buffers in NetBSD. It can be checked and and enabled with sysctl: 
          
     net.inet.tcp.recvbuf_auto = 0      net.inet.tcp.recvbuf_auto = 0
     net.inet.tcp.recvbuf_inc = 16384      net.inet.tcp.recvbuf_inc = 16384
     net.inet.tcp.recvbuf_max = 262144      net.inet.tcp.recvbuf_max = 262144
     net.inet.tcp.sendbuf_auto = 0      net.inet.tcp.sendbuf_auto = 0
     net.inet.tcp.sendbuf_inc = 8192      net.inet.tcp.sendbuf_inc = 8192
     net.inet.tcp.sendbuf_max = 262144      net.inet.tcp.sendbuf_max = 262144
          
   
 The automatic setting for sendbuf and recvbuf is disabled in the default installation.    The automatic setting for sendbuf and recvbuf is disabled in the default installation.  
 The initial value for maximal send buffer and receive buffer is both 256k, which is very tiny.  The initial value for maximal send buffer and receive buffer is both 256k, which is very tiny. 
   
 A reasonable value is 16 megabytes, so you may set it to that value after you turned it on with sysctl:  A reasonable value is 16 megabytes, so you may set it to that value after you turned it on with sysctl: 
          
     net.inet.tcp.recvbuf_auto=1      net.inet.tcp.recvbuf_auto=1
     net.inet.tcp.sendbuf_auto=1      net.inet.tcp.sendbuf_auto=1
     net.inet.tcp.sendbuf_max=16777216      net.inet.tcp.sendbuf_max=16777216 
     net.inet.tcp.recvbuf_max=16777216      net.inet.tcp.recvbuf_max=16777216
          
   
 ##  Disk I/O  ##  Disk I/O 
   
 You may enable **experimental** buffer queue strategy for better responsiveness under high disk I/O load.    You may enable **experimental** buffer queue strategy for better responsiveness under high disk I/O load.  
 This options is _likely_ to stable but not yet the default.  This options is _likely_ to stable but not yet the default. 
   
 Enable them with the following lines in your kernel configuration file:  Enable them with the following lines in your kernel configuration file: 
          
       options         BUFQ_READPRIO        options         BUFQ_READPRIO
       options         BUFQ_PRIOCSCAN        options         BUFQ_PRIOCSCAN
          
   
 ##  Using optimized FLAGS with GCC  ##  Using optimized FLAGS with GCC 
   
 NOTE: Trying to utilise heavy optimalisations can make your system hard to debug, cause unpredictable behaviour or kill your pet. Especially use of -mtune is highly discouraged, because it does not improve performance considerably or at all compared to -march=i686, and gcc4 can't handle it correctly at least on athlon CPUs.  NOTE: Trying to utilise heavy optimalisations can make your system hard to debug, cause unpredictable behaviour or kill your pet. Especially use of -mtune is highly discouraged, because it does not improve performance considerably or at all compared to -march=i686, and gcc4 can't handle it correctly at least on athlon CPUs. 
   
 You can put something like this into your mk.conf, when you compile your packages and your system.  You can put something like this into your mk.conf, when you compile your packages and your system. 
          
     CPUFLAGS+=-march=i686      CPUFLAGS+=-march=i686
     COPTS+=-O2      COPTS+=-O2
          
   
 FIXME: This is only for building world  FIXME: This is only for building world 
          
     CFLAGS+="-O2 -march=i686"      CFLAGS+="-O2 -march=i686"
          
   
 FIXME: For packages  FIXME: For packages 
   
 For more detailed information about the possible CFLAG values, please read the [GNU C Compiler documentation](http://netbsd.gw.com/cgi-bin/man-cgi?gcc++NetBSD-current).  For more detailed information about the possible CFLAG values, please read the [GNU C Compiler documentation](http://netbsd.gw.com/cgi-bin/man-cgi?gcc++NetBSD-current). 
   
 #  References  #  References 
   
   * "17.4. Managing Kernel Resources". PostgreSQL 8.3beta3 Documentation. PostgreSQL Global Development Group. December 2007. Retrieved December 2, 2007. [[1]](http://www.postgresql.org/docs/8.3/static/kernel-resources.html)    * "17.4. Managing Kernel Resources". PostgreSQL 8.3beta3 Documentation. PostgreSQL Global Development Group. December 2007. Retrieved December 2, 2007. [[1]](http://www.postgresql.org/docs/8.3/static/kernel-resources.html)
   
   * "Performance Tuning a NetBSD Server". Eric Radman. Retrieved December 3, 2007. [[2]](http://eradman.com/article/bsdtuning1)    * "Performance Tuning a NetBSD Server". Eric Radman. Retrieved December 3, 2007. [[2]](http://eradman.com/article/bsdtuning1)
   
   * "TCP Tuning Guide". Lawrence Berkeley National Laboratory. Nov 15, 2007. Retrieved December 4, 2007. [[3]](http://www-didc.lbl.gov/TCP-tuning/)    * "TCP Tuning Guide". Lawrence Berkeley National Laboratory. Nov 15, 2007. Retrieved December 4, 2007. [[3]](http://www-didc.lbl.gov/TCP-tuning/)
   
 #  See also  #  See also 
   
   * [[Configuration examples]]    * [[Configuration examples]]
   
   

Removed from v.1.1  
changed lines
  Added in v.1.2


CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb