File:  [NetBSD Developer Wiki] / wikisrc / tutorials / tuning_netbsd_for_performance.mdwn
Revision 1.12: download - view: text, annotated - select for diffs
Wed Apr 17 16:51:15 2019 UTC (14 months, 3 weeks ago) by sevan
Branches: MAIN
CVS tags: HEAD
remove old wiki retrieved time stamp in links

    1: **Contents**
    2: 
    3: [[!toc levels=3]]
    4: 
    5: #  Tuning the kernel 
    6: 
    7: ##  Process and file descriptor limits 
    8: 
    9: **Before reading:**
   10: 
   11: _These are mostly only demonstative values on how to tune your system for different needs. They are not some kind of an ultimate optional values. This article mostly aims to provide a quick overview on the ways to fine tune your system settings and being aware of the limitations._
   12: 
   13: ###  maxusers 
   14: 
   15: The name is a bit misleading, because it doesn't set the number of users on the system, but used in the formula to calculate maximal number of allowed processes. 
   16: 
   17: You can find it in your kernel configuration file, something like this: 
   18:     
   19:     maxusers        32
   20:     
   21: 
   22: This is the default value, so if we look at the formulae we get process limit values: 
   23:     
   24:     /usr/src/sys/param.h:
   25:        #define  NPROC    (20 + 16 * MAXUSERS)
   26:     
   27:     /usr/src/sys/conf/param.c:
   28:        #define  MAXFILES (3 * (NPROC + MAXUSERS) + 80)
   29:     
   30: 
   31: So we got 532 for NPROC (maximal number of processes) and 1772 for MAXFILES (maximal number of open file descriptors). 
   32: 
   33: Some say that the maxusers should be set to the amount of RAM in megabytes.  
   34: For reference, FreeBSD sets is automaticaly by this formula, but limits it's maximum to 384. 
   35: 
   36: Setting it to 64 is always a safe bet if you don't want too much experimenting. Just change it in your kernel configuration file: 
   37:     
   38:     maxusers        64
   39:     
   40: 
   41: Compile the new kernel with build.sh or manualy, install the new kernel and reboot. 
   42: 
   43: You can check your limits with sysctl: 
   44: 
   45: With maxusers 32 
   46:     
   47:     $ sysctl proc.curproc.rlimit.maxproc
   48:     proc.curproc.rlimit.maxproc.soft = 160
   49:     proc.curproc.rlimit.maxproc.hard = 532
   50:     
   51:     
   52:     $ sysctl proc.curproc.rlimit.descriptors
   53:     proc.curproc.rlimit.descriptors.soft = 64
   54:     proc.curproc.rlimit.descriptors.hard = 1772
   55:     
   56: 
   57: With maxusers 64 
   58: 
   59: You can check your limits with sysctl: 
   60:     
   61:     $ sysctl proc.curproc.rlimit.maxproc
   62:     proc.curproc.rlimit.maxproc.soft = 160
   63:     proc.curproc.rlimit.maxproc.hard = 1044
   64:     
   65:     
   66:     $ sysctl proc.curproc.rlimit.descriptors
   67:     proc.curproc.rlimit.descriptors.soft = 64
   68:     proc.curproc.rlimit.descriptors.hard = 3404
   69:     
   70: 
   71: ###  login.conf 
   72: 
   73: So you can change the hard limits now. Let's see the soft limits. 
   74: 
   75: or with ulimit: 
   76:     
   77:     $ ulimit -a
   78:     core file size          (blocks, -c) unlimited
   79:     data seg size           (kbytes, -d) 131072
   80:     file size               (blocks, -f) unlimited
   81:     max locked memory       (kbytes, -l) 80920
   82:     max memory size         (kbytes, -m) 242760
   83:     open files                      (-n) 64
   84:     pipe size            (512 bytes, -p) 1
   85:     stack size              (kbytes, -s) 2048
   86:     cpu time               (seconds, -t) unlimited
   87:     max user processes              (-u) 160
   88:     virtual memory          (kbytes, -v) 133120
   89:     
   90: 
   91:   
   92: You can set it with the file /etc/login.conf: 
   93:     
   94:     default:\
   95:         :path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin /usr/pkg/bin /usr/pkg/sbin /usr/local/bin:\
   96:         :umask=022:\
   97:         :datasize-max=3072M:\
   98:         :datasize-cur=1024M:\
   99:         :maxproc-max=1044:\
  100:         :maxproc-cur=512:\
  101:         :openfiles-cur=256:\
  102:         :stacksize-cur=8M:
  103:     
  104: 
  105: Next time you start the sytem, all users belonging to the _default_ login group will have the following limits: 
  106:     
  107:     $ ulimit -a
  108:     coredump(blocks)     unlimited
  109:     data(KiB)            1048576
  110:     file(blocks)         unlimited
  111:     lockedmem(KiB)       124528
  112:     memory(KiB)          373584
  113:     nofiles(descriptors) 256
  114:     processes            512
  115:     stack(KiB)           8192
  116:     time(cpu-seconds)    unlimited
  117:     
  118: 
  119: You may set different limits for different user, thus different services: 
  120:     
  121:     database:\
  122:        :ignorenologin:\
  123:        :datasize=infinity:\
  124:        :maxproc=infinity:\
  125:        :openfiles-cur=1024:\
  126:        :stacksize-cur=48M:
  127:     
  128: 
  129: You should run this command after editing your login.conf: 
  130:     
  131:     $ cap_mkdb /etc/login.conf
  132:     
  133: 
  134: You can assign the newly created login class to the desired user by doing something like this: 
  135:     
  136:     $ usermod -L database pgsql
  137:     
  138: 
  139: Let's check our limits again with sysctl: 
  140:     
  141:     $ sysctl proc.curproc.rlimit.maxproc
  142:     proc.curproc.rlimit.maxproc.soft = 512
  143:     proc.curproc.rlimit.maxproc.hard = 1044
  144:     
  145:     $ sysctl proc.curproc.rlimit.descriptors
  146:     proc.curproc.rlimit.descriptors.soft = 256
  147:     proc.curproc.rlimit.descriptors.hard = 3404
  148:     
  149: 
  150: Much reasonable for a modern system. 
  151: 
  152: ##  System V interprocess communication 
  153: 
  154: Shared memory and semaphores are part of the System V IPC. Using and fine tuning shared memory and semaphores can give you increased performance on your NetBSD server. 
  155: 
  156: You can check it's settings with sysctl: 
  157:     
  158:     $ sysctl kern.ipc
  159:     kern.ipc.sysvmsg = 1
  160:     kern.ipc.sysvsem = 1
  161:     kern.ipc.sysvshm = 1
  162:     kern.ipc.shmmax = 8388608
  163:     kern.ipc.shmmni = 128
  164:     kern.ipc.shmseg = 128
  165:     kern.ipc.shmmaxpgs = 2048
  166:     kern.ipc.shm_use_phys = 0
  167:     kern.ipc.msgmni = 40
  168:     kern.ipc.msgseg = 2048
  169:     kern.ipc.semmni = 10
  170:     kern.ipc.semmns = 60
  171:     kern.ipc.semmnu = 30
  172:     
  173: 
  174: As you can see, the default maximum size of shared memory segment (shmmax) is 8 megabytes by default, but for a postgresql server you will most likely need about 128 megabytes. 
  175: 
  176: Note, that you cannot set shmmax directly with syctl, but you need to set the value in pages size with _kern.ipc.shmmaxpgs_. 
  177: 
  178: The default PAGE_SIZE is 4096, so if you want to set it to 128M, you have to do: 
  179:     
  180:     grimnismal# sysctl -w kern.ipc.shmmaxpgs=32768
  181:     kern.ipc.shmmaxpgs: 4096 -> 32768
  182:     
  183: 
  184: So the formula is: 128 * 1024 * 1024 / 4096 = 32768 
  185: 
  186: **You can make any sysctl change permanent by setting it in /etc/sysctl.conf**
  187: 
  188: You can also get detailed information on System V interprocess communication (IPC) facilities on the system with the following command: 
  189:     
  190:     
  191:     $ ipcs
  192:     IPC status from <running system> as of Mon Dec  3 18:52:00 2007
  193:     
  194:     Message Queues:
  195:     T        ID     KEY        MODE       OWNER    GROUP
  196:     
  197:     Shared Memory:
  198:     T        ID     KEY        MODE       OWNER    GROUP
  199:     m     65536    5432001 --rw-------    pgsql    pgsql
  200:     
  201:     Semaphores:
  202:     T        ID     KEY        MODE       OWNER    GROUP
  203:     s     65536    5432001 --rw-------    pgsql    pgsql
  204:     s     65537    5432002 --rw-------    pgsql    pgsql
  205:     s     65538    5432003 --rw-------    pgsql    pgsql
  206:     
  207: 
  208:   
  209: You can also force shared memory to stay in physical memory. This means that they will be never paged out to swap.  
  210: You may set this behaviour with the _kern.ipc.shm_use_phys_ sysctl. 
  211: 
  212: ##  TCP Performance 
  213: 
  214: ### Socket buffers
  215: 
  216: TCP uses what is called the “congestion window” to determine how many packets can be sent at one time. The larger the congestion window size, the higher the throughput. The maximum congestion window is related to the amount of buffer space that the kernel allocates for each socket. 
  217: 
  218: So on high bandwidth line the bottleneck could be the buffer sizes. 
  219: 
  220: Here's the formula for a network link's throughput: 
  221:     
  222:     Throughput = buffer size / latency
  223:     
  224: 
  225: So if we reorganise it a bit, we get the formula of the ideal buffer size: 
  226:     
  227:     buffer size = 2 * delay * bandwidth
  228:     
  229: 
  230: The delay is the network latency, which is most commonly known as "ping". 
  231: 
  232: I think I don't have to introduce this tool: 
  233:     
  234:     $ ping yahoo.com
  235:     PING yahoo.com (66.94.234.13): 56 data bytes
  236:     64 bytes from 66.94.234.13: icmp_seq=0 ttl=50 time=195.596 ms
  237:     64 bytes from 66.94.234.13: icmp_seq=1 ttl=50 time=188.883 ms
  238:     64 bytes from 66.94.234.13: icmp_seq=2 ttl=51 time=192.023 ms
  239:     ^C
  240:     ----yahoo.com PING Statistics----
  241:     3 packets transmitted, 3 packets received, 0.0% packet loss
  242:     round-trip min/avg/max/stddev = 188.883/192.167/195.596/3.359 ms
  243:     
  244: 
  245: However ping(1) will give you the round-trip of the network link -- which is the twice of delay -- so the final formula is the following: 
  246:     
  247:     buffer size = RTT * bandwidth
  248:     
  249: 
  250: Fortunately, there is an automatic control for those buffers in NetBSD. It can be checked and and enabled with sysctl: 
  251:     
  252:     net.inet.tcp.recvbuf_auto = 0
  253:     net.inet.tcp.recvbuf_inc = 16384
  254:     net.inet.tcp.recvbuf_max = 262144
  255:     net.inet.tcp.sendbuf_auto = 0
  256:     net.inet.tcp.sendbuf_inc = 8192
  257:     net.inet.tcp.sendbuf_max = 262144
  258:     
  259: 
  260: The default values for the maximal send and receive buffers are set to 256 KBytes, which is very tiny. 
  261: 
  262: A reasonable value for newer systems would be 16 MBytes, so you may set it to that value after you turned it on with sysctl: 
  263:     
  264:     net.inet.tcp.recvbuf_auto=1
  265:     net.inet.tcp.sendbuf_auto=1
  266:     net.inet.tcp.sendbuf_max=16777216 
  267:     net.inet.tcp.recvbuf_max=16777216
  268: 
  269: Just remember that your application has to avoid using SO_RCVBUF or SO_SNDBUF if it wants to take advantage of using automatically sized buffers.
  270: 
  271: ### Increase the initial window size
  272: 
  273: RFC 6928 permits the extension of the initial window size to 10 segments. By default NetBSD uses 4 segments as specified in the RFC 3390.
  274: You can increase it by using the following sysctl's:
  275: 
  276:     net.inet.tcp.init_win=10
  277:     net.inet.tcp.init_win_local=10
  278: 
  279: 
  280: ### IP queue
  281: 
  282: If you are seeing drops due to the limited IP queue (check the net.inet.ip.ifq.drops sysctl), you can increase that by using:
  283: 
  284:     net.inet.ip.ifq.maxlen = 4096
  285: 
  286: ### Other settings
  287: 
  288: If you still are seeing low throughput, maybe it's time for desperate measures ! Try to change the congestion algorithm to cubic using:
  289: 
  290:     net.inet.tcp.congctl.selected=cubic
  291: 
  292: Or try to decrease the limit (expressed in hz ticks) at which the system fires a delayed ACK (for an odd numbered packet). Usually one hz is 10ms but you may want to double check using the kern.clockrate sysctl, and dividing one second to the value there. So, to decrease delack_ticks to 50ms use:
  293: 
  294:     net.inet.tcp.delack_ticks=5
  295: 
  296: 
  297: ##  Disk I/O 
  298: 
  299: You may enable additional buffer queue strategies for better responsiveness under high disk I/O load.  
  300: 
  301: Enable them with the following lines in your kernel configuration file: 
  302:     
  303:       options         BUFQ_READPRIO
  304:       options         BUFQ_PRIOCSCAN
  305:     
  306: 
  307: ##  Using optimized FLAGS with GCC 
  308: 
  309: NOTE: Trying to utilise heavy optimalisations can make your system hard to debug, cause unpredictable behaviour or kill your pet. Especially use of -mtune is highly discouraged, because it does not improve performance considerably or at all compared to -march=i686, and gcc4 can't handle it correctly at least on athlon CPUs. 
  310: 
  311: You can put something like this into your mk.conf, when you compile your packages and your system. 
  312:     
  313:     CPUFLAGS+=-march=i686
  314:     COPTS+=-O2
  315:     
  316: 
  317: FIXME: This is only for building world 
  318:     
  319:     CFLAGS+="-O2 -march=i686"
  320:     
  321: 
  322: FIXME: For packages 
  323: 
  324: For more detailed information about the possible CFLAG values, please read the [GNU C Compiler documentation](http://netbsd.gw.com/cgi-bin/man-cgi?gcc++NetBSD-current). 
  325: 
  326: #  References 
  327: 
  328:   * ["17.4. Managing Kernel Resources". PostgreSQL 8.3 Documentation. PostgreSQL Global Development Group.](http://www.postgresql.org/docs/8.3/static/kernel-resources.html)
  329: 
  330:   * ["Performance Tuning a NetBSD Server". Eric Radman](https://web.archive.org/web/20071202110649/http://eradman.com/article/bsdtuning1)
  331: 
  332:   * ["TCP Tuning Guide". Lawrence Berkeley National Laboratory](http://www-didc.lbl.gov/TCP-tuning/)
  333: 
  334: #  See also 
  335: 
  336:   * [Configuration examples](../../examples/configuration_examples)
  337: 
  338: 

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb