Annotation of wikisrc/tutorials/tuning_netbsd_for_performance.mdwn, revision 1.1

1.1     ! mspo        1: **Contents**
        !             2: 
        !             3: [[!toc levels=3]]
        !             4: 
        !             5: #  Tuning the kernel 
        !             6: 
        !             7: ##  Process and file descriptor limits 
        !             8: 
        !             9: **Before reading:**
        !            10: 
        !            11: _These are mostly only demonstative values on how to tune your system for different needs. They are not some kind of an ultimate optional values. This article mostly aims to provide a quick overview on the ways to fine tune your system settings and being aware of the limitations._
        !            12: 
        !            13: ###  maxusers 
        !            14: 
        !            15: The name is a bit misleading, because it doesn't set the number of users on the system, but used in the formula to calculate maximal number of allowed processes. 
        !            16: 
        !            17: You can find it in your kernel configuration file, something like this: 
        !            18:     
        !            19:     maxusers        32
        !            20:     
        !            21: 
        !            22: This is the default value, so if we look at the formulae we get process limit values: 
        !            23:     
        !            24:     /usr/src/sys/param.h:
        !            25:        #define  NPROC    (20 + 16 * MAXUSERS)
        !            26:     
        !            27:     /usr/src/sys/conf/param.c:
        !            28:        #define  MAXFILES (3 * (NPROC + MAXUSERS) + 80)
        !            29:     
        !            30: 
        !            31: So we got 532 for NPROC (maximal number of processes) and 1772 for MAXFILES (maximal number of open file descriptors). 
        !            32: 
        !            33: Some say that the maxusers should be set to the amount of RAM in megabytes.  
        !            34: For reference, FreeBSD sets is automaticaly by this formula, but limits it's maximum to 384. 
        !            35: 
        !            36: Setting it to 64 is always a safe bet if you don't want too much experimenting. Just change it in your kernel configuration file: 
        !            37:     
        !            38:     maxusers        64
        !            39:     
        !            40: 
        !            41: Compile the new kernel with build.sh or manualy, install the new kernel and reboot. 
        !            42: 
        !            43: You can check your limits with sysctl: 
        !            44: 
        !            45: With maxusers 32 
        !            46:     
        !            47:     $ sysctl proc.curproc.rlimit.maxproc
        !            48:     proc.curproc.rlimit.maxproc.soft = 160
        !            49:     proc.curproc.rlimit.maxproc.hard = 532
        !            50:     
        !            51:     
        !            52:     $ sysctl proc.curproc.rlimit.descriptors
        !            53:     proc.curproc.rlimit.descriptors.soft = 64
        !            54:     proc.curproc.rlimit.descriptors.hard = 1772
        !            55:     
        !            56: 
        !            57: With maxusers 64 
        !            58: 
        !            59: You can check your limits with sysctl: 
        !            60:     
        !            61:     $ sysctl proc.curproc.rlimit.maxproc
        !            62:     proc.curproc.rlimit.maxproc.soft = 160
        !            63:     proc.curproc.rlimit.maxproc.hard = 1044
        !            64:     
        !            65:     
        !            66:     $ sysctl proc.curproc.rlimit.descriptors
        !            67:     proc.curproc.rlimit.descriptors.soft = 64
        !            68:     proc.curproc.rlimit.descriptors.hard = 3404
        !            69:     
        !            70: 
        !            71: ###  login.conf 
        !            72: 
        !            73: So you can change the hard limits now. Let's see the soft limits. 
        !            74: 
        !            75: or with ulimit: 
        !            76:     
        !            77:     $ ulimit -a
        !            78:     core file size          (blocks, -c) unlimited
        !            79:     data seg size           (kbytes, -d) 131072
        !            80:     file size               (blocks, -f) unlimited
        !            81:     max locked memory       (kbytes, -l) 80920
        !            82:     max memory size         (kbytes, -m) 242760
        !            83:     open files                      (-n) 64
        !            84:     pipe size            (512 bytes, -p) 1
        !            85:     stack size              (kbytes, -s) 2048
        !            86:     cpu time               (seconds, -t) unlimited
        !            87:     max user processes              (-u) 160
        !            88:     virtual memory          (kbytes, -v) 133120
        !            89:     
        !            90: 
        !            91:   
        !            92: You can set it with the file /etc/login.conf: 
        !            93:     
        !            94:     default:\
        !            95:         :path=/usr/bin /bin /usr/sbin /sbin /usr/X11R6/bin /usr/pkg/bin /usr/pkg/sbin /usr/local/bin:\
        !            96:         :umask=022:\
        !            97:         :datasize-max=3072M:\
        !            98:         :datasize-cur=1024M:\
        !            99:         :maxproc-max=1044:\
        !           100:         :maxproc-cur=512:\
        !           101:         :openfiles-cur=256:\
        !           102:         :stacksize-cur=8M:
        !           103:     
        !           104: 
        !           105: Next time you start the sytem, all users belonging to the _default_ login group will have the following limits: 
        !           106:     
        !           107:     $ ulimit -a
        !           108:     coredump(blocks)     unlimited
        !           109:     data(KiB)            1048576
        !           110:     file(blocks)         unlimited
        !           111:     lockedmem(KiB)       124528
        !           112:     memory(KiB)          373584
        !           113:     nofiles(descriptors) 256
        !           114:     processes            512
        !           115:     stack(KiB)           8192
        !           116:     time(cpu-seconds)    unlimited
        !           117:     
        !           118: 
        !           119: You may set different limits for different user, thus different services: 
        !           120:     
        !           121:     database:\
        !           122:        :ignorenologin:\
        !           123:        :datasize=infinity:\
        !           124:        :maxproc=infinity:\
        !           125:        :openfiles-cur=1024:\
        !           126:        :stacksize-cur=48M:
        !           127:     
        !           128: 
        !           129: You should run this command after editing your login.conf: 
        !           130:     
        !           131:     $ cap_mkdb /etc/login.conf
        !           132:     
        !           133: 
        !           134: You can assign the newly created login class to the desired user by doing something like this: 
        !           135:     
        !           136:     $ usermod -L database pgsql
        !           137:     
        !           138: 
        !           139: Let's check our limits again with sysctl: 
        !           140:     
        !           141:     $ sysctl proc.curproc.rlimit.maxproc
        !           142:     proc.curproc.rlimit.maxproc.soft = 512
        !           143:     proc.curproc.rlimit.maxproc.hard = 1044
        !           144:     
        !           145:     $ sysctl proc.curproc.rlimit.descriptors
        !           146:     proc.curproc.rlimit.descriptors.soft = 256
        !           147:     proc.curproc.rlimit.descriptors.hard = 3404
        !           148:     
        !           149: 
        !           150: Much reasonable for a modern system. 
        !           151: 
        !           152: ##  System V interprocess communication 
        !           153: 
        !           154: Shared memory and semaphores are part of the System V IPC. Using and fine tuning shared memory and semaphores can give you increased performance on your NetBSD server. 
        !           155: 
        !           156: You can check it's settings with sysctl: 
        !           157:     
        !           158:     $ sysctl kern.ipc
        !           159:     kern.ipc.sysvmsg = 1
        !           160:     kern.ipc.sysvsem = 1
        !           161:     kern.ipc.sysvshm = 1
        !           162:     kern.ipc.shmmax = 8388608
        !           163:     kern.ipc.shmmni = 128
        !           164:     kern.ipc.shmseg = 128
        !           165:     kern.ipc.shmmaxpgs = 2048
        !           166:     kern.ipc.shm_use_phys = 0
        !           167:     kern.ipc.msgmni = 40
        !           168:     kern.ipc.msgseg = 2048
        !           169:     kern.ipc.semmni = 10
        !           170:     kern.ipc.semmns = 60
        !           171:     kern.ipc.semmnu = 30
        !           172:     
        !           173: 
        !           174: As you can see, the default maximum size of shared memory segment (shmmax) is 8 megabytes by default, but for a postgresql server you will most likely need about 128 megabytes. 
        !           175: 
        !           176: Note, that you cannot set shmmax directly with syctl, but you need to set the value in pages size with _kern.ipc.shmmaxpgs_. 
        !           177: 
        !           178: The default PAGE_SIZE is 4096, so if you want to set it to 128M, you have to do: 
        !           179:     
        !           180:     grimnismal# sysctl -w kern.ipc.shmmaxpgs=32768
        !           181:     kern.ipc.shmmaxpgs: 4096 -> 32768
        !           182:     
        !           183: 
        !           184: So the formula is: 128*1024*1024/4096 = 32768 
        !           185: 
        !           186: **You can make any sysctl change permanent by setting it in /etc/sysctl.conf**
        !           187: 
        !           188: You can also get detailed information on System V interprocess communication (IPC) facilities on the system with the following command: 
        !           189:     
        !           190:     
        !           191:     $ ipcs
        !           192:     IPC status from <running system> as of Mon Dec  3 18:52:00 2007
        !           193:     
        !           194:     Message Queues:
        !           195:     T        ID     KEY        MODE       OWNER    GROUP
        !           196:     
        !           197:     Shared Memory:
        !           198:     T        ID     KEY        MODE       OWNER    GROUP
        !           199:     m     65536    5432001 --rw-------    pgsql    pgsql
        !           200:     
        !           201:     Semaphores:
        !           202:     T        ID     KEY        MODE       OWNER    GROUP
        !           203:     s     65536    5432001 --rw-------    pgsql    pgsql
        !           204:     s     65537    5432002 --rw-------    pgsql    pgsql
        !           205:     s     65538    5432003 --rw-------    pgsql    pgsql
        !           206:     
        !           207: 
        !           208:   
        !           209: You can also force shared memory to stay in physical memory. This means that they will be never paged out to swap.  
        !           210: You may set this behaviour with the _kern.ipc.shm_use_phys_ sysctl. 
        !           211: 
        !           212: ##  TCP Performance 
        !           213: 
        !           214: TCP uses what is called the “congestion window” to determine how many packets can be sent at one time. The larger the congestion window size, the higher the throughput. The maximum congestion window is related to the amount of buffer space that the kernel allocates for each socket. 
        !           215: 
        !           216: So on high bandwidth line the bottleneck could be the buffer sizes. 
        !           217: 
        !           218: Here's the formula for a network link's throughput: 
        !           219:     
        !           220:     Throughput = buffer size / latency
        !           221:     
        !           222: 
        !           223: So if we reorganise it a bit, we get the formula of the ideal buffer size: 
        !           224:     
        !           225:     buffer size = 2 * delay * bandwidth
        !           226:     
        !           227: 
        !           228: The delay is the network latency, which is most commonly known as "ping". 
        !           229: 
        !           230: I think I don't have to introduce this tool: 
        !           231:     
        !           232:     $ ping yahoo.com
        !           233:     PING yahoo.com (66.94.234.13): 56 data bytes
        !           234:     64 bytes from 66.94.234.13: icmp_seq=0 ttl=50 time=195.596 ms
        !           235:     64 bytes from 66.94.234.13: icmp_seq=1 ttl=50 time=188.883 ms
        !           236:     64 bytes from 66.94.234.13: icmp_seq=2 ttl=51 time=192.023 ms
        !           237:     ^C
        !           238:     ----yahoo.com PING Statistics----
        !           239:     3 packets transmitted, 3 packets received, 0.0% packet loss
        !           240:     round-trip min/avg/max/stddev = 188.883/192.167/195.596/3.359 ms
        !           241:     
        !           242: 
        !           243: However ping(1) will give you the round-trip of the network link -- which is the twice of delay -- so the final formula is the following: 
        !           244:     
        !           245:     buffer size = RTT * bandwidth
        !           246:     
        !           247: 
        !           248: Fortunately, there is an automatic control for those buffers in NetBSD. It can be checked and and enabled with sysctl: 
        !           249:     
        !           250:     net.inet.tcp.recvbuf_auto = 0
        !           251:     net.inet.tcp.recvbuf_inc = 16384
        !           252:     net.inet.tcp.recvbuf_max = 262144
        !           253:     net.inet.tcp.sendbuf_auto = 0
        !           254:     net.inet.tcp.sendbuf_inc = 8192
        !           255:     net.inet.tcp.sendbuf_max = 262144
        !           256:     
        !           257: 
        !           258: The automatic setting for sendbuf and recvbuf is disabled in the default installation.  
        !           259: The initial value for maximal send buffer and receive buffer is both 256k, which is very tiny. 
        !           260: 
        !           261: A reasonable value is 16 megabytes, so you may set it to that value after you turned it on with sysctl: 
        !           262:     
        !           263:     net.inet.tcp.recvbuf_auto=1
        !           264:     net.inet.tcp.sendbuf_auto=1
        !           265:     net.inet.tcp.sendbuf_max=16777216 
        !           266:     net.inet.tcp.recvbuf_max=16777216
        !           267:     
        !           268: 
        !           269: ##  Disk I/O 
        !           270: 
        !           271: You may enable **experimental** buffer queue strategy for better responsiveness under high disk I/O load.  
        !           272: This options is _likely_ to stable but not yet the default. 
        !           273: 
        !           274: Enable them with the following lines in your kernel configuration file: 
        !           275:     
        !           276:       options         BUFQ_READPRIO
        !           277:       options         BUFQ_PRIOCSCAN
        !           278:     
        !           279: 
        !           280: ##  Using optimized FLAGS with GCC 
        !           281: 
        !           282: NOTE: Trying to utilise heavy optimalisations can make your system hard to debug, cause unpredictable behaviour or kill your pet. Especially use of -mtune is highly discouraged, because it does not improve performance considerably or at all compared to -march=i686, and gcc4 can't handle it correctly at least on athlon CPUs. 
        !           283: 
        !           284: You can put something like this into your mk.conf, when you compile your packages and your system. 
        !           285:     
        !           286:     CPUFLAGS+=-march=i686
        !           287:     COPTS+=-O2
        !           288:     
        !           289: 
        !           290: FIXME: This is only for building world 
        !           291:     
        !           292:     CFLAGS+="-O2 -march=i686"
        !           293:     
        !           294: 
        !           295: FIXME: For packages 
        !           296: 
        !           297: For more detailed information about the possible CFLAG values, please read the [GNU C Compiler documentation](http://netbsd.gw.com/cgi-bin/man-cgi?gcc++NetBSD-current). 
        !           298: 
        !           299: #  References 
        !           300: 
        !           301:   * "17.4. Managing Kernel Resources". PostgreSQL 8.3beta3 Documentation. PostgreSQL Global Development Group. December 2007. Retrieved December 2, 2007. [[1]](http://www.postgresql.org/docs/8.3/static/kernel-resources.html)
        !           302: 
        !           303:   * "Performance Tuning a NetBSD Server". Eric Radman. Retrieved December 3, 2007. [[2]](http://eradman.com/article/bsdtuning1)
        !           304: 
        !           305:   * "TCP Tuning Guide". Lawrence Berkeley National Laboratory. Nov 15, 2007. Retrieved December 4, 2007. [[3]](http://www-didc.lbl.gov/TCP-tuning/)
        !           306: 
        !           307: #  See also 
        !           308: 
        !           309:   * [[Configuration examples]]
        !           310: 
        !           311: 

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb