File:  [NetBSD Developer Wiki] / wikisrc / guide / ccd.mdwn
Revision 1.2: download - view: text, annotated - select for diffs
Wed Mar 6 23:38:39 2013 UTC (7 years, 6 months ago) by jdf
Branches: MAIN
CVS tags: HEAD
 * Cosmetics for ccd
 * Moving cgd from the guide to the wiki

    1: # Concatenated Disk Device (CCD) configuration
    2: 
    3: The CCD driver allows the user to *concatenate* several physical disks into one
    4: pseudo volume. While RAIDframe (see [[NetBSD RAIDframe|guide/rf]]) also allows
    5: doing this to create RAID level 0 sets, it does not allow you to do striping
    6: across disks of different geometry, which is where CCD comes in handy. CCD also
    7: allows for an *interleave* to improve disk performance with a gained space loss.
    8: This example will not cover that feature.
    9: 
   10: The steps required to setup a CCD are as follows:
   11: 
   12:  1. Install physical media
   13:  2. Configure kernel support
   14:  3. Disklabel each volume member of the CCD
   15:  4. Configure the CCD conf file
   16:  5. Initialize the CCD device
   17:  6. Create a filesystem on the new CCD device
   18:  7. Mount the CCD filesystem
   19: 
   20: This example features a CCD setup on NetBSD/sparc 1.5. The CCD will reside on 4
   21: SCSI disks in a generic external Sun disk pack chassis connected to the external
   22: 50 pin SCSI port.
   23: 
   24: ## Install physical media
   25: 
   26: This step is at your own discretion, depending on your platform and the hardware
   27: at your disposal.
   28: 
   29: From my DMESG:
   30: 
   31:     Disk #1:
   32:       probe(esp0:0:0): max sync rate 10.00MB/s
   33:       sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
   34:       sd0: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
   35:     
   36:     Disk #2
   37:       probe(esp0:1:0): max sync rate 10.00MB/s
   38:       sd1 at scsibus0 target 1 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
   39:       sd1: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
   40:     
   41:     Disk #3
   42:       probe(esp0:2:0): max sync rate 10.00MB/s
   43:       sd2 at scsibus0 target 2 lun 0: <SEAGATE, ST11200N SUN1.05, 9500> SCSI2 0/direct fixed
   44:       sd2: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors
   45:     
   46:     Disk #4
   47:       probe(esp0:3:0): max sync rate 10.00MB/s
   48:       sd3 at scsibus0 target 3 lun 0: <SEAGATE, ST11200N SUN1.05, 8808 > SCSI2 0
   49:       sd3: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors
   50: 
   51: ## Configure Kernel Support
   52: 
   53: The following kernel configuration directive is needed to provide CCD device
   54: support. It is enabled in the GENERIC kernel:
   55: 
   56:     pseudo-device  ccd  4    # concatenated disk devices
   57: 
   58: In my kernel config, I also hard code SCSI ID associations to `/dev` device
   59: entries to prevent bad things from happening:
   60: 
   61:     sd0     at scsibus0 target 0 lun ?
   62:     # SCSI disk drives
   63:     sd1     at scsibus0 target 1 lun ?
   64:     # SCSI disk drives
   65:     sd2     at scsibus0 target 2 lun ?
   66:     # SCSI disk drives
   67:     sd3     at scsibus0 target 3 lun ?
   68:     # SCSI disk drives
   69:     sd4     at scsibus0 target 4 lun ?
   70:     # SCSI disk drives
   71:     sd5     at scsibus0 target 5 lun ?
   72:     # SCSI disk drives
   73:     sd6     at scsibus0 target 6 lun ?
   74:     # SCSI disk drives
   75: 
   76: ## Disklabel each volume member of the CCD
   77: 
   78: Each member disk of the CCD will need a special file system established. In this
   79: example, I will need to disklabel:
   80: 
   81:  * `/dev/rsd0c`  
   82:  * `/dev/rsd1c`  
   83:  * `/dev/rsd2c`  
   84:  * `/dev/rsd3c`
   85: 
   86: *Note*: Always remember to disklabel the character device, not the block device, 
   87: in `/dev/r{s,w}d*`.
   88: 
   89: *Note*: On all platforms, the `c` slice is symbolic of the entire NetBSD 
   90: partition and is reserved.
   91: 
   92: You will probably want to remove any pre-existing disklabels on the disks in the
   93: CCD. This can be accomplished in one of two ways with the
   94: [dd(1)](http://netbsd.gw.com/cgi-bin/man-cgi?dd+1+NetBSD-5.0.1+i386) command:
   95: 
   96:     # dd if=/dev/zero of=/dev/rsd0c bs=8k count=1
   97:     # dd if=/dev/zero of=/dev/rsd1c bs=8k count=1
   98:     # dd if=/dev/zero of=/dev/rsd2c bs=8k count=1
   99:     # dd if=/dev/zero of=/dev/rsd3c bs=8k count=1
  100: 
  101: If your port uses a MBR (Master Boot Record) to partition the disks so that the
  102: NetBSD partitions are only part of the overall disk, and other OSs like Windows
  103: or Linux use other parts, you can void the MBR and all partitions on disk by
  104: using the command:
  105: 
  106:     # dd if=/dev/zero of=/dev/rsd0d bs=8k count=1
  107:     # dd if=/dev/zero of=/dev/rsd1d bs=8k count=1
  108:     # dd if=/dev/zero of=/dev/rsd2d bs=8k count=1
  109:     # dd if=/dev/zero of=/dev/rsd3d bs=8k count=1
  110: 
  111: This will make all data on the entire disk inaccessible. Note that the entire
  112: disk is slice `d` on i386 (and some other ports), and `c` elsewhere (e.g. on
  113: sparc). See the `kern.rawpartition` sysctl -- `3` means `d`, `2` means `c`.
  114: 
  115: The default disklabel for the disk will look similar to this:
  116: 
  117:     # disklabel -r sd0
  118:     [...snip...]
  119:     bytes/sector: 512
  120:     sectors/track: 116
  121:     tracks/cylinder: 9
  122:     sectors/cylinder: 1044
  123:     cylinders: 3992
  124:     total sectors: 4197405
  125:     [..snip...]
  126:     3 partitions:
  127:     #        size   offset    fstype   [fsize bsize   cpg]
  128:       c:  4197405       0     unused     1024  8192        # (Cyl.    0 - 4020*)
  129: 
  130: You will need to create one *slice* on the NetBSD partition of the disk that
  131: consumes the entire partition. The slice must begin at least one cylinder offset
  132: from the beginning of the disk/partition to provide space for the special CCD
  133: disklabel. The offset should be 1x sectors/cylinder (see following note).
  134: Therefore, the `size` value should be `total sectors` minus 1x
  135: `sectors/cylinder`. Edit your disklabel accordingly:
  136: 
  137:     # disklabel -e sd0
  138: 
  139: *Note*: The offset of a slice of type `ccd` must be a multiple of the
  140: `sectors/cylinder` value.
  141: 
  142: *Note*: Be sure to `export EDITOR=[path to your favorite editor]` before
  143: editing the disklabels.
  144: 
  145: *Note*: The slice must be fstype `ccd`.
  146: 
  147: Because there will only be one slice on this partition, you can recycle the `c`
  148: slice (normally reserved for symbolic uses). Change your disklabel to the
  149: following:
  150: 
  151:     3 partitions:
  152:     #        size   offset    fstype   [fsize bsize   cpg]
  153:       c:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
  154: 
  155: Optionally you can setup a slice other than `c` to use, simply adjust
  156: accordingly below:
  157: 
  158:     3 partitions:
  159:     #        size   offset    fstype   [fsize bsize   cpg]
  160:       a:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
  161:       c:  4197405       0     unused     1024  8192        # (Cyl. 0 - 4020*)
  162: 
  163: Be sure to write the label when you have completed. Disklabel will object to
  164: your disklabel and prompt you to re-edit if it does not pass its sanity checks.
  165: 
  166: ## Configure the CCD
  167: 
  168: Once all disks are properly labeled, you will need to generate a configuration
  169: file, `/etc/ccd.conf`. The file does not exist by default, and you will need to
  170: create a new one. The format is:
  171: 
  172:     #ccd    ileave    flags   component    devices
  173: 
  174: *Note*: For the `ileave`, if a value of zero is used then the disks are
  175: concatenated, but if you use a value equal to the `sectors/track` number the
  176: disks are interleaved.
  177: 
  178: Example in this case:
  179: 
  180:     # more /etc/ccd.conf
  181:     ccd0  0  none /dev/sd0c /dev/sd1c /dev/sd2c /dev/sd3c
  182: 
  183: *Note*: The CCD driver expects block device files as components. Be sure not to
  184: use character device files in the configuration.
  185: 
  186: ## Initialize the CCD device
  187: 
  188: Once you are confident that your CCD configuration is sane, you can initialize
  189: the device using the
  190: [ccdconfig(8)](http://netbsd.gw.com/cgi-bin/man-cgi?ccdconfig+8+NetBSD-5.0.1+i386)
  191: command: Configure:
  192: 
  193:     # ccdconfig -C -f /etc/ccd.conf
  194: 
  195: Unconfigure:
  196: 
  197:     # ccdconfig -u -f /etc/ccd.conf
  198: 
  199: Initializing the CCD device will activate `/dev` entries: `/dev/{,r}ccd#`:
  200: 
  201:     # ls -la  /dev/{,r}ccd0*
  202:     brw-r-----  1 root  operator   9, 0 Apr 28 21:35 /dev/ccd0a
  203:     brw-r-----  1 root  operator   9, 1 Apr 28 21:35 /dev/ccd0b
  204:     brw-r-----  1 root  operator   9, 2 May 12 00:10 /dev/ccd0c
  205:     brw-r-----  1 root  operator   9, 3 Apr 28 21:35 /dev/ccd0d
  206:     brw-r-----  1 root  operator   9, 4 Apr 28 21:35 /dev/ccd0e
  207:     brw-r-----  1 root  operator   9, 5 Apr 28 21:35 /dev/ccd0f
  208:     brw-r-----  1 root  operator   9, 6 Apr 28 21:35 /dev/ccd0g
  209:     brw-r-----  1 root  operator   9, 7 Apr 28 21:35 /dev/ccd0h
  210:     crw-r-----  1 root  operator  23, 0 Jun 12 20:40 /dev/rccd0a
  211:     crw-r-----  1 root  operator  23, 1 Apr 28 21:35 /dev/rccd0b
  212:     crw-r-----  1 root  operator  23, 2 Jun 12 20:58 /dev/rccd0c
  213:     crw-r-----  1 root  operator  23, 3 Apr 28 21:35 /dev/rccd0d
  214:     crw-r-----  1 root  operator  23, 4 Apr 28 21:35 /dev/rccd0e
  215:     crw-r-----  1 root  operator  23, 5 Apr 28 21:35 /dev/rccd0f
  216:     crw-r-----  1 root  operator  23, 6 Apr 28 21:35 /dev/rccd0g
  217:     crw-r-----  1 root  operator  23, 7 Apr 28 21:35 /dev/rccd0h
  218: 
  219: ## Create a 4.2BSD/UFS filesystem on the new CCD device
  220: 
  221: You may now disklabel the new virtual disk device associated with your CCD:
  222: 
  223:     # disklabel -e ccd0
  224: 
  225: Once again, there will be only one slice, so you may either recycle the `c`
  226: slice or create a separate slice for use.
  227: 
  228:     # disklabel -r ccd0
  229:     # /dev/rccd0c:
  230:     type: ccd
  231:     disk: ccd
  232:     label: default label
  233:     flags:
  234:     bytes/sector: 512
  235:     sectors/track: 2048
  236:     tracks/cylinder: 1
  237:     sectors/cylinder: 2048
  238:     cylinders: 6107
  239:     total sectors: 12508812
  240:     rpm: 3600
  241:     interleave: 1
  242:     trackskew: 0
  243:     cylinderskew: 0
  244:     headswitch: 0           # microseconds
  245:     track-to-track seek: 0  # microseconds
  246:     drivedata: 0
  247:     #        size   offset    fstype   [fsize bsize   cpg]
  248:       c: 12508812        0    4.2BSD     1024  8192    16  # (Cyl. 0 - 6107*)
  249: 
  250: The filesystem will then need to be formatted:
  251: 
  252:     # newfs /dev/rccd0c
  253:     Warning: 372 sector(s) in last cylinder unallocated
  254:     /dev/rccd0c:    12508812 sectors in 6108 cylinders of 1 tracks, 2048 sectors
  255:             6107.8MB in 382 cyl groups (16 c/g, 16.00MB/g, 3968 i/g)
  256:     
  257:     super-block backups (for fsck -b #) at:
  258:     [...]
  259: 
  260: ## Mount the filesystem
  261: 
  262: Once you have a created a file system on the CCD device, you can then mount the
  263: file system against a mount point on your system. Be sure to mount the slice
  264: labeled type `ffs` or `4.2BSD`:
  265: 
  266:     # mount /dev/ccd0c /mnt
  267: 
  268: Then:
  269: 
  270:     # export BLOCKSIZE=1024; df
  271:     Filesystem  1K-blocks     Used   Avail Capacity  Mounted on
  272:     /dev/sd6a      376155   320290   37057    89%    /
  273:     /dev/ccd0c    6058800       1  5755859     0%    /mnt
  274: 
  275: Congratulations, you now have a working CCD. To configure the CCD device at boot
  276: time, set `ccd=yes` in `/etc/rc.conf`. You can adjust `/etc/fstab` to get the
  277: filesystem mounted at boot:
  278: 
  279:     /dev/ccd0c  /home ffs     rw    1 2
  280: 

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb