Annotation of wikisrc/guide/ccd.mdwn, revision 1.3

1.1       jdf         1: # Concatenated Disk Device (CCD) configuration
                      2: 
                      3: The CCD driver allows the user to *concatenate* several physical disks into one
                      4: pseudo volume. While RAIDframe (see [[NetBSD RAIDframe|guide/rf]]) also allows
                      5: doing this to create RAID level 0 sets, it does not allow you to do striping
                      6: across disks of different geometry, which is where CCD comes in handy. CCD also
                      7: allows for an *interleave* to improve disk performance with a gained space loss.
                      8: This example will not cover that feature.
                      9: 
                     10: The steps required to setup a CCD are as follows:
                     11: 
                     12:  1. Install physical media
                     13:  2. Configure kernel support
                     14:  3. Disklabel each volume member of the CCD
                     15:  4. Configure the CCD conf file
                     16:  5. Initialize the CCD device
                     17:  6. Create a filesystem on the new CCD device
                     18:  7. Mount the CCD filesystem
                     19: 
                     20: This example features a CCD setup on NetBSD/sparc 1.5. The CCD will reside on 4
                     21: SCSI disks in a generic external Sun disk pack chassis connected to the external
                     22: 50 pin SCSI port.
                     23: 
                     24: ## Install physical media
                     25: 
                     26: This step is at your own discretion, depending on your platform and the hardware
                     27: at your disposal.
                     28: 
                     29: From my DMESG:
                     30: 
                     31:     Disk #1:
                     32:       probe(esp0:0:0): max sync rate 10.00MB/s
                     33:       sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
                     34:       sd0: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
                     35:     
                     36:     Disk #2
                     37:       probe(esp0:1:0): max sync rate 10.00MB/s
                     38:       sd1 at scsibus0 target 1 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
                     39:       sd1: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
                     40:     
                     41:     Disk #3
                     42:       probe(esp0:2:0): max sync rate 10.00MB/s
                     43:       sd2 at scsibus0 target 2 lun 0: <SEAGATE, ST11200N SUN1.05, 9500> SCSI2 0/direct fixed
                     44:       sd2: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors
                     45:     
                     46:     Disk #4
                     47:       probe(esp0:3:0): max sync rate 10.00MB/s
                     48:       sd3 at scsibus0 target 3 lun 0: <SEAGATE, ST11200N SUN1.05, 8808 > SCSI2 0
                     49:       sd3: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors
                     50: 
                     51: ## Configure Kernel Support
                     52: 
                     53: The following kernel configuration directive is needed to provide CCD device
                     54: support. It is enabled in the GENERIC kernel:
                     55: 
                     56:     pseudo-device  ccd  4    # concatenated disk devices
                     57: 
                     58: In my kernel config, I also hard code SCSI ID associations to `/dev` device
                     59: entries to prevent bad things from happening:
                     60: 
                     61:     sd0     at scsibus0 target 0 lun ?
                     62:     # SCSI disk drives
                     63:     sd1     at scsibus0 target 1 lun ?
                     64:     # SCSI disk drives
                     65:     sd2     at scsibus0 target 2 lun ?
                     66:     # SCSI disk drives
                     67:     sd3     at scsibus0 target 3 lun ?
                     68:     # SCSI disk drives
                     69:     sd4     at scsibus0 target 4 lun ?
                     70:     # SCSI disk drives
                     71:     sd5     at scsibus0 target 5 lun ?
                     72:     # SCSI disk drives
                     73:     sd6     at scsibus0 target 6 lun ?
                     74:     # SCSI disk drives
                     75: 
                     76: ## Disklabel each volume member of the CCD
                     77: 
                     78: Each member disk of the CCD will need a special file system established. In this
                     79: example, I will need to disklabel:
                     80: 
1.3     ! jdf        81:  * `/dev/rsd0c`
        !            82:  * `/dev/rsd1c`
        !            83:  * `/dev/rsd2c`
1.1       jdf        84:  * `/dev/rsd3c`
                     85: 
1.3     ! jdf        86: *Note*: Always remember to disklabel the character device, not the block device,
1.2       jdf        87: in `/dev/r{s,w}d*`.
1.1       jdf        88: 
1.3     ! jdf        89: *Note*: On all platforms, the `c` slice is symbolic of the entire NetBSD
1.2       jdf        90: partition and is reserved.
1.1       jdf        91: 
                     92: You will probably want to remove any pre-existing disklabels on the disks in the
                     93: CCD. This can be accomplished in one of two ways with the
                     94: [dd(1)](http://netbsd.gw.com/cgi-bin/man-cgi?dd+1+NetBSD-5.0.1+i386) command:
                     95: 
                     96:     # dd if=/dev/zero of=/dev/rsd0c bs=8k count=1
                     97:     # dd if=/dev/zero of=/dev/rsd1c bs=8k count=1
                     98:     # dd if=/dev/zero of=/dev/rsd2c bs=8k count=1
                     99:     # dd if=/dev/zero of=/dev/rsd3c bs=8k count=1
                    100: 
                    101: If your port uses a MBR (Master Boot Record) to partition the disks so that the
                    102: NetBSD partitions are only part of the overall disk, and other OSs like Windows
                    103: or Linux use other parts, you can void the MBR and all partitions on disk by
                    104: using the command:
                    105: 
                    106:     # dd if=/dev/zero of=/dev/rsd0d bs=8k count=1
                    107:     # dd if=/dev/zero of=/dev/rsd1d bs=8k count=1
                    108:     # dd if=/dev/zero of=/dev/rsd2d bs=8k count=1
                    109:     # dd if=/dev/zero of=/dev/rsd3d bs=8k count=1
                    110: 
                    111: This will make all data on the entire disk inaccessible. Note that the entire
                    112: disk is slice `d` on i386 (and some other ports), and `c` elsewhere (e.g. on
                    113: sparc). See the `kern.rawpartition` sysctl -- `3` means `d`, `2` means `c`.
                    114: 
                    115: The default disklabel for the disk will look similar to this:
                    116: 
                    117:     # disklabel -r sd0
                    118:     [...snip...]
                    119:     bytes/sector: 512
                    120:     sectors/track: 116
                    121:     tracks/cylinder: 9
                    122:     sectors/cylinder: 1044
                    123:     cylinders: 3992
                    124:     total sectors: 4197405
                    125:     [..snip...]
                    126:     3 partitions:
                    127:     #        size   offset    fstype   [fsize bsize   cpg]
                    128:       c:  4197405       0     unused     1024  8192        # (Cyl.    0 - 4020*)
                    129: 
                    130: You will need to create one *slice* on the NetBSD partition of the disk that
                    131: consumes the entire partition. The slice must begin at least one cylinder offset
                    132: from the beginning of the disk/partition to provide space for the special CCD
                    133: disklabel. The offset should be 1x sectors/cylinder (see following note).
                    134: Therefore, the `size` value should be `total sectors` minus 1x
                    135: `sectors/cylinder`. Edit your disklabel accordingly:
                    136: 
                    137:     # disklabel -e sd0
                    138: 
                    139: *Note*: The offset of a slice of type `ccd` must be a multiple of the
                    140: `sectors/cylinder` value.
                    141: 
                    142: *Note*: Be sure to `export EDITOR=[path to your favorite editor]` before
                    143: editing the disklabels.
                    144: 
                    145: *Note*: The slice must be fstype `ccd`.
                    146: 
                    147: Because there will only be one slice on this partition, you can recycle the `c`
                    148: slice (normally reserved for symbolic uses). Change your disklabel to the
                    149: following:
                    150: 
                    151:     3 partitions:
                    152:     #        size   offset    fstype   [fsize bsize   cpg]
                    153:       c:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
                    154: 
                    155: Optionally you can setup a slice other than `c` to use, simply adjust
                    156: accordingly below:
                    157: 
                    158:     3 partitions:
                    159:     #        size   offset    fstype   [fsize bsize   cpg]
                    160:       a:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
                    161:       c:  4197405       0     unused     1024  8192        # (Cyl. 0 - 4020*)
                    162: 
                    163: Be sure to write the label when you have completed. Disklabel will object to
                    164: your disklabel and prompt you to re-edit if it does not pass its sanity checks.
                    165: 
                    166: ## Configure the CCD
                    167: 
                    168: Once all disks are properly labeled, you will need to generate a configuration
                    169: file, `/etc/ccd.conf`. The file does not exist by default, and you will need to
                    170: create a new one. The format is:
                    171: 
                    172:     #ccd    ileave    flags   component    devices
                    173: 
                    174: *Note*: For the `ileave`, if a value of zero is used then the disks are
                    175: concatenated, but if you use a value equal to the `sectors/track` number the
                    176: disks are interleaved.
                    177: 
                    178: Example in this case:
                    179: 
                    180:     # more /etc/ccd.conf
                    181:     ccd0  0  none /dev/sd0c /dev/sd1c /dev/sd2c /dev/sd3c
                    182: 
                    183: *Note*: The CCD driver expects block device files as components. Be sure not to
                    184: use character device files in the configuration.
                    185: 
                    186: ## Initialize the CCD device
                    187: 
                    188: Once you are confident that your CCD configuration is sane, you can initialize
                    189: the device using the
                    190: [ccdconfig(8)](http://netbsd.gw.com/cgi-bin/man-cgi?ccdconfig+8+NetBSD-5.0.1+i386)
                    191: command: Configure:
                    192: 
                    193:     # ccdconfig -C -f /etc/ccd.conf
                    194: 
                    195: Unconfigure:
                    196: 
                    197:     # ccdconfig -u -f /etc/ccd.conf
                    198: 
                    199: Initializing the CCD device will activate `/dev` entries: `/dev/{,r}ccd#`:
                    200: 
                    201:     # ls -la  /dev/{,r}ccd0*
                    202:     brw-r-----  1 root  operator   9, 0 Apr 28 21:35 /dev/ccd0a
                    203:     brw-r-----  1 root  operator   9, 1 Apr 28 21:35 /dev/ccd0b
                    204:     brw-r-----  1 root  operator   9, 2 May 12 00:10 /dev/ccd0c
                    205:     brw-r-----  1 root  operator   9, 3 Apr 28 21:35 /dev/ccd0d
                    206:     brw-r-----  1 root  operator   9, 4 Apr 28 21:35 /dev/ccd0e
                    207:     brw-r-----  1 root  operator   9, 5 Apr 28 21:35 /dev/ccd0f
                    208:     brw-r-----  1 root  operator   9, 6 Apr 28 21:35 /dev/ccd0g
                    209:     brw-r-----  1 root  operator   9, 7 Apr 28 21:35 /dev/ccd0h
                    210:     crw-r-----  1 root  operator  23, 0 Jun 12 20:40 /dev/rccd0a
                    211:     crw-r-----  1 root  operator  23, 1 Apr 28 21:35 /dev/rccd0b
                    212:     crw-r-----  1 root  operator  23, 2 Jun 12 20:58 /dev/rccd0c
                    213:     crw-r-----  1 root  operator  23, 3 Apr 28 21:35 /dev/rccd0d
                    214:     crw-r-----  1 root  operator  23, 4 Apr 28 21:35 /dev/rccd0e
                    215:     crw-r-----  1 root  operator  23, 5 Apr 28 21:35 /dev/rccd0f
                    216:     crw-r-----  1 root  operator  23, 6 Apr 28 21:35 /dev/rccd0g
                    217:     crw-r-----  1 root  operator  23, 7 Apr 28 21:35 /dev/rccd0h
                    218: 
                    219: ## Create a 4.2BSD/UFS filesystem on the new CCD device
                    220: 
                    221: You may now disklabel the new virtual disk device associated with your CCD:
                    222: 
                    223:     # disklabel -e ccd0
                    224: 
                    225: Once again, there will be only one slice, so you may either recycle the `c`
                    226: slice or create a separate slice for use.
                    227: 
                    228:     # disklabel -r ccd0
                    229:     # /dev/rccd0c:
                    230:     type: ccd
                    231:     disk: ccd
                    232:     label: default label
                    233:     flags:
                    234:     bytes/sector: 512
                    235:     sectors/track: 2048
                    236:     tracks/cylinder: 1
                    237:     sectors/cylinder: 2048
                    238:     cylinders: 6107
                    239:     total sectors: 12508812
                    240:     rpm: 3600
                    241:     interleave: 1
                    242:     trackskew: 0
                    243:     cylinderskew: 0
                    244:     headswitch: 0           # microseconds
                    245:     track-to-track seek: 0  # microseconds
                    246:     drivedata: 0
                    247:     #        size   offset    fstype   [fsize bsize   cpg]
                    248:       c: 12508812        0    4.2BSD     1024  8192    16  # (Cyl. 0 - 6107*)
                    249: 
                    250: The filesystem will then need to be formatted:
                    251: 
                    252:     # newfs /dev/rccd0c
                    253:     Warning: 372 sector(s) in last cylinder unallocated
                    254:     /dev/rccd0c:    12508812 sectors in 6108 cylinders of 1 tracks, 2048 sectors
                    255:             6107.8MB in 382 cyl groups (16 c/g, 16.00MB/g, 3968 i/g)
                    256:     
                    257:     super-block backups (for fsck -b #) at:
                    258:     [...]
                    259: 
                    260: ## Mount the filesystem
                    261: 
                    262: Once you have a created a file system on the CCD device, you can then mount the
                    263: file system against a mount point on your system. Be sure to mount the slice
                    264: labeled type `ffs` or `4.2BSD`:
                    265: 
                    266:     # mount /dev/ccd0c /mnt
                    267: 
                    268: Then:
                    269: 
                    270:     # export BLOCKSIZE=1024; df
                    271:     Filesystem  1K-blocks     Used   Avail Capacity  Mounted on
                    272:     /dev/sd6a      376155   320290   37057    89%    /
                    273:     /dev/ccd0c    6058800       1  5755859     0%    /mnt
                    274: 
                    275: Congratulations, you now have a working CCD. To configure the CCD device at boot
                    276: time, set `ccd=yes` in `/etc/rc.conf`. You can adjust `/etc/fstab` to get the
                    277: filesystem mounted at boot:
                    278: 
                    279:     /dev/ccd0c  /home ffs     rw    1 2
                    280: 

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb