Annotation of wikisrc/guide/ccd.mdwn, revision 1.4

1.4     ! jdf         1: **Contents**
        !             2: 
        !             3: [[!toc levels=3]]
        !             4: 
1.1       jdf         5: # Concatenated Disk Device (CCD) configuration
                      6: 
                      7: The CCD driver allows the user to *concatenate* several physical disks into one
                      8: pseudo volume. While RAIDframe (see [[NetBSD RAIDframe|guide/rf]]) also allows
                      9: doing this to create RAID level 0 sets, it does not allow you to do striping
                     10: across disks of different geometry, which is where CCD comes in handy. CCD also
                     11: allows for an *interleave* to improve disk performance with a gained space loss.
                     12: This example will not cover that feature.
                     13: 
                     14: The steps required to setup a CCD are as follows:
                     15: 
                     16:  1. Install physical media
                     17:  2. Configure kernel support
                     18:  3. Disklabel each volume member of the CCD
                     19:  4. Configure the CCD conf file
                     20:  5. Initialize the CCD device
                     21:  6. Create a filesystem on the new CCD device
                     22:  7. Mount the CCD filesystem
                     23: 
                     24: This example features a CCD setup on NetBSD/sparc 1.5. The CCD will reside on 4
                     25: SCSI disks in a generic external Sun disk pack chassis connected to the external
                     26: 50 pin SCSI port.
                     27: 
                     28: ## Install physical media
                     29: 
                     30: This step is at your own discretion, depending on your platform and the hardware
                     31: at your disposal.
                     32: 
                     33: From my DMESG:
                     34: 
                     35:     Disk #1:
                     36:       probe(esp0:0:0): max sync rate 10.00MB/s
                     37:       sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
                     38:       sd0: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
                     39:     
                     40:     Disk #2
                     41:       probe(esp0:1:0): max sync rate 10.00MB/s
                     42:       sd1 at scsibus0 target 1 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
                     43:       sd1: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
                     44:     
                     45:     Disk #3
                     46:       probe(esp0:2:0): max sync rate 10.00MB/s
                     47:       sd2 at scsibus0 target 2 lun 0: <SEAGATE, ST11200N SUN1.05, 9500> SCSI2 0/direct fixed
                     48:       sd2: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors
                     49:     
                     50:     Disk #4
                     51:       probe(esp0:3:0): max sync rate 10.00MB/s
                     52:       sd3 at scsibus0 target 3 lun 0: <SEAGATE, ST11200N SUN1.05, 8808 > SCSI2 0
                     53:       sd3: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors
                     54: 
                     55: ## Configure Kernel Support
                     56: 
                     57: The following kernel configuration directive is needed to provide CCD device
                     58: support. It is enabled in the GENERIC kernel:
                     59: 
                     60:     pseudo-device  ccd  4    # concatenated disk devices
                     61: 
                     62: In my kernel config, I also hard code SCSI ID associations to `/dev` device
                     63: entries to prevent bad things from happening:
                     64: 
                     65:     sd0     at scsibus0 target 0 lun ?
                     66:     # SCSI disk drives
                     67:     sd1     at scsibus0 target 1 lun ?
                     68:     # SCSI disk drives
                     69:     sd2     at scsibus0 target 2 lun ?
                     70:     # SCSI disk drives
                     71:     sd3     at scsibus0 target 3 lun ?
                     72:     # SCSI disk drives
                     73:     sd4     at scsibus0 target 4 lun ?
                     74:     # SCSI disk drives
                     75:     sd5     at scsibus0 target 5 lun ?
                     76:     # SCSI disk drives
                     77:     sd6     at scsibus0 target 6 lun ?
                     78:     # SCSI disk drives
                     79: 
                     80: ## Disklabel each volume member of the CCD
                     81: 
                     82: Each member disk of the CCD will need a special file system established. In this
                     83: example, I will need to disklabel:
                     84: 
1.3       jdf        85:  * `/dev/rsd0c`
                     86:  * `/dev/rsd1c`
                     87:  * `/dev/rsd2c`
1.1       jdf        88:  * `/dev/rsd3c`
                     89: 
1.3       jdf        90: *Note*: Always remember to disklabel the character device, not the block device,
1.2       jdf        91: in `/dev/r{s,w}d*`.
1.1       jdf        92: 
1.3       jdf        93: *Note*: On all platforms, the `c` slice is symbolic of the entire NetBSD
1.2       jdf        94: partition and is reserved.
1.1       jdf        95: 
                     96: You will probably want to remove any pre-existing disklabels on the disks in the
                     97: CCD. This can be accomplished in one of two ways with the
                     98: [dd(1)](http://netbsd.gw.com/cgi-bin/man-cgi?dd+1+NetBSD-5.0.1+i386) command:
                     99: 
                    100:     # dd if=/dev/zero of=/dev/rsd0c bs=8k count=1
                    101:     # dd if=/dev/zero of=/dev/rsd1c bs=8k count=1
                    102:     # dd if=/dev/zero of=/dev/rsd2c bs=8k count=1
                    103:     # dd if=/dev/zero of=/dev/rsd3c bs=8k count=1
                    104: 
                    105: If your port uses a MBR (Master Boot Record) to partition the disks so that the
                    106: NetBSD partitions are only part of the overall disk, and other OSs like Windows
                    107: or Linux use other parts, you can void the MBR and all partitions on disk by
                    108: using the command:
                    109: 
                    110:     # dd if=/dev/zero of=/dev/rsd0d bs=8k count=1
                    111:     # dd if=/dev/zero of=/dev/rsd1d bs=8k count=1
                    112:     # dd if=/dev/zero of=/dev/rsd2d bs=8k count=1
                    113:     # dd if=/dev/zero of=/dev/rsd3d bs=8k count=1
                    114: 
                    115: This will make all data on the entire disk inaccessible. Note that the entire
                    116: disk is slice `d` on i386 (and some other ports), and `c` elsewhere (e.g. on
                    117: sparc). See the `kern.rawpartition` sysctl -- `3` means `d`, `2` means `c`.
                    118: 
                    119: The default disklabel for the disk will look similar to this:
                    120: 
                    121:     # disklabel -r sd0
                    122:     [...snip...]
                    123:     bytes/sector: 512
                    124:     sectors/track: 116
                    125:     tracks/cylinder: 9
                    126:     sectors/cylinder: 1044
                    127:     cylinders: 3992
                    128:     total sectors: 4197405
                    129:     [..snip...]
                    130:     3 partitions:
                    131:     #        size   offset    fstype   [fsize bsize   cpg]
                    132:       c:  4197405       0     unused     1024  8192        # (Cyl.    0 - 4020*)
                    133: 
                    134: You will need to create one *slice* on the NetBSD partition of the disk that
                    135: consumes the entire partition. The slice must begin at least one cylinder offset
                    136: from the beginning of the disk/partition to provide space for the special CCD
                    137: disklabel. The offset should be 1x sectors/cylinder (see following note).
                    138: Therefore, the `size` value should be `total sectors` minus 1x
                    139: `sectors/cylinder`. Edit your disklabel accordingly:
                    140: 
                    141:     # disklabel -e sd0
                    142: 
                    143: *Note*: The offset of a slice of type `ccd` must be a multiple of the
                    144: `sectors/cylinder` value.
                    145: 
                    146: *Note*: Be sure to `export EDITOR=[path to your favorite editor]` before
                    147: editing the disklabels.
                    148: 
                    149: *Note*: The slice must be fstype `ccd`.
                    150: 
                    151: Because there will only be one slice on this partition, you can recycle the `c`
                    152: slice (normally reserved for symbolic uses). Change your disklabel to the
                    153: following:
                    154: 
                    155:     3 partitions:
                    156:     #        size   offset    fstype   [fsize bsize   cpg]
                    157:       c:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
                    158: 
                    159: Optionally you can setup a slice other than `c` to use, simply adjust
                    160: accordingly below:
                    161: 
                    162:     3 partitions:
                    163:     #        size   offset    fstype   [fsize bsize   cpg]
                    164:       a:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
                    165:       c:  4197405       0     unused     1024  8192        # (Cyl. 0 - 4020*)
                    166: 
                    167: Be sure to write the label when you have completed. Disklabel will object to
                    168: your disklabel and prompt you to re-edit if it does not pass its sanity checks.
                    169: 
                    170: ## Configure the CCD
                    171: 
                    172: Once all disks are properly labeled, you will need to generate a configuration
                    173: file, `/etc/ccd.conf`. The file does not exist by default, and you will need to
                    174: create a new one. The format is:
                    175: 
                    176:     #ccd    ileave    flags   component    devices
                    177: 
                    178: *Note*: For the `ileave`, if a value of zero is used then the disks are
                    179: concatenated, but if you use a value equal to the `sectors/track` number the
                    180: disks are interleaved.
                    181: 
                    182: Example in this case:
                    183: 
                    184:     # more /etc/ccd.conf
                    185:     ccd0  0  none /dev/sd0c /dev/sd1c /dev/sd2c /dev/sd3c
                    186: 
                    187: *Note*: The CCD driver expects block device files as components. Be sure not to
                    188: use character device files in the configuration.
                    189: 
                    190: ## Initialize the CCD device
                    191: 
                    192: Once you are confident that your CCD configuration is sane, you can initialize
                    193: the device using the
                    194: [ccdconfig(8)](http://netbsd.gw.com/cgi-bin/man-cgi?ccdconfig+8+NetBSD-5.0.1+i386)
                    195: command: Configure:
                    196: 
                    197:     # ccdconfig -C -f /etc/ccd.conf
                    198: 
                    199: Unconfigure:
                    200: 
                    201:     # ccdconfig -u -f /etc/ccd.conf
                    202: 
                    203: Initializing the CCD device will activate `/dev` entries: `/dev/{,r}ccd#`:
                    204: 
                    205:     # ls -la  /dev/{,r}ccd0*
                    206:     brw-r-----  1 root  operator   9, 0 Apr 28 21:35 /dev/ccd0a
                    207:     brw-r-----  1 root  operator   9, 1 Apr 28 21:35 /dev/ccd0b
                    208:     brw-r-----  1 root  operator   9, 2 May 12 00:10 /dev/ccd0c
                    209:     brw-r-----  1 root  operator   9, 3 Apr 28 21:35 /dev/ccd0d
                    210:     brw-r-----  1 root  operator   9, 4 Apr 28 21:35 /dev/ccd0e
                    211:     brw-r-----  1 root  operator   9, 5 Apr 28 21:35 /dev/ccd0f
                    212:     brw-r-----  1 root  operator   9, 6 Apr 28 21:35 /dev/ccd0g
                    213:     brw-r-----  1 root  operator   9, 7 Apr 28 21:35 /dev/ccd0h
                    214:     crw-r-----  1 root  operator  23, 0 Jun 12 20:40 /dev/rccd0a
                    215:     crw-r-----  1 root  operator  23, 1 Apr 28 21:35 /dev/rccd0b
                    216:     crw-r-----  1 root  operator  23, 2 Jun 12 20:58 /dev/rccd0c
                    217:     crw-r-----  1 root  operator  23, 3 Apr 28 21:35 /dev/rccd0d
                    218:     crw-r-----  1 root  operator  23, 4 Apr 28 21:35 /dev/rccd0e
                    219:     crw-r-----  1 root  operator  23, 5 Apr 28 21:35 /dev/rccd0f
                    220:     crw-r-----  1 root  operator  23, 6 Apr 28 21:35 /dev/rccd0g
                    221:     crw-r-----  1 root  operator  23, 7 Apr 28 21:35 /dev/rccd0h
                    222: 
                    223: ## Create a 4.2BSD/UFS filesystem on the new CCD device
                    224: 
                    225: You may now disklabel the new virtual disk device associated with your CCD:
                    226: 
                    227:     # disklabel -e ccd0
                    228: 
                    229: Once again, there will be only one slice, so you may either recycle the `c`
                    230: slice or create a separate slice for use.
                    231: 
                    232:     # disklabel -r ccd0
                    233:     # /dev/rccd0c:
                    234:     type: ccd
                    235:     disk: ccd
                    236:     label: default label
                    237:     flags:
                    238:     bytes/sector: 512
                    239:     sectors/track: 2048
                    240:     tracks/cylinder: 1
                    241:     sectors/cylinder: 2048
                    242:     cylinders: 6107
                    243:     total sectors: 12508812
                    244:     rpm: 3600
                    245:     interleave: 1
                    246:     trackskew: 0
                    247:     cylinderskew: 0
                    248:     headswitch: 0           # microseconds
                    249:     track-to-track seek: 0  # microseconds
                    250:     drivedata: 0
                    251:     #        size   offset    fstype   [fsize bsize   cpg]
                    252:       c: 12508812        0    4.2BSD     1024  8192    16  # (Cyl. 0 - 6107*)
                    253: 
                    254: The filesystem will then need to be formatted:
                    255: 
                    256:     # newfs /dev/rccd0c
                    257:     Warning: 372 sector(s) in last cylinder unallocated
                    258:     /dev/rccd0c:    12508812 sectors in 6108 cylinders of 1 tracks, 2048 sectors
                    259:             6107.8MB in 382 cyl groups (16 c/g, 16.00MB/g, 3968 i/g)
                    260:     
                    261:     super-block backups (for fsck -b #) at:
                    262:     [...]
                    263: 
                    264: ## Mount the filesystem
                    265: 
                    266: Once you have a created a file system on the CCD device, you can then mount the
                    267: file system against a mount point on your system. Be sure to mount the slice
                    268: labeled type `ffs` or `4.2BSD`:
                    269: 
                    270:     # mount /dev/ccd0c /mnt
                    271: 
                    272: Then:
                    273: 
                    274:     # export BLOCKSIZE=1024; df
                    275:     Filesystem  1K-blocks     Used   Avail Capacity  Mounted on
                    276:     /dev/sd6a      376155   320290   37057    89%    /
                    277:     /dev/ccd0c    6058800       1  5755859     0%    /mnt
                    278: 
                    279: Congratulations, you now have a working CCD. To configure the CCD device at boot
                    280: time, set `ccd=yes` in `/etc/rc.conf`. You can adjust `/etc/fstab` to get the
                    281: filesystem mounted at boot:
                    282: 
                    283:     /dev/ccd0c  /home ffs     rw    1 2
                    284: 

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb