Annotation of wikisrc/guide/ccd.mdwn, revision 1.1

1.1     ! jdf         1: # Concatenated Disk Device (CCD) configuration
        !             2: 
        !             3: The CCD driver allows the user to *concatenate* several physical disks into one
        !             4: pseudo volume. While RAIDframe (see [[NetBSD RAIDframe|guide/rf]]) also allows
        !             5: doing this to create RAID level 0 sets, it does not allow you to do striping
        !             6: across disks of different geometry, which is where CCD comes in handy. CCD also
        !             7: allows for an *interleave* to improve disk performance with a gained space loss.
        !             8: This example will not cover that feature.
        !             9: 
        !            10: The steps required to setup a CCD are as follows:
        !            11: 
        !            12:  1. Install physical media
        !            13:  2. Configure kernel support
        !            14:  3. Disklabel each volume member of the CCD
        !            15:  4. Configure the CCD conf file
        !            16:  5. Initialize the CCD device
        !            17:  6. Create a filesystem on the new CCD device
        !            18:  7. Mount the CCD filesystem
        !            19: 
        !            20: This example features a CCD setup on NetBSD/sparc 1.5. The CCD will reside on 4
        !            21: SCSI disks in a generic external Sun disk pack chassis connected to the external
        !            22: 50 pin SCSI port.
        !            23: 
        !            24: ## Install physical media
        !            25: 
        !            26: This step is at your own discretion, depending on your platform and the hardware
        !            27: at your disposal.
        !            28: 
        !            29: From my DMESG:
        !            30: 
        !            31:     Disk #1:
        !            32:       probe(esp0:0:0): max sync rate 10.00MB/s
        !            33:       sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
        !            34:       sd0: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
        !            35:     
        !            36:     Disk #2
        !            37:       probe(esp0:1:0): max sync rate 10.00MB/s
        !            38:       sd1 at scsibus0 target 1 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
        !            39:       sd1: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
        !            40:     
        !            41:     Disk #3
        !            42:       probe(esp0:2:0): max sync rate 10.00MB/s
        !            43:       sd2 at scsibus0 target 2 lun 0: <SEAGATE, ST11200N SUN1.05, 9500> SCSI2 0/direct fixed
        !            44:       sd2: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors
        !            45:     
        !            46:     Disk #4
        !            47:       probe(esp0:3:0): max sync rate 10.00MB/s
        !            48:       sd3 at scsibus0 target 3 lun 0: <SEAGATE, ST11200N SUN1.05, 8808 > SCSI2 0
        !            49:       sd3: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors
        !            50: 
        !            51: ## Configure Kernel Support
        !            52: 
        !            53: The following kernel configuration directive is needed to provide CCD device
        !            54: support. It is enabled in the GENERIC kernel:
        !            55: 
        !            56:     pseudo-device  ccd  4    # concatenated disk devices
        !            57: 
        !            58: In my kernel config, I also hard code SCSI ID associations to `/dev` device
        !            59: entries to prevent bad things from happening:
        !            60: 
        !            61:     sd0     at scsibus0 target 0 lun ?
        !            62:     # SCSI disk drives
        !            63:     sd1     at scsibus0 target 1 lun ?
        !            64:     # SCSI disk drives
        !            65:     sd2     at scsibus0 target 2 lun ?
        !            66:     # SCSI disk drives
        !            67:     sd3     at scsibus0 target 3 lun ?
        !            68:     # SCSI disk drives
        !            69:     sd4     at scsibus0 target 4 lun ?
        !            70:     # SCSI disk drives
        !            71:     sd5     at scsibus0 target 5 lun ?
        !            72:     # SCSI disk drives
        !            73:     sd6     at scsibus0 target 6 lun ?
        !            74:     # SCSI disk drives
        !            75: 
        !            76: ## Disklabel each volume member of the CCD
        !            77: 
        !            78: Each member disk of the CCD will need a special file system established. In this
        !            79: example, I will need to disklabel:
        !            80: 
        !            81:  * `/dev/rsd0c`  
        !            82:  * `/dev/rsd1c`  
        !            83:  * `/dev/rsd2c`  
        !            84:  * `/dev/rsd3c`
        !            85: 
        !            86: ### Note
        !            87: 
        !            88: Always remember to disklabel the character device, not the block device, in
        !            89: `/dev/r{s,w}d*`.
        !            90: 
        !            91: ### Note
        !            92: 
        !            93: On all platforms, the `c` slice is symbolic of the entire NetBSD partition and
        !            94: is reserved.
        !            95: 
        !            96: You will probably want to remove any pre-existing disklabels on the disks in the
        !            97: CCD. This can be accomplished in one of two ways with the
        !            98: [dd(1)](http://netbsd.gw.com/cgi-bin/man-cgi?dd+1+NetBSD-5.0.1+i386) command:
        !            99: 
        !           100:     # dd if=/dev/zero of=/dev/rsd0c bs=8k count=1
        !           101:     # dd if=/dev/zero of=/dev/rsd1c bs=8k count=1
        !           102:     # dd if=/dev/zero of=/dev/rsd2c bs=8k count=1
        !           103:     # dd if=/dev/zero of=/dev/rsd3c bs=8k count=1
        !           104: 
        !           105: If your port uses a MBR (Master Boot Record) to partition the disks so that the
        !           106: NetBSD partitions are only part of the overall disk, and other OSs like Windows
        !           107: or Linux use other parts, you can void the MBR and all partitions on disk by
        !           108: using the command:
        !           109: 
        !           110:     # dd if=/dev/zero of=/dev/rsd0d bs=8k count=1
        !           111:     # dd if=/dev/zero of=/dev/rsd1d bs=8k count=1
        !           112:     # dd if=/dev/zero of=/dev/rsd2d bs=8k count=1
        !           113:     # dd if=/dev/zero of=/dev/rsd3d bs=8k count=1
        !           114: 
        !           115: This will make all data on the entire disk inaccessible. Note that the entire
        !           116: disk is slice `d` on i386 (and some other ports), and `c` elsewhere (e.g. on
        !           117: sparc). See the `kern.rawpartition` sysctl -- `3` means `d`, `2` means `c`.
        !           118: 
        !           119: The default disklabel for the disk will look similar to this:
        !           120: 
        !           121:     # disklabel -r sd0
        !           122:     [...snip...]
        !           123:     bytes/sector: 512
        !           124:     sectors/track: 116
        !           125:     tracks/cylinder: 9
        !           126:     sectors/cylinder: 1044
        !           127:     cylinders: 3992
        !           128:     total sectors: 4197405
        !           129:     [..snip...]
        !           130:     3 partitions:
        !           131:     #        size   offset    fstype   [fsize bsize   cpg]
        !           132:       c:  4197405       0     unused     1024  8192        # (Cyl.    0 - 4020*)
        !           133: 
        !           134: You will need to create one *slice* on the NetBSD partition of the disk that
        !           135: consumes the entire partition. The slice must begin at least one cylinder offset
        !           136: from the beginning of the disk/partition to provide space for the special CCD
        !           137: disklabel. The offset should be 1x sectors/cylinder (see following note).
        !           138: Therefore, the `size` value should be `total sectors` minus 1x
        !           139: `sectors/cylinder`. Edit your disklabel accordingly:
        !           140: 
        !           141:     # disklabel -e sd0
        !           142: 
        !           143: *Note*: The offset of a slice of type `ccd` must be a multiple of the
        !           144: `sectors/cylinder` value.
        !           145: 
        !           146: *Note*: Be sure to `export EDITOR=[path to your favorite editor]` before
        !           147: editing the disklabels.
        !           148: 
        !           149: *Note*: The slice must be fstype `ccd`.
        !           150: 
        !           151: Because there will only be one slice on this partition, you can recycle the `c`
        !           152: slice (normally reserved for symbolic uses). Change your disklabel to the
        !           153: following:
        !           154: 
        !           155:     3 partitions:
        !           156:     #        size   offset    fstype   [fsize bsize   cpg]
        !           157:       c:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
        !           158: 
        !           159: Optionally you can setup a slice other than `c` to use, simply adjust
        !           160: accordingly below:
        !           161: 
        !           162:     3 partitions:
        !           163:     #        size   offset    fstype   [fsize bsize   cpg]
        !           164:       a:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
        !           165:       c:  4197405       0     unused     1024  8192        # (Cyl. 0 - 4020*)
        !           166: 
        !           167: Be sure to write the label when you have completed. Disklabel will object to
        !           168: your disklabel and prompt you to re-edit if it does not pass its sanity checks.
        !           169: 
        !           170: ## Configure the CCD
        !           171: 
        !           172: Once all disks are properly labeled, you will need to generate a configuration
        !           173: file, `/etc/ccd.conf`. The file does not exist by default, and you will need to
        !           174: create a new one. The format is:
        !           175: 
        !           176:     #ccd    ileave    flags   component    devices
        !           177: 
        !           178: *Note*: For the `ileave`, if a value of zero is used then the disks are
        !           179: concatenated, but if you use a value equal to the `sectors/track` number the
        !           180: disks are interleaved.
        !           181: 
        !           182: Example in this case:
        !           183: 
        !           184:     # more /etc/ccd.conf
        !           185:     ccd0  0  none /dev/sd0c /dev/sd1c /dev/sd2c /dev/sd3c
        !           186: 
        !           187: *Note*: The CCD driver expects block device files as components. Be sure not to
        !           188: use character device files in the configuration.
        !           189: 
        !           190: ## Initialize the CCD device
        !           191: 
        !           192: Once you are confident that your CCD configuration is sane, you can initialize
        !           193: the device using the
        !           194: [ccdconfig(8)](http://netbsd.gw.com/cgi-bin/man-cgi?ccdconfig+8+NetBSD-5.0.1+i386)
        !           195: command: Configure:
        !           196: 
        !           197:     # ccdconfig -C -f /etc/ccd.conf
        !           198: 
        !           199: Unconfigure:
        !           200: 
        !           201:     # ccdconfig -u -f /etc/ccd.conf
        !           202: 
        !           203: Initializing the CCD device will activate `/dev` entries: `/dev/{,r}ccd#`:
        !           204: 
        !           205:     # ls -la  /dev/{,r}ccd0*
        !           206:     brw-r-----  1 root  operator   9, 0 Apr 28 21:35 /dev/ccd0a
        !           207:     brw-r-----  1 root  operator   9, 1 Apr 28 21:35 /dev/ccd0b
        !           208:     brw-r-----  1 root  operator   9, 2 May 12 00:10 /dev/ccd0c
        !           209:     brw-r-----  1 root  operator   9, 3 Apr 28 21:35 /dev/ccd0d
        !           210:     brw-r-----  1 root  operator   9, 4 Apr 28 21:35 /dev/ccd0e
        !           211:     brw-r-----  1 root  operator   9, 5 Apr 28 21:35 /dev/ccd0f
        !           212:     brw-r-----  1 root  operator   9, 6 Apr 28 21:35 /dev/ccd0g
        !           213:     brw-r-----  1 root  operator   9, 7 Apr 28 21:35 /dev/ccd0h
        !           214:     crw-r-----  1 root  operator  23, 0 Jun 12 20:40 /dev/rccd0a
        !           215:     crw-r-----  1 root  operator  23, 1 Apr 28 21:35 /dev/rccd0b
        !           216:     crw-r-----  1 root  operator  23, 2 Jun 12 20:58 /dev/rccd0c
        !           217:     crw-r-----  1 root  operator  23, 3 Apr 28 21:35 /dev/rccd0d
        !           218:     crw-r-----  1 root  operator  23, 4 Apr 28 21:35 /dev/rccd0e
        !           219:     crw-r-----  1 root  operator  23, 5 Apr 28 21:35 /dev/rccd0f
        !           220:     crw-r-----  1 root  operator  23, 6 Apr 28 21:35 /dev/rccd0g
        !           221:     crw-r-----  1 root  operator  23, 7 Apr 28 21:35 /dev/rccd0h
        !           222: 
        !           223: ## Create a 4.2BSD/UFS filesystem on the new CCD device
        !           224: 
        !           225: You may now disklabel the new virtual disk device associated with your CCD:
        !           226: 
        !           227:     # disklabel -e ccd0
        !           228: 
        !           229: Once again, there will be only one slice, so you may either recycle the `c`
        !           230: slice or create a separate slice for use.
        !           231: 
        !           232:     # disklabel -r ccd0
        !           233:     # /dev/rccd0c:
        !           234:     type: ccd
        !           235:     disk: ccd
        !           236:     label: default label
        !           237:     flags:
        !           238:     bytes/sector: 512
        !           239:     sectors/track: 2048
        !           240:     tracks/cylinder: 1
        !           241:     sectors/cylinder: 2048
        !           242:     cylinders: 6107
        !           243:     total sectors: 12508812
        !           244:     rpm: 3600
        !           245:     interleave: 1
        !           246:     trackskew: 0
        !           247:     cylinderskew: 0
        !           248:     headswitch: 0           # microseconds
        !           249:     track-to-track seek: 0  # microseconds
        !           250:     drivedata: 0
        !           251:     #        size   offset    fstype   [fsize bsize   cpg]
        !           252:       c: 12508812        0    4.2BSD     1024  8192    16  # (Cyl. 0 - 6107*)
        !           253: 
        !           254: The filesystem will then need to be formatted:
        !           255: 
        !           256:     # newfs /dev/rccd0c
        !           257:     Warning: 372 sector(s) in last cylinder unallocated
        !           258:     /dev/rccd0c:    12508812 sectors in 6108 cylinders of 1 tracks, 2048 sectors
        !           259:             6107.8MB in 382 cyl groups (16 c/g, 16.00MB/g, 3968 i/g)
        !           260:     
        !           261:     super-block backups (for fsck -b #) at:
        !           262:     [...]
        !           263: 
        !           264: ## Mount the filesystem
        !           265: 
        !           266: Once you have a created a file system on the CCD device, you can then mount the
        !           267: file system against a mount point on your system. Be sure to mount the slice
        !           268: labeled type `ffs` or `4.2BSD`:
        !           269: 
        !           270:     # mount /dev/ccd0c /mnt
        !           271: 
        !           272: Then:
        !           273: 
        !           274:     # export BLOCKSIZE=1024; df
        !           275:     Filesystem  1K-blocks     Used   Avail Capacity  Mounted on
        !           276:     /dev/sd6a      376155   320290   37057    89%    /
        !           277:     /dev/ccd0c    6058800       1  5755859     0%    /mnt
        !           278: 
        !           279: Congratulations, you now have a working CCD. To configure the CCD device at boot
        !           280: time, set `ccd=yes` in `/etc/rc.conf`. You can adjust `/etc/fstab` to get the
        !           281: filesystem mounted at boot:
        !           282: 
        !           283:     /dev/ccd0c  /home ffs     rw    1 2
        !           284: 

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb