File:  [NetBSD Developer Wiki] / wikisrc / guide / ccd.mdwn
Revision 1.1: download - view: text, annotated - select for diffs
Wed Mar 6 23:15:12 2013 UTC (13 months, 1 week ago) by jdf
Branches: MAIN
CVS tags: HEAD
Get ccd from the guide to the wiki

# Concatenated Disk Device (CCD) configuration

The CCD driver allows the user to *concatenate* several physical disks into one
pseudo volume. While RAIDframe (see [[NetBSD RAIDframe|guide/rf]]) also allows
doing this to create RAID level 0 sets, it does not allow you to do striping
across disks of different geometry, which is where CCD comes in handy. CCD also
allows for an *interleave* to improve disk performance with a gained space loss.
This example will not cover that feature.

The steps required to setup a CCD are as follows:

 1. Install physical media
 2. Configure kernel support
 3. Disklabel each volume member of the CCD
 4. Configure the CCD conf file
 5. Initialize the CCD device
 6. Create a filesystem on the new CCD device
 7. Mount the CCD filesystem

This example features a CCD setup on NetBSD/sparc 1.5. The CCD will reside on 4
SCSI disks in a generic external Sun disk pack chassis connected to the external
50 pin SCSI port.

## Install physical media

This step is at your own discretion, depending on your platform and the hardware
at your disposal.

From my DMESG:

    Disk #1:
      probe(esp0:0:0): max sync rate 10.00MB/s
      sd0 at scsibus0 target 0 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
      sd0: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
    
    Disk #2
      probe(esp0:1:0): max sync rate 10.00MB/s
      sd1 at scsibus0 target 1 lun 0: <SEAGATE, ST32430N SUN2.1G, 0444> SCSI2 0/direct fixed
      sd1: 2049 MB, 3992 cyl, 9 head, 116 sec, 512 bytes/sect x 4197405 sectors
    
    Disk #3
      probe(esp0:2:0): max sync rate 10.00MB/s
      sd2 at scsibus0 target 2 lun 0: <SEAGATE, ST11200N SUN1.05, 9500> SCSI2 0/direct fixed
      sd2: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors
    
    Disk #4
      probe(esp0:3:0): max sync rate 10.00MB/s
      sd3 at scsibus0 target 3 lun 0: <SEAGATE, ST11200N SUN1.05, 8808 > SCSI2 0
      sd3: 1005 MB, 1872 cyl, 15 head, 73 sec, 512 bytes/sect x 2059140 sectors

## Configure Kernel Support

The following kernel configuration directive is needed to provide CCD device
support. It is enabled in the GENERIC kernel:

    pseudo-device  ccd  4    # concatenated disk devices

In my kernel config, I also hard code SCSI ID associations to `/dev` device
entries to prevent bad things from happening:

    sd0     at scsibus0 target 0 lun ?
    # SCSI disk drives
    sd1     at scsibus0 target 1 lun ?
    # SCSI disk drives
    sd2     at scsibus0 target 2 lun ?
    # SCSI disk drives
    sd3     at scsibus0 target 3 lun ?
    # SCSI disk drives
    sd4     at scsibus0 target 4 lun ?
    # SCSI disk drives
    sd5     at scsibus0 target 5 lun ?
    # SCSI disk drives
    sd6     at scsibus0 target 6 lun ?
    # SCSI disk drives

## Disklabel each volume member of the CCD

Each member disk of the CCD will need a special file system established. In this
example, I will need to disklabel:

 * `/dev/rsd0c`  
 * `/dev/rsd1c`  
 * `/dev/rsd2c`  
 * `/dev/rsd3c`

### Note

Always remember to disklabel the character device, not the block device, in
`/dev/r{s,w}d*`.

### Note

On all platforms, the `c` slice is symbolic of the entire NetBSD partition and
is reserved.

You will probably want to remove any pre-existing disklabels on the disks in the
CCD. This can be accomplished in one of two ways with the
[dd(1)](http://netbsd.gw.com/cgi-bin/man-cgi?dd+1+NetBSD-5.0.1+i386) command:

    # dd if=/dev/zero of=/dev/rsd0c bs=8k count=1
    # dd if=/dev/zero of=/dev/rsd1c bs=8k count=1
    # dd if=/dev/zero of=/dev/rsd2c bs=8k count=1
    # dd if=/dev/zero of=/dev/rsd3c bs=8k count=1

If your port uses a MBR (Master Boot Record) to partition the disks so that the
NetBSD partitions are only part of the overall disk, and other OSs like Windows
or Linux use other parts, you can void the MBR and all partitions on disk by
using the command:

    # dd if=/dev/zero of=/dev/rsd0d bs=8k count=1
    # dd if=/dev/zero of=/dev/rsd1d bs=8k count=1
    # dd if=/dev/zero of=/dev/rsd2d bs=8k count=1
    # dd if=/dev/zero of=/dev/rsd3d bs=8k count=1

This will make all data on the entire disk inaccessible. Note that the entire
disk is slice `d` on i386 (and some other ports), and `c` elsewhere (e.g. on
sparc). See the `kern.rawpartition` sysctl -- `3` means `d`, `2` means `c`.

The default disklabel for the disk will look similar to this:

    # disklabel -r sd0
    [...snip...]
    bytes/sector: 512
    sectors/track: 116
    tracks/cylinder: 9
    sectors/cylinder: 1044
    cylinders: 3992
    total sectors: 4197405
    [..snip...]
    3 partitions:
    #        size   offset    fstype   [fsize bsize   cpg]
      c:  4197405       0     unused     1024  8192        # (Cyl.    0 - 4020*)

You will need to create one *slice* on the NetBSD partition of the disk that
consumes the entire partition. The slice must begin at least one cylinder offset
from the beginning of the disk/partition to provide space for the special CCD
disklabel. The offset should be 1x sectors/cylinder (see following note).
Therefore, the `size` value should be `total sectors` minus 1x
`sectors/cylinder`. Edit your disklabel accordingly:

    # disklabel -e sd0

*Note*: The offset of a slice of type `ccd` must be a multiple of the
`sectors/cylinder` value.

*Note*: Be sure to `export EDITOR=[path to your favorite editor]` before
editing the disklabels.

*Note*: The slice must be fstype `ccd`.

Because there will only be one slice on this partition, you can recycle the `c`
slice (normally reserved for symbolic uses). Change your disklabel to the
following:

    3 partitions:
    #        size   offset    fstype   [fsize bsize   cpg]
      c:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)

Optionally you can setup a slice other than `c` to use, simply adjust
accordingly below:

    3 partitions:
    #        size   offset    fstype   [fsize bsize   cpg]
      a:  4196361     1044       ccd                       # (Cyl. 1 - 4020*)
      c:  4197405       0     unused     1024  8192        # (Cyl. 0 - 4020*)

Be sure to write the label when you have completed. Disklabel will object to
your disklabel and prompt you to re-edit if it does not pass its sanity checks.

## Configure the CCD

Once all disks are properly labeled, you will need to generate a configuration
file, `/etc/ccd.conf`. The file does not exist by default, and you will need to
create a new one. The format is:

    #ccd    ileave    flags   component    devices

*Note*: For the `ileave`, if a value of zero is used then the disks are
concatenated, but if you use a value equal to the `sectors/track` number the
disks are interleaved.

Example in this case:

    # more /etc/ccd.conf
    ccd0  0  none /dev/sd0c /dev/sd1c /dev/sd2c /dev/sd3c

*Note*: The CCD driver expects block device files as components. Be sure not to
use character device files in the configuration.

## Initialize the CCD device

Once you are confident that your CCD configuration is sane, you can initialize
the device using the
[ccdconfig(8)](http://netbsd.gw.com/cgi-bin/man-cgi?ccdconfig+8+NetBSD-5.0.1+i386)
command: Configure:

    # ccdconfig -C -f /etc/ccd.conf

Unconfigure:

    # ccdconfig -u -f /etc/ccd.conf

Initializing the CCD device will activate `/dev` entries: `/dev/{,r}ccd#`:

    # ls -la  /dev/{,r}ccd0*
    brw-r-----  1 root  operator   9, 0 Apr 28 21:35 /dev/ccd0a
    brw-r-----  1 root  operator   9, 1 Apr 28 21:35 /dev/ccd0b
    brw-r-----  1 root  operator   9, 2 May 12 00:10 /dev/ccd0c
    brw-r-----  1 root  operator   9, 3 Apr 28 21:35 /dev/ccd0d
    brw-r-----  1 root  operator   9, 4 Apr 28 21:35 /dev/ccd0e
    brw-r-----  1 root  operator   9, 5 Apr 28 21:35 /dev/ccd0f
    brw-r-----  1 root  operator   9, 6 Apr 28 21:35 /dev/ccd0g
    brw-r-----  1 root  operator   9, 7 Apr 28 21:35 /dev/ccd0h
    crw-r-----  1 root  operator  23, 0 Jun 12 20:40 /dev/rccd0a
    crw-r-----  1 root  operator  23, 1 Apr 28 21:35 /dev/rccd0b
    crw-r-----  1 root  operator  23, 2 Jun 12 20:58 /dev/rccd0c
    crw-r-----  1 root  operator  23, 3 Apr 28 21:35 /dev/rccd0d
    crw-r-----  1 root  operator  23, 4 Apr 28 21:35 /dev/rccd0e
    crw-r-----  1 root  operator  23, 5 Apr 28 21:35 /dev/rccd0f
    crw-r-----  1 root  operator  23, 6 Apr 28 21:35 /dev/rccd0g
    crw-r-----  1 root  operator  23, 7 Apr 28 21:35 /dev/rccd0h

## Create a 4.2BSD/UFS filesystem on the new CCD device

You may now disklabel the new virtual disk device associated with your CCD:

    # disklabel -e ccd0

Once again, there will be only one slice, so you may either recycle the `c`
slice or create a separate slice for use.

    # disklabel -r ccd0
    # /dev/rccd0c:
    type: ccd
    disk: ccd
    label: default label
    flags:
    bytes/sector: 512
    sectors/track: 2048
    tracks/cylinder: 1
    sectors/cylinder: 2048
    cylinders: 6107
    total sectors: 12508812
    rpm: 3600
    interleave: 1
    trackskew: 0
    cylinderskew: 0
    headswitch: 0           # microseconds
    track-to-track seek: 0  # microseconds
    drivedata: 0
    #        size   offset    fstype   [fsize bsize   cpg]
      c: 12508812        0    4.2BSD     1024  8192    16  # (Cyl. 0 - 6107*)

The filesystem will then need to be formatted:

    # newfs /dev/rccd0c
    Warning: 372 sector(s) in last cylinder unallocated
    /dev/rccd0c:    12508812 sectors in 6108 cylinders of 1 tracks, 2048 sectors
            6107.8MB in 382 cyl groups (16 c/g, 16.00MB/g, 3968 i/g)
    
    super-block backups (for fsck -b #) at:
    [...]

## Mount the filesystem

Once you have a created a file system on the CCD device, you can then mount the
file system against a mount point on your system. Be sure to mount the slice
labeled type `ffs` or `4.2BSD`:

    # mount /dev/ccd0c /mnt

Then:

    # export BLOCKSIZE=1024; df
    Filesystem  1K-blocks     Used   Avail Capacity  Mounted on
    /dev/sd6a      376155   320290   37057    89%    /
    /dev/ccd0c    6058800       1  5755859     0%    /mnt

Congratulations, you now have a working CCD. To configure the CCD device at boot
time, set `ccd=yes` in `/etc/rc.conf`. You can adjust `/etc/fstab` to get the
filesystem mounted at boot:

    /dev/ccd0c  /home ffs     rw    1 2


CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb