[BACK]Return to set-up_raidframe.mdwn CVS log [TXT] [DIR] Up to [NetBSD Developer Wiki] / wikisrc

Diff for /wikisrc/set-up_raidframe.mdwn between versions 1.1 and 1.2

version 1.1, 2011/05/13 13:25:35 version 1.2, 2011/05/15 14:21:19
Line 1 Line 1
 [[!meta  title="Setting up raidframe(4) on NetBSD"]]  [[!meta  title="Setting up raidframe(4) on NetBSD"]]
   
 After posting lots of questions, and getting lots of expert help, I  After posting lots of questions, and getting lots of expert help, I finally got my new server up and running with raid!  Since I asked so many questions, it was suggested that I recap the process so that someone else might be able to make it work without having to ask!
 finally got my new server up and running with raid!  Since I asked  
 so many questions, it was suggested that I recap the process so that  
 someone else might be able to make it work without having to ask!  
   
 So ...  So ...
   
 Here's what I did recently to get raidframe(4) up and running on my  Here's what I did recently to get raidframe(4) up and running on my new server.
 new server.  
   
 0. This is really important!  Read the man page for raidctl(8).  It  0. This is really important!  Read the man page for raidctl(8).  It is a long man page, but it is important to understand how things work.  And there are some actual examples towards the end, which will only make sense if you've read the intro material.  Don't be intimidated by all the detail, just RTFM before you start!
    is a long man page, but it is important to understand how things  
    work.  And there are some actual examples towards the end, which  
    will only make sense if you've read the intro material.  Don't be  
    intimidated by all the detail, just RTFM before you start!  
   
 1. Use dd(1) to completely erase the drives.  This is useful to make  
    sure that any existing labels/wedges/etc. get removed, and also  
    makes it easier to initialize the raidset parity.  On Western  
    Digital SATA-II 500GB drive, it took less than two hours (each)  
    for this step.  (The operation starts off at ~110GB/sec, but will  
    get slower as the heads get closer to the center of the drive   
    where there are fewer sectors per cylinder.)  
   
         dd if=/dev/zero bs=32k of=/dev/rwd2a  
         dd if=/dev/zero bs=32k of=/dev/rwd3a  
   
 2. This step might not be necessary but I wasn't taking any chances!  
    Also, this step is only applicable on ports which use DOS disk  
    layouts.  For other ports, you should be able to skip this step,   
    and proceed directly to Step 3.  
   
    Use fdisk(8) to set up DOS/Windows partition data.  Make sure you  
    use the "-A 2048" option for alignment.  
   
         fdisk -uai -A 2048 wd2a  
         fdisk -uai -A 2048 wd3a  
   
    (If you are using an older version of fdisk and get the error  
   
         fdisk: Bad argument to the -A flag.  
   
    you can specify "-A 2048/2048" instead.)  
   
    Set the partition 0 type to 169 (NetBSD), and let it occupy the  
    entire drive.  If you're going to boot from this raidset, you  
    should also make this the active partition, and update the boot  
    blocks when prompted.  
   
    When finished, this is what my drive reports:  
   
         Partitions aligned to 16065 sector boundaries, offset 63  
   
         Partition table:  
         0: NetBSD (sysid 169)  
             start 2048, size 976771120 (476939 MB, Cyls 0/32/33-60801/80/63),  
                 Active  
                 PBR is not bootable: All bytes are identical (0x00)  
         1: <UNUSED>  
         2: <UNUSED>  
         3: <UNUSED>  
         Bootselector disabled.  
         First active partition: 0  
   
    (The "PBR is not bootable" line will not appear if you later use  
    installboot(8) - see step 12 below.)  
   
 3. Use disklabel(8) to provide a NetBSD disklabel on each member of  
    the raidset.  Create an 'e' partition that describes the whole  
    disk - so it is essentially a copy of the 'c' partition.  Make  
    sure that the offset of your 'e' partition is large enough to  
    accomodate any architecture-specific bootstrap requirements, and  
    that the offset is a multiple of the hard drive's native transfer  
    size.  
   
    Set the fstype on your 'e' partition to RAID (4.2BSD will also work,  
    but not if you want to use RAID_AUTOCONFIG).  
   
    Here's my partition info:  
   
    5 partitions:  
    #        size  offset  fstype [fsize bsize cpg/sgs]  
     c: 976771120    2048  unused      0     0        # (Cyl.    2*- 969020)  
     d: 976773168       0  unused      0     0        # (Cyl.    0 - 969020)  
     e: 976771120    2048    RAID                     # (Cyl.    2*- 969020)  
   
 4: Now create a raid.conf file to describe the raidset.  Mine is a  
    simple RAID-1 (mirror):  
   
         START array  
         #numrow numcol numspare  
         1 2 0  
   
         # Identify physical disks  
         START disks  
         /dev/wd2e  
         /dev/wd3e  
   
         # Layout is simple - 64 sectors per stripe  
         START layout  
         #Sect/StripeUnit StripeUnit/ParityUnit StripeUnit/ReconUnit RaidLevel  
         128 1 1 1  
   
         # No spares  
         #START spare  
   
         # Command queueing  
         START queue  
         fifo 100  
   
 5. Create the raidset using raidctl(8).  Until you do this for the first  
    time, there are no raid "component labels" on the member drives, so you  
    will need to use the "-C" option;  if the raidset has been previously  
    created, you should use "-c" instead (this is what /etc/rc.d/raidframe  
    uses at system startup).  
   
         raidctl -C /etc/raid1.conf raid1  1. Use dd(1) to completely erase the drives.  This is useful to make sure that any existing labels/wedges/etc. get removed, and also makes it easier to initialize the raidset parity.  On Western Digital SATA-II 500GB drive, it took less than two hours (each) for this step.  (The operation starts off at ~110GB/sec, but will get slower as the heads get closer to the center of the drive where there are fewer sectors per cylinder.)
   
           dd if=/dev/zero bs=32k of=/dev/rwd2a
           dd if=/dev/zero bs=32k of=/dev/rwd3a
   
   2. This step might not be necessary but I wasn't taking any chances!  Also, this step is only applicable on ports which use DOS disk layouts.  For other ports, you should be able to skip this step, and proceed directly to Step 3.
   
    Use fdisk(8) to set up DOS/Windows partition data.  Make sure you use the "-A 2048" option for alignment.
   
           fdisk -uai -A 2048 wd2a
           fdisk -uai -A 2048 wd3a
   
    (If you are using an older version of fdisk and get the error
   
           fdisk: Bad argument to the -A flag.
   
    you can specify "-A 2048/2048" instead.)
   
    Set the partition 0 type to 169 (NetBSD), and let it occupy the entire drive.  If you're going to boot from this raidset, you should also make this the active partition, and update the boot blocks when prompted.
   
    When finished, this is what my drive reports:
   
           Partitions aligned to 16065 sector boundaries, offset 63
   
           Partition table:
           0: NetBSD (sysid 169)
               start 2048, size 976771120 (476939 MB, Cyls 0/32/33-60801/80/63),
                   Active
                   PBR is not bootable: All bytes are identical (0x00)
           1: <UNUSED>
           2: <UNUSED>
           3: <UNUSED>
           Bootselector disabled.
           First active partition: 0
   
    (The "PBR is not bootable" line will not appear if you later use installboot(8) - see step 12 below.)
   
   3. Use disklabel(8) to provide a NetBSD disklabel on each member of the raidset.  Create an 'e' partition that describes the whole disk - so it is essentially a copy of the 'c' partition.  Make sure that the offset of your 'e' partition is large enough to accomodate any architecture-specific bootstrap requirements, and that the offset is a multiple of the hard drive's native transfer size.
   
    Set the fstype on your 'e' partition to RAID (4.2BSD will also work, but not if you want to use RAID_AUTOCONFIG).
   
    Here's my partition info:
   
           5 partitions:
           #        size  offset  fstype [fsize bsize cpg/sgs]
            c: 976771120    2048  unused      0     0        # (Cyl.    2*- 969020)
            d: 976773168       0  unused      0     0        # (Cyl.    0 - 969020)
            e: 976771120    2048    RAID                     # (Cyl.    2*- 969020)
   
   4. Now create a raid.conf file to describe the raidset.  Mine is a simple RAID-1 (mirror):
   
           START array
           #numrow numcol numspare
           1 2 0
   
           # Identify physical disks
           START disks
           /dev/wd2e
           /dev/wd3e
   
           # Layout is simple - 64 sectors per stripe
           START layout
           #Sect/StripeUnit StripeUnit/ParityUnit StripeUnit/ReconUnit RaidLevel
           128 1 1 1
   
           # No spares
           #START spare
   
           # Command queueing
           START queue
           fifo 100
   
   5. Create the raidset using raidctl(8).  Until you do this for the first time, there are no raid "component labels" on the member drives, so you will need to use the "-C" option;  if the raidset has been previously created, you should use "-c" instead (this is what /etc/rc.d/raidframe uses at system startup).
   
           raidctl -C /etc/raid1.conf raid1
   
 6. Next, initialize the component labels with raidctl.  6. Next, initialize the component labels with raidctl.
   
         raidctl -I 201105061405 raid1          raidctl -I 201105061405 raid1
   
    (For serial number, I just used the current date/time; any 64-bit integer will work.)
   
   7. Now use raidctl to initialize the parity of the raidset.  On my 500GB drives, this took about 1.5 hours.
   
           raidctl -i raid1
   
    It won't tie up your session, as the parity operation happens in the kernel in the background.  You can monitor the progress using
   
           raidctl -s raid1
   
   8. When parity is finished, you can treat the raidset as a normal disk. You will need to write a NetBSD disklabel (this label is separate from the labels on the physical drives).  Make certain that all the offsets are multiples of the drive's native block size; otherwise you will have unaligned transfers and performance will be very bad!  (This is also why, in step 2 above, we use "-A 2048" for fdisk rather than letting it default to 63!")
   
    Here is what my drive looks like:
   
           7 partitions:
           #        size    offset  fstype [fsize bsize cpg/sgs]
            c: 976770944         0  unused      0     0       # (Cyl.      0 - 953877*)
            d: 976770944         0  unused      0     0       # (Cyl.      0 - 953877*)
            e: 536870912         0  4.2BSD   2048 16384    0  # (Cyl.      0 - 524287)
            f: 104857600 536870912  4.2BSD   2048 16384    0  # (Cyl. 524288 - 626687)
            g: 335042432 641728512  4.2BSD   2048 16384    0  # (Cyl. 626688 - 953877*)
   
   9. Now, just as you would for a real disk, use newfs to initialize the filesystems.  Then mount them and start using your new raidset!
   
           newfs /dev/raid1e
           mount /dev/raid1e /mnt
   
   10. If your kernel includes the RAID_AUTOCONFIG option, you can use raidctl to let your system automatically reconfigure the raidset at every boot.  (If you don't do this, system startup will still take care of this, as long as your configuration file is properly named "raid[0-9].conf" or "raid[1-9][0-9].conf" and is located in /etc/ directory.)
   
           raidctl -A yes raid1
   
   11. If you're going to boot from the raidset, you probably want to have an 'a' partition.  Here's the partition from my other, bootable, raidset:
   
    (For serial number, I just used the current date/time; any 64-bit          6 partitions:
    integer will work.)          #        size    offset  fstype [fsize bsize cpg/sgs]
            a:  41943040         0  4.2BSD   2048 16384    0  # (Cyl.      0 -  40959)
            b:  62914560  41943040    swap                    # (Cyl.  40960 - 102399)
            c: 488395008         0  unused      0     0       # (Cyl.      0 - 476948*)
            d: 488395008         0  unused      0     0       # (Cyl.      0 - 476948*)
            e: 125829120 104857600  4.2BSD   2048 16384    0  # (Cyl. 102400 - 225279)
            f: 257708288 230686720  4.2BSD   2048 16384    0  # (Cyl. 225280 - 476948*)
   
 7. Now use raidctl to initialize the parity of the raidset.  On my 500GB  12. You will also need to run installboot(8) to make the raidset bootable. Do this for each member drive:
    drives, this took about 1.5 hours.  
   
         raidctl -i raid1          mount /dev/raid0a /mnt
           installboot /dev/wd0a /usr/mdec/bootxx_ffsv1 /mnt/boot
    It won't tie up your session, as the parity operation happens in the          installboot /dev/wd1a /usr/mdec/bootxx_ffsv1 /mnt/boot
    kernel in the background.  You can monitor the progress using  
   
         raidctl -s raid1  
   
 8. When parity is finished, you can treat the raidset as a normal disk.  
    You will need to write a NetBSD disklabel (this label is separate from  
    the labels on the physical drives).  Make certain that all the offsets  
    are multiples of the drive's native block size; otherwise you will have  
    unaligned transfers and performance will be very bad!  (This is also  
    why, in step 2 above, we use "-A 2048" for fdisk rather than letting it  
    default to 63!")  
   
    Here is what my drive looks like:  
   
    7 partitions:  
    #        size    offset  fstype [fsize bsize cpg/sgs]  
     c: 976770944         0  unused      0     0       # (Cyl.      0 - 953877*)  
     d: 976770944         0  unused      0     0       # (Cyl.      0 - 953877*)  
     e: 536870912         0  4.2BSD   2048 16384    0  # (Cyl.      0 - 524287)  
     f: 104857600 536870912  4.2BSD   2048 16384    0  # (Cyl. 524288 - 626687)  
     g: 335042432 641728512  4.2BSD   2048 16384    0  # (Cyl. 626688 - 953877*)  
   
 9. Now, just as you would for a real disk, use newfs to initialize the  
    filesystems.  Then mount them and start using your new raidset!  
   
         newfs /dev/raid1e  
         mount /dev/raid1e /mnt  
   
 10.If your kernel includes the RAID_AUTOCONFIG option, you can use  
    raidctl to let your system automatically reconfigure the raidset at  
    every boot.  (If you don't do this, system startup will still take  
    care of this, as long as your configuration file is properly named  
    "raid[0-9].conf" or "raid[1-9][0-9].conf" and is located in /etc  
    directory.)  
   
         raidctl -A yes raid1  
   
 11.If you're going to boot from the raidset, you probably want to have  
    an 'a' partition.  Here's the partition from my other, bootable,  
    raidset:  
   
    6 partitions:  
    #        size    offset  fstype [fsize bsize cpg/sgs]  
     a:  41943040         0  4.2BSD   2048 16384    0  # (Cyl.      0 -  40959)  
     b:  62914560  41943040    swap                    # (Cyl.  40960 - 102399)  
     c: 488395008         0  unused      0     0       # (Cyl.      0 - 476948*)  
     d: 488395008         0  unused      0     0       # (Cyl.      0 - 476948*)  
     e: 125829120 104857600  4.2BSD   2048 16384    0  # (Cyl. 102400 - 225279)  
     f: 257708288 230686720  4.2BSD   2048 16384    0  # (Cyl. 225280 - 476948*)  
   
 12.You will also need to run installboot(8) to make the raidset bootable.  
    Do this for each member drive:  
   
         mount /dev/raid0a /mnt  
         installboot /dev/wd0a /usr/mdec/bootxx_ffsv1 /mnt/boot  
         installboot /dev/wd1a /usr/mdec/bootxx_ffsv1 /mnt/boot  
   
 13.Finally, you need to make sure your kernel has the RAID_AUTOCONFIG  13. Finally, you need to make sure your kernel has the RAID_AUTOCONFIG option, and enable the raidset as the root device:
    option, and enable the raidset as the root device:  
   
         raidctl -A root raid0          raidctl -A root raid0
Removed from v.1.1  
changed lines
  Added in v.1.2

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb