Annotation of wikisrc/set-up_raidframe.mdwn, revision 1.7

1.1       pgoyette    1: [[!meta  title="Setting up raidframe(4) on NetBSD"]]
                      2: 
1.2       wiki        3: After posting lots of questions, and getting lots of expert help, I finally got my new server up and running with raid!  Since I asked so many questions, it was suggested that I recap the process so that someone else might be able to make it work without having to ask!
1.1       pgoyette    4: 
                      5: So ...
                      6: 
1.7     ! jruoho      7: Here's what I did recently to get  [[!template id=man name="raid" section="4"]] up and running on my new server.
1.1       pgoyette    8: 
1.7     ! jruoho      9: 1. This is really important!  Read the man page for  [[!template id=man name="raidctl" section="8"]].  It is a long man page, but it is important to understand how things work.  And there are some actual examples towards the end, which will only make sense if you've read the intro material.  Don't be intimidated by all the detail, just RTFM before you start!
1.1       pgoyette   10: 
1.7     ! jruoho     11: 2. Use [[!template id=man name="dd" section="1"]] to completely erase the drives.  This is useful to make sure that any existing labels/wedges/etc. get removed, and also makes it easier to initialize the raidset parity.  On Western Digital SATA-II 500GB drive, it took less than two hours (each) for this step.  (The operation starts off at ~110GB/sec, but will get slower as the heads get closer to the center of the drive where there are fewer sectors per cylinder.)
1.2       wiki       12: 
                     13:        dd if=/dev/zero bs=32k of=/dev/rwd2a
                     14:        dd if=/dev/zero bs=32k of=/dev/rwd3a
                     15: 
1.5       wiki       16: 3. This step might not be necessary but I wasn't taking any chances!  Also, this step is only applicable on ports which use DOS disk layouts.  For other ports, you should be able to skip this step, and proceed directly to Step 4.
1.2       wiki       17: 
1.7     ! jruoho     18:  Use  [[!template id=man name="fdisk" section="8"]] to set up DOS/Windows partition data.  Make sure you use the "-A 2048" option for alignment.
1.2       wiki       19: 
                     20:        fdisk -uai -A 2048 wd2a
                     21:        fdisk -uai -A 2048 wd3a
                     22: 
                     23:  (If you are using an older version of fdisk and get the error
                     24: 
                     25:        fdisk: Bad argument to the -A flag.
                     26: 
                     27:  you can specify "-A 2048/2048" instead.)
                     28: 
                     29:  Set the partition 0 type to 169 (NetBSD), and let it occupy the entire drive.  If you're going to boot from this raidset, you should also make this the active partition, and update the boot blocks when prompted.
                     30: 
                     31:  When finished, this is what my drive reports:
                     32: 
                     33:        Partitions aligned to 16065 sector boundaries, offset 63
                     34: 
                     35:        Partition table:
                     36:        0: NetBSD (sysid 169)
                     37:            start 2048, size 976771120 (476939 MB, Cyls 0/32/33-60801/80/63),
                     38:                Active
                     39:                PBR is not bootable: All bytes are identical (0x00)
                     40:        1: <UNUSED>
                     41:        2: <UNUSED>
                     42:        3: <UNUSED>
                     43:        Bootselector disabled.
                     44:        First active partition: 0
                     45: 
1.7     ! jruoho     46:  (The "PBR is not bootable" line will not appear if you later use [[!template id=man name="installboot" section="8"]] - see step 14 below.)
1.2       wiki       47: 
1.7     ! jruoho     48: 4. Use  [[!template id=man name="disklabel" section="8"]] to provide a NetBSD disklabel on each member of the raidset.  Create an 'e' partition that describes the whole disk - so it is essentially a copy of the 'c' partition.  Make sure that the offset of your 'e' partition is large enough to accomodate any architecture-specific bootstrap requirements, and that the offset is a multiple of the hard drive's native transfer size.
1.2       wiki       49: 
                     50:  Set the fstype on your 'e' partition to RAID (4.2BSD will also work, but not if you want to use RAID_AUTOCONFIG).
                     51: 
                     52:  Here's my partition info:
                     53: 
                     54:        5 partitions:
                     55:        #        size  offset  fstype [fsize bsize cpg/sgs]
                     56:         c: 976771120    2048  unused      0     0        # (Cyl.    2*- 969020)
                     57:         d: 976773168       0  unused      0     0        # (Cyl.    0 - 969020)
                     58:         e: 976771120    2048    RAID                     # (Cyl.    2*- 969020)
                     59: 
1.5       wiki       60: 5. Now create a raid.conf file to describe the raidset.  Mine is a simple RAID-1 (mirror):
1.2       wiki       61: 
                     62:        START array
                     63:        #numrow numcol numspare
                     64:        1 2 0
                     65: 
                     66:        # Identify physical disks
                     67:        START disks
                     68:        /dev/wd2e
                     69:        /dev/wd3e
                     70: 
                     71:        # Layout is simple - 64 sectors per stripe
                     72:        START layout
                     73:        #Sect/StripeUnit StripeUnit/ParityUnit StripeUnit/ReconUnit RaidLevel
                     74:        128 1 1 1
                     75: 
                     76:        # No spares
                     77:        #START spare
                     78: 
                     79:        # Command queueing
                     80:        START queue
                     81:        fifo 100
                     82: 
1.7     ! jruoho     83: 6. Create the raidset using  [[!template id=man name="raidctl" section="8"]].  Until you do this for the first time, there are no raid "component labels" on the member drives, so you will need to use the "-C" option;  if the raidset has been previously created, you should use "-c" instead (this is what /etc/rc.d/raidframe uses at system startup).
1.2       wiki       84: 
                     85:        raidctl -C /etc/raid1.conf raid1
1.1       pgoyette   86: 
1.5       wiki       87: 7. Next, initialize the component labels with raidctl.
1.1       pgoyette   88: 
1.2       wiki       89:        raidctl -I 201105061405 raid1
                     90: 
                     91:  (For serial number, I just used the current date/time; any 64-bit integer will work.)
                     92: 
1.5       wiki       93: 8. Now use raidctl to initialize the parity of the raidset.  On my 500GB drives, this took about 1.5 hours.
1.2       wiki       94: 
                     95:        raidctl -i raid1
                     96: 
                     97:  It won't tie up your session, as the parity operation happens in the kernel in the background.  You can monitor the progress using
                     98: 
                     99:        raidctl -s raid1
                    100: 
1.6       wiki      101: 9. When parity is finished, you can treat the raidset as a normal disk. You will need to write a NetBSD disklabel (this label is separate from the labels on the physical drives).  Make certain that all the offsets are multiples of the drive's native block size; otherwise you will have unaligned transfers and performance will be very bad!  (This is also why, in step 3 above, we use "-A 2048" for fdisk rather than letting it default to 63!")
1.2       wiki      102: 
                    103:  Here is what my drive looks like:
                    104: 
                    105:        7 partitions:
                    106:        #        size    offset  fstype [fsize bsize cpg/sgs]
                    107:         c: 976770944         0  unused      0     0       # (Cyl.      0 - 953877*)
                    108:         d: 976770944         0  unused      0     0       # (Cyl.      0 - 953877*)
                    109:         e: 536870912         0  4.2BSD   2048 16384    0  # (Cyl.      0 - 524287)
                    110:         f: 104857600 536870912  4.2BSD   2048 16384    0  # (Cyl. 524288 - 626687)
                    111:         g: 335042432 641728512  4.2BSD   2048 16384    0  # (Cyl. 626688 - 953877*)
                    112: 
1.5       wiki      113: 10. Now, just as you would for a real disk, use newfs to initialize the filesystems.  Then mount them and start using your new raidset!
1.2       wiki      114: 
                    115:        newfs /dev/raid1e
                    116:        mount /dev/raid1e /mnt
                    117: 
1.5       wiki      118: 11. If your kernel includes the RAID_AUTOCONFIG option, you can use raidctl to let your system automatically reconfigure the raidset at every boot.  (If you don't do this, system startup will still take care of this, as long as your configuration file is properly named "raid[0-9].conf" or "raid[1-9][0-9].conf" and is located in /etc/ directory.)
1.2       wiki      119: 
                    120:        raidctl -A yes raid1
                    121: 
1.5       wiki      122: 12. If you're going to boot from the raidset, you probably want to have an 'a' partition.  My bootable raidset has members wd0 and wd1, which are labeled like this:
1.4       wiki      123: 
                    124:        5 partitions:
                    125:        #        size    offset     fstype [fsize bsize cpg/sgs]
                    126:         c: 488395120      2048     unused      0     0        # (Cyl.      2*- 484520)
                    127:         d: 488397168         0     unused      0     0        # (Cyl.      0 - 484520)
                    128:         e: 488395120      2048       RAID                     # (Cyl.      2*- 484520)
                    129: 
1.5       wiki      130: 13. And the raidset itself has these partitions:
1.1       pgoyette  131: 
1.2       wiki      132:        6 partitions:
                    133:        #        size    offset  fstype [fsize bsize cpg/sgs]
                    134:         a:  41943040         0  4.2BSD   2048 16384    0  # (Cyl.      0 -  40959)
                    135:         b:  62914560  41943040    swap                    # (Cyl.  40960 - 102399)
                    136:         c: 488395008         0  unused      0     0       # (Cyl.      0 - 476948*)
                    137:         d: 488395008         0  unused      0     0       # (Cyl.      0 - 476948*)
                    138:         e: 125829120 104857600  4.2BSD   2048 16384    0  # (Cyl. 102400 - 225279)
                    139:         f: 257708288 230686720  4.2BSD   2048 16384    0  # (Cyl. 225280 - 476948*)
1.1       pgoyette  140: 
1.7     ! jruoho    141: 14. You will also need to run  [[!template id=man name="installboot" section="8"]] to make the raidset bootable. Do this for each member drive.  Please note that installboot is run for the RAID partition on each member disk, and NOT on the raidn disk!  This also assumes that the RAID partition (the 'e' partition, see step 4 above) starts at the beginning of the whole-NetBSD 'c' partition.  If you don't do this, I don't think that the i386 MBR boot code will be able to find your bootstrap image.
1.1       pgoyette  142: 
1.2       wiki      143:        mount /dev/raid0a /mnt
1.3       wiki      144:        installboot /dev/wd0e /usr/mdec/bootxx_ffsv1 /mnt/boot
                    145:        installboot /dev/wd1e /usr/mdec/bootxx_ffsv1 /mnt/boot
1.1       pgoyette  146: 
1.5       wiki      147: 15. Finally, you need to make sure your kernel has the RAID_AUTOCONFIG option, and enable the raidset as the root device:
1.1       pgoyette  148: 
1.2       wiki      149:        raidctl -A root raid0

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb