1: # NetBSD RAIDframe
2:
3: ## RAIDframe Introduction
4:
5: ### About RAIDframe
6:
7: NetBSD uses the [CMU RAIDframe](http://www.pdl.cmu.edu/RAIDframe/) software fo
8: its RAID subsystem. NetBSD is the primary platform for RAIDframe development
9: RAIDframe can also be found in older versions of FreeBSD and OpenBSD. NetBS
10: also has another way of bundling disks, th
11: [ccd(4)](http://netbsd.gw.com/cgi-bin/man-cgi?ccd+4+NetBSD-5.0.1+i386) subsyste
12: (see [Concatenated Disk Device](/guide/ccd)). You should possess some [basi
13: knowledge](http://www.acnc.com/04_00.html) about RAID concepts and terminolog
14: before continuing. You should also be at least familiar with the differen
15: levels of RAID - Adaptec provides an [excellen
16: reference](http://www.adaptec.com/en-US/_common/compatibility/_education/RAID_level_compar_wp.htm)
17: and the [raid(4)](http://netbsd.gw.com/cgi-bin/man-cgi?raid+4+NetBSD-5.0.1+i386
18: manpage contains a short overview too.
19:
20: ### A warning about Data Integrity, Backups, and High Availability
21:
22: RAIDframe is a Software RAID implementation, as opposed to Hardware RAID. A
23: such, it does not need special disk controllers supported by NetBSD. Syste
24: administrators should give a great deal of consideration to whether softwar
25: RAID or hardware RAID is more appropriate for their "Mission Critical
26: applications. For some projects you might consider the use of many of th
27: hardware RAID devices [supported b
28: NetBSD](http://www.NetBSD.org/support/hardware/). It is truly at your discretio
29: what type of RAID you use, but it is recommend that you consider factors suc
30: as: manageability, commercial vendor support, load-balancing and failover, etc.
31:
32: Depending on the RAID level used, RAIDframe does provide redundancy in the even
33: of a hardware failure. However, it is *not* a replacement for reliable backups
34: Software and user-error can still cause data loss. RAIDframe may be used as
35: mechanism for facilitating backups in systems without backup hardware, but thi
36: is not an ideal configuration. Finally, with regard to "high availability", RAI
37: is only a very small component to ensuring data availability.
38:
39: Once more for good measure: *Back up your data!*
40:
41: ### Hardware versus Software RAID
42:
43: If you run a server, it will most probably already have a Hardware RAI
44: controller. There are reasons for and against using a Software RAID, dependin
45: on the scenario.
46:
47: In general, a Software RAID is well suited for low-IO system disks. If you run
48: Software RAID, you can exchange disks and disk controllers, or even move th
49: disks to a completely different machine. The computational overhead for the RAI
50: is negligible if there is only few disk IO operations.
51:
52: If you need much IO, you should use a Hardware RAID. With a Software RAID, th
53: redundancy data has to be transferred via the bus your disk controller i
54: connected to. With a Hardware RAID, you transfer data only once - the redundanc
55: computation and transfer is done by the controller.
56:
57: ### Getting Help
58:
59: If you encounter problems using RAIDframe, you have several options fo
60: obtaining help.
61:
62: 1. Read the RAIDframe man pages
63: [raid(4)](http://netbsd.gw.com/cgi-bin/man-cgi?raid+4+NetBSD-5.0.1+i386) an
64: [raidctl(8)](http://netbsd.gw.com/cgi-bin/man-cgi?raidctl+8+NetBSD-5.0.1+i386
65: thoroughly.
66:
67: 2. Search the mailing list archives. Unfortunately, there is no NetBSD lis
68: dedicated to RAIDframe support. Depending on the nature of the problem, posts
69: tend to end up in a variety of lists. At a very minimum, search
70: [netbsd-help](http://mail-index.NetBSD.org/netbsd-help/),
71: [netbsd-users@NetBSD.org](http://mail-index.NetBSD.org/netbsd-users/),
72: [current-users@NetBSD.org](http://mail-index.NetBSD.org/current-users/). Also
73: search the list for the NetBSD platform on which you are using RAIDframe:
74: port-*`${ARCH}`*@NetBSD.org.
75:
76: ### Caution
77:
78: Because RAIDframe is constantly undergoing development, some information i
79: mailing list archives has the potential of being dated and inaccurate.
80:
81: 3. Search the [Problem Repor
82: database](http://www.NetBSD.org/support/send-pr.html).
83:
84: 4. If your problem persists: Post to the mailing list most appropriat
85: (judgment call). Collect as much verbosely detailed information as possibl
86: before posting: Include you
87: [dmesg(8)](http://netbsd.gw.com/cgi-bin/man-cgi?dmesg+8+NetBSD-5.0.1+i386
88: output from `/var/run/dmesg.boot`, your kerne
89: [config(5)](http://netbsd.gw.com/cgi-bin/man-cgi?config+5+NetBSD-5.0.1+i386)
90: your `/etc/raid[0-9].conf`, any relevant errors on `/dev/console`
91: `/var/log/messages`, or to `stdout/stderr` o
92: [raidctl(8)](http://netbsd.gw.com/cgi-bin/man-cgi?raidctl+8+NetBSD-5.0.1+i386)
93: The output of **raidctl -s** (if available) will be useful as well. Als
94: include details on the troubleshooting steps you've taken thus far, exactl
95: when the problem started, and any notes on recent changes that may hav
96: prompted the problem to develop. Remember to be patient when waiting for
97: response.
98:
99: ## Setup RAIDframe Support
100:
101: The use of RAID will require software and hardware configuration changes.
102:
103: ### Kernel Support
104:
105: The GENERIC kernel already has support for RAIDframe. If you have built a custo
106: kernel for your environment the kernel configuration must have the followin
107: options:
108:
109: pseudo-device raid 8 # RAIDframe disk driver
110: options RAID_AUTOCONFIG # auto-configuration of RAID components
111:
112: The RAID support must be detected by the NetBSD kernel, which can be checked b
113: looking at the output of th
114: [dmesg(8)](http://netbsd.gw.com/cgi-bin/man-cgi?dmesg+8+NetBSD-5.0.1+i386
115: command.
116:
117: # dmesg|grep -i raid
118: Kernelized RAIDframe activated
119:
120: Historically, the kernel must also contain static mappings between bus addresse
121: and device nodes in `/dev`. This used to ensure consistency of devices withi
122: RAID sets in the event of a device failure after reboot. Since NetBSD 1.6
123: however, using the auto-configuration features of RAIDframe has been recommende
124: over statically mapping devices. The auto-configuration features allow drives t
125: move around on the system, and RAIDframe will automatically determine whic
126: components belong to which RAID sets.
127:
128: ### Power Redundancy and Disk Caching
129:
130: If your system has an Uninterruptible Power Supply (UPS), if your system ha
131: redundant power supplies, or your disk controller has a battery, you shoul
132: consider enabling the read and write caches on your drives. On systems wit
133: redundant power, this will improve drive performance. On systems withou
134: redundant power, the write cache could endanger the integrity of RAID data i
135: the event of a power loss.
136:
137: The [dkctl(8)](http://netbsd.gw.com/cgi-bin/man-cgi?dkctl+8+NetBSD-5.0.1+i386
138: utility to can be used for this on all kinds of disks that support the operatio
139: (SCSI, EIDE, SATA, ...):
140:
141: # dkctl wd0 getcache
142: /dev/rwd0d: read cache enabled
143: /dev/rwd0d: read cache enable is not changeable
144: /dev/rwd0d: write cache enable is changeable
145: /dev/rwd0d: cache parameters are not savable
146: # dkctl wd0 setcache rw
147: # dkctl wd0 getcache
148: /dev/rwd0d: read cache enabled
149: /dev/rwd0d: write-back cache enabled
150: /dev/rwd0d: read cache enable is not changeable
151: /dev/rwd0d: write cache enable is changeable
152: /dev/rwd0d: cache parameters are not savable
153:
154: ## Example: RAID-1 Root Disk
155:
156: This example explains how to setup RAID-1 root disk. With RAID-1 components ar
157: mirrored and therefore the server can be fully functional in the event of
158: single component failure. The goal is to provide a level of redundancy that wil
159: allow the system to encounter a component failure on either component disk i
160: the RAID and:
161:
162: * Continue normal operations until a maintenance window can be scheduled.
163: * Or, in the unlikely event that the component failure causes a system reboot
164: be able to quickly reconfigure the system to boot from the remaining
165: component (platform dependent).
166:
167: 
168:
169: **RAID-1 Disk Logical Layout**
170:
171: Because RAID-1 provides both redundancy and performance improvements, its mos
172: practical application is on critical "system" partitions such as `/`, `/usr`
173: `/var`, `swap`, etc., where read operations are more frequent than writ
174: operations. For other file systems, such as `/home` or `/var/`, other RAI
175: levels might be considered (see the references above). If one were simpl
176: creating a generic RAID-1 volume for a non-root file system, the cookie-cutte
177: examples from the man page could be followed, but because the root volume mus
178: be bootable, certain special steps must be taken during initial setup.
179:
180: *Note*: This example will outline a process that differs only slightly betwee
181: the i386 and sparc64 platforms. In an attempt to reduce excessive duplication o
182: content, where differences do exist and are cosmetic in nature, they will b
183: pointed out using a section such as this. If the process is drasticall
184: different, the process will branch into separate, platform dependent steps.
185:
186: ### Pseudo-Process Outline
187:
188: Although a much more refined process could be developed using a custom copy o
189: NetBSD installed on custom-developed removable media, presently the NetBS
190: install media lacks RAIDframe tools and support, so the following pseudo process
191: has become the de facto standard for setting up RAID-1 Root.
192:
193: 1. Install a stock NetBSD onto Disk0 of your system.
194:
195: 
196:
197: **Perform generic install onto Disk0/wd0**
198:
199: 2. Use the installed system on Disk0/wd0 to setup a RAID Set composed o
200: Disk1/wd1 only.
201:
202: 
203: **Setup RAID Set**
204:
205: 3. Reboot the system off the Disk1/wd1 with the newly created RAID volume.
206:
207: 
208:
209: **Reboot using Disk1/wd1 of RAID**
210:
211: 4. Add / re-sync Disk0/wd0 back into the RAID set.
212:
213: 
214:
215: **Mirror Disk1/wd1 back to Disk0/wd0**
216:
217: ### Hardware Review
218:
219: At present, the alpha, amd64, i386, pmax, sparc, sparc64, and vax NetBS
220: platforms support booting from RAID-1. Booting is not supported from any othe
221: RAID level. Booting from a RAID set is accomplished by teaching the 1st stag
222: boot loader to understand both 4.2BSD/FFS and RAID partitions. The 1st boo
223: block code only needs to know enough about the disk partitions and file system
224: to be able to read the 2nd stage boot blocks. Therefore, at any time, th
225: system's BIOS / firmware must be able to read a drive with 1st stage boot block
226: installed. On the i386 platform, configuring this is entirely dependent on th
227: vendor of the controller card / host bus adapter to which your disks ar
228: connected. On sparc64 this is controlled by the IEEE 1275 Sun OpenBoot Firmware.
229:
230: This article assumes two identical IDE disks (`/dev/wd{0,1}`) which we are goin
231: to mirror (RAID-1). These disks are identified as:
232:
233: # grep ^wd /var/run/dmesg.boot
234: wd0 at atabus0 drive 0: <WDC WD100BB-75CLB0>
235: wd0: drive supports 16-sector PIO transfers, LBA addressing
236: wd0: 9541 MB, 19386 cyl, 16 head, 63 sec, 512 bytes/sect x 19541088 sectors
237: wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
238: wd0(piixide0:0:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA data transfers)
239:
240: wd1 at atabus1 drive 0: <WDC WD100BB-75CLB0>
241: wd1: drive supports 16-sector PIO transfers, LBA addressing
242: wd1: 9541 MB, 19386 cyl, 16 head, 63 sec, 512 bytes/sect x 19541088 sectors
243: wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
244: wd1(piixide0:1:0): using PIO mode 4, Ultra-DMA mode 2 (Ultra/33) (using DMA data transfers)
245:
246: *Note*: If you are using SCSI, replace `/dev/[r]wd{0,1}` with `/dev/[r]sd{0,1}`.
247:
248: In this example, both disks are jumpered as Master on separate channels on th
249: same controller. You usually wouldn't want to have both disks on the same bus o
250: the same controller; this creates a single point of failure. Ideally you woul
251: have the disks on separate channels on separate controllers. Nonetheless, i
252: most cases the most critical point is the hard disk, so having redundan
253: channels or controllers is not that important. Plus, having more channels o
254: controllers increases costs. Some SCSI controllers have multiple channels on th
255: same controller, however, a SCSI bus reset on one channel could adversely affec
256: the other channel if the ASIC/IC becomes overloaded. The trade-off with tw
257: controllers is that twice the bandwidth is used on the system bus. For purpose
258: of simplification, this example shows two disks on different channels on th
259: same controller.
260:
261: *Note*: RAIDframe requires that all components be of the same size. Actually, i
262: will use the lowest common denominator among components of dissimilar sizes. Fo
263: purposes of illustration, the example uses two disks of identical geometries
264: Also, consider the availability of replacement disks if a component suffers
265: critical hardware failure.
266:
267: *Tip*: Two disks of identical vendor model numbers could have differen
268: geometries if the drive possesses "grown defects". Use a low-level program t
269: examine the grown defects table of the disk. These disks are obviousl
270: suboptimal candidates for use in RAID and should be avoided.
271:
272: ### Initial Install on Disk0/wd0
273:
274: Perform a very generic installation onto your Disk0/wd0. Follow the `INSTALL
275: instructions for your platform. Install all the sets but do not bothe
276: customizing anything other than the kernel as it will be overwritten.
277:
278: *Tip*: On i386, during the sysinst install, when prompted if you want to `us
279: the entire disk for NetBSD`, answer `yes`.
280:
281: * [Installing NetBSD: Preliminary considerations and preparations](/guide/inst)
282: * [NetBSD/i386 Install Directions](http://ftp.NetBSD.org/pub/NetBSD/NetBSD-5.0.2/i386/INSTALL.html)
283: * [NetBSD/sparc64 Install Directions](http://ftp.NetBSD.org/pub/NetBSD/NetBSD-5.0.2/sparc64/INSTALL.html)
284:
285: Once the installation is complete, you should examine th
286: [disklabel(8)](http://netbsd.gw.com/cgi-bin/man-cgi?disklabel+8+NetBSD-5.0.1+i386
287: and [fdisk(8)](http://netbsd.gw.com/cgi-bin/man-cgi?fdisk+8+NetBSD-5.0.1+i386)
288: [sunlabel(8)](http://netbsd.gw.com/cgi-bin/man-cgi?sunlabel+8+NetBSD-5.0.1+i386
289: outputs on the system:
290:
291: # df
292: Filesystem 1K-blocks Used Avail %Cap Mounted on
293: /dev/wd0a 9487886 502132 8511360 5% /
294:
295: On i386:
296:
297: # disklabel -r wd0
298: type: unknown
299: disk: Disk00
300: label:
301: flags:
302: bytes/sector: 512
303: sectors/track: 63
304: tracks/cylinder: 16
305: sectors/cylinder: 1008
306: cylinders: 19386
307: total sectors: 19541088
308: rpm: 3600
309: interleave: 1
310: trackskew: 0
311: cylinderskew: 0
312: headswitch: 0 # microseconds
313: track-to-track seek: 0 # microseconds
314: drivedata: 0
315:
316: 16 partitions:
317: # size offset fstype [fsize bsize cpg/sgs]
318: a: 19276992 63 4.2BSD 1024 8192 46568 # (Cyl. 0* - 19124*)
319: b: 264033 19277055 swap # (Cyl. 19124* - 19385)
320: c: 19541025 63 unused 0 0 # (Cyl. 0* - 19385)
321: d: 19541088 0 unused 0 0 # (Cyl. 0 - 19385)
322:
323: # fdisk /dev/rwd0d
324: Disk: /dev/rwd0d
325: NetBSD disklabel disk geometry:
326: cylinders: 19386, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
327: total sectors: 19541088
328:
329: BIOS disk geometry:
330: cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
331: total sectors: 19541088
332:
333: Partition table:
334: 0: NetBSD (sysid 169)
335: start 63, size 19541025 (9542 MB, Cyls 0-1216/96/1), Active
336: 1: <UNUSED>
337: 2: <UNUSED>
338: 3: <UNUSED>
339: Bootselector disabled.
340: First active partition: 0
341:
342: On Sparc64 the command and output differ slightly:
343:
344: # disklabel -r wd0
345: type: unknown
346: disk: Disk0
347: [...snip...]
348: 8 partitions:
349: # size offset fstype [fsize bsize cpg/sgs]
350: a: 19278000 0 4.2BSD 1024 8192 46568 # (Cyl. 0 - 19124)
351: b: 263088 19278000 swap # (Cyl. 19125 - 19385)
352: c: 19541088 0 unused 0 0 # (Cyl. 0 - 19385)
353:
354: # sunlabel /dev/rwd0c
355: sunlabel> P
356: a: start cyl = 0, size = 19278000 (19125/0/0 - 9413.09Mb)
357: b: start cyl = 19125, size = 263088 (261/0/0 - 128.461Mb)
358: c: start cyl = 0, size = 19541088 (19386/0/0 - 9541.55Mb)
359:
360: ### Preparing Disk1/wd1
361:
362: Once you have a stock install of NetBSD on Disk0/wd0, you are ready to begin
363: Disk1/wd1 will be visible and unused by the system. To setup Disk1/wd1, you wil
364: us
365: [disklabel(8)](http://netbsd.gw.com/cgi-bin/man-cgi?disklabel+8+NetBSD-5.0.1+i386
366: to allocate the entire second disk to the RAID-1 set.
367:
368: *Tip*: The best way to ensure that Disk1/wd1 is completely empty is to 'zero
369: out the first few sectors of the disk wit
370: [dd(1)](http://netbsd.gw.com/cgi-bin/man-cgi?dd+1+NetBSD-5.0.1+i386) . This wil
371: erase the MBR (i386) or Sun disk label (sparc64), as well as the NetBSD dis
372: label. If you make a mistake at any point during the RAID setup process, you ca
373: always refer to this process to restore the disk to an empty state.
374:
375: *Note*: On sparc64, use `/dev/rwd1c` instead of `/dev/rwd1d`!
376:
377: # dd if=/dev/zero of=/dev/rwd1d bs=8k count=1
378: 1+0 records in
379: 1+0 records out
380: 8192 bytes transferred in 0.003 secs (2730666 bytes/sec)
381:
382: Once this is complete, on i386, verify that both the MBR and NetBSD disk label
383: are gone. On sparc64, verify that the Sun Disk label is gone as well.
384:
385: On i386:
386:
387: # fdisk /dev/rwd1d
388:
389: fdisk: primary partition table invalid, no magic in sector 0
390: Disk: /dev/rwd1d
391: NetBSD disklabel disk geometry:
392: cylinders: 19386, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
393: total sectors: 19541088
394:
395: BIOS disk geometry:
396: cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
397: total sectors: 19541088
398:
399: Partition table:
400: 0: <UNUSED>
401: 1: <UNUSED>
402: 2: <UNUSED>
403: 3: <UNUSED>
404: Bootselector disabled.
405:
406: # disklabel -r wd1
407:
408: [...snip...]
409: 16 partitions:
410: # size offset fstype [fsize bsize cpg/sgs]
411: c: 19541025 63 unused 0 0 # (Cyl. 0* - 19385)
412: d: 19541088 0 unused 0 0 # (Cyl. 0 - 19385)
413:
414: On sparc64:
415:
416: # sunlabel /dev/rwd1c
417:
418: sunlabel: bogus label on `/dev/wd1c' (bad magic number)
419:
420: # disklabel -r wd1
421:
422: [...snip...]
423: 3 partitions:
424: # size offset fstype [fsize bsize cpg/sgs]
425: c: 19541088 0 unused 0 0 # (Cyl. 0 - 19385)
426: disklabel: boot block size 0
427: disklabel: super block size 0
428:
429: Now that you are certain the second disk is empty, on i386 you must establis
430: the MBR on the second disk using the values obtained from Disk0/wd0 above. W
431: must remember to mark the NetBSD partition active or the system will not boot
432: You must also create a NetBSD disklabel on Disk1/wd1 that will enable a RAI
433: volume to exist upon it. On sparc64, you will need to simpl
434: [disklabel(8)](http://netbsd.gw.com/cgi-bin/man-cgi?disklabel+8+NetBSD-5.0.1+i386
435: the second disk which will write the proper Sun Disk Label.
436:
437: *Tip*
438: [disklabel(8)](http://netbsd.gw.com/cgi-bin/man-cgi?disklabel+8+NetBSD-5.0.1+i386
439: will use your shell' s environment variable `$EDITOR` variable to edit th
440: disklabel. The default i
441: [vi(1)](http://netbsd.gw.com/cgi-bin/man-cgi?vi+1+NetBSD-5.0.1+i386)
442:
443: On i386:
444:
445: # fdisk -0ua /dev/rwd1d
446: fdisk: primary partition table invalid, no magic in sector 0
447: Disk: /dev/rwd1d
448: NetBSD disklabel disk geometry:
449: cylinders: 19386, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
450: total sectors: 19541088
451:
452: BIOS disk geometry:
453: cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
454: total sectors: 19541088
455:
456: Do you want to change our idea of what BIOS thinks? [n]
457:
458: Partition 0:
459: <UNUSED>
460: The data for partition 0 is:
461: <UNUSED>
462: sysid: [0..255 default: 169]
463: start: [0..1216cyl default: 63, 0cyl, 0MB]
464: size: [0..1216cyl default: 19541025, 1216cyl, 9542MB]
465: bootmenu: []
466: Do you want to change the active partition? [n] y
467: Choosing 4 will make no partition active.
468: active partition: [0..4 default: 0] 0
469: Are you happy with this choice? [n] y
470:
471: We haven't written the MBR back to disk yet. This is your last chance.
472: Partition table:
473: 0: NetBSD (sysid 169)
474: start 63, size 19541025 (9542 MB, Cyls 0-1216/96/1), Active
475: 1: <UNUSED>
476: 2: <UNUSED>
477: 3: <UNUSED>
478: Bootselector disabled.
479: Should we write new partition table? [n] y
480:
481: # disklabel -r -e -I wd1
482: type: unknown
483: disk: Disk1
484: label:
485: flags:
486: bytes/sector: 512
487: sectors/track: 63
488: tracks/cylinder: 16
489: sectors/cylinder: 1008
490: cylinders: 19386
491: total sectors: 19541088
492: [...snip...]
493: 16 partitions:
494: # size offset fstype [fsize bsize cpg/sgs]
495: a: 19541025 63 RAID # (Cyl. 0*-19385)
496: c: 19541025 63 unused 0 0 # (Cyl. 0*-19385)
497: d: 19541088 0 unused 0 0 # (Cyl. 0 -19385)
498:
499: On sparc64:
500:
501: # disklabel -r -e -I wd1
502: type: unknown
503: disk: Disk1
504: label:
505: flags:
506: bytes/sector: 512
507: sectors/track: 63
508: tracks/cylinder: 16
509: sectors/cylinder: 1008
510: cylinders: 19386
511: total sectors: 19541088
512: [...snip...]
513: 3 partitions:
514: # size offset fstype [fsize bsize cpg/sgs]
515: a: 19541088 0 RAID # (Cyl. 0 - 19385)
516: c: 19541088 0 unused 0 0 # (Cyl. 0 - 19385)
517:
518: # sunlabel /dev/rwd1
519: sunlabel> P
520: a: start cyl = 0, size = 19541088 (19386/0/0 - 9541.55Mb)
521: c: start cyl = 0, size = 19541088 (19386/0/0 - 9541.55Mb)
522:
523: *Note*: On i386, the `c:` and `d:` slices are reserved. `c:` represents th
524: NetBSD portion of the disk. `d:` represents the entire disk. Because we want t
525: allocate the entire NetBSD MBR partition to RAID, and because `a:` reside
526: within the bounds of `c:`, the `a:` and `c:` slices have same size and offse
527: values and sizes. The offset must start at a track boundary (an increment o
528: sectors matching the sectors/track value in the disk label). On sparc64 however
529: `c:` represents the entire NetBSD partition in the Sun disk label and `d:` i
530: not reserved. Also note that sparc64's `c:` and `a:` require no offset from th
531: beginning of the disk, however if they should need to be, the offset must star
532: at a cylinder boundary (an increment of sectors matching the sectors/cylinde
533: value).
534:
535: ### Initializing the RAID Device
536:
537: Next we create the configuration file for the RAID set / volume. Traditionally
538: RAIDframe configuration files belong in `/etc` and would be read and initialize
539: at boot time, however, because we are creating a bootable RAID volume, th
540: configuration data will actually be written into the RAID volume using th
541: *auto-configure* feature. Therefore, files are needed only during the initia
542: setup and should not reside in `/etc`.
543:
544: # vi /var/tmp/raid0.conf
545: START array
546: 1 2 0
547:
548: START disks
549: absent
550: /dev/wd1a
551:
552: START layout
553: 128 1 1 1
554:
555: START queue
556: fifo 100
557:
558: Note that `absent` means a non-existing disk. This will allow us to establis
559: the RAID volume with a bogus component that we will substitute for Disk0/wd0 a
560: a later time.
561:
562: Next we configure the RAID device and initialize the serial number to somethin
563: unique. In this example we use a "YYYYMMDD*`Revision`*" scheme. The format yo
564: choose is entirely at your discretion, however the scheme you choose shoul
565: ensure that no two RAID sets use the same serial number at the same time.
566:
567: After that we initialize the RAID set for the first time, safely ignoring th
568: errors regarding the bogus component.
569:
570: # raidctl -v -C /var/tmp/raid0.conf raid0
571: Ignoring missing component at column 0
572: raid0: Component absent being configured at col: 0
573: Column: 0 Num Columns: 0
574: Version: 0 Serial Number: 0 Mod Counter: 0
575: Clean: No Status: 0
576: Number of columns do not match for: absent
577: absent is not clean!
578: raid0: Component /dev/wd1a being configured at col: 1
579: Column: 0 Num Columns: 0
580: Version: 0 Serial Number: 0 Mod Counter: 0
581: Clean: No Status: 0
582: Column out of alignment for: /dev/wd1a
583: Number of columns do not match for: /dev/wd1a
584: /dev/wd1a is not clean!
585: raid0: There were fatal errors
586: raid0: Fatal errors being ignored.
587: raid0: RAID Level 1
588: raid0: Components: component0[**FAILED**] /dev/wd1a
589: raid0: Total Sectors: 19540864 (9541 MB)
590: # raidctl -v -I 2009122601 raid0
591: # raidctl -v -i raid0
592: Initiating re-write of parity
593: raid0: Error re-writing parity!
594: Parity Re-write status:
595:
596: # tail -1 /var/log/messages
597: Dec 26 00:00:30 /netbsd: raid0: Error re-writing parity!
598: # raidctl -v -s raid0
599: Components:
600: component0: failed
601: /dev/wd1a: optimal
602: No spares.
603: component0 status is: failed. Skipping label.
604: Component label for /dev/wd1a:
605: Row: 0, Column: 1, Num Rows: 1, Num Columns: 2
606: Version: 2, Serial Number: 2009122601, Mod Counter: 7
607: Clean: No, Status: 0
608: sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
609: Queue size: 100, blocksize: 512, numBlocks: 19540864
610: RAID Level: 1
611: Autoconfig: No
612: Root partition: No
613: Last configured as: raid0
614: Parity status: DIRTY
615: Reconstruction is 100% complete.
616: Parity Re-write is 100% complete.
617: Copyback is 100% complete.
618:
619: ### Setting up Filesystems
620:
621: *Caution*: The root filesystem must begin at sector 0 of the RAID device. I
622: not, the primary boot loader will be unable to find the secondary boot loader.
623:
624: The RAID device is now configured and available. The RAID device is a pseud
625: disk-device. It will be created with a default disk label. You must no
626: determine the proper sizes for disklabel slices for your production environment
627: For purposes of simplification in this example, our system will have 8.
628: gigabytes dedicated to `/` as `/dev/raid0a` and the rest allocated to `swap
629: as `/dev/raid0b`.
630:
631: *Caution*: This is an unrealistic disk layout for a production server; th
632: NetBSD Guide can expand on proper partitioning technique. See [Installin
633: NetBSD: Preliminary considerations and preparations*](inst).
634:
635: *Note*: Note that 1 GB is 2\*1024\*1024=2097152 blocks (1 block is 512 bytes, o
636: 0.5 kilobytes). Despite what the underlying hardware composing a RAID set is
637: the RAID pseudo disk will always have 512 bytes/sector.
638:
639: *Note*: In our example, the space allocated to the underlying `a:` slic
640: composing the RAID set differed between i386 and sparc64, therefore the tota
641: sectors of the RAID volumes differs:
642:
643: On i386:
644:
645: # disklabel -r -e -I raid0
646: type: RAID
647: disk: raid
648: label: fictitious
649: flags:
650: bytes/sector: 512
651: sectors/track: 128
652: tracks/cylinder: 8
653: sectors/cylinder: 1024
654: cylinders: 19082
655: total sectors: 19540864
656: rpm: 3600
657: interleave: 1
658: trackskew: 0
659: cylinderskew: 0
660: headswitch: 0 # microseconds
661: track-to-track seek: 0 # microseconds
662: drivedata: 0
663:
664: # size offset fstype [fsize bsize cpg/sgs]
665: a: 19015680 0 4.2BSD 0 0 0 # (Cyl. 0 - 18569)
666: b: 525184 19015680 swap # (Cyl. 18570 - 19082*)
667: d: 19540864 0 unused 0 0 # (Cyl. 0 - 19082*)
668:
669: On sparc64:
670:
671: # disklabel -r -e -I raid0
672: [...snip...]
673: total sectors: 19539968
674: [...snip...]
675: 3 partitions:
676: # size offset fstype [fsize bsize cpg/sgs]
677: a: 19251200 0 4.2BSD 0 0 0 # (Cyl. 0 - 18799)
678: b: 288768 19251200 swap # (Cyl. 18800 - 19081)
679: c: 19539968 0 unused 0 0 # (Cyl. 0 - 19081)
680:
681: Next, format the newly created `/` partition as a 4.2BSD FFSv1 File System:
682:
683: # newfs -O 1 /dev/rraid0a
684: /dev/rraid0a: 9285.0MB (19015680 sectors) block size 16384, fragment size 2048
685: using 51 cylinder groups of 182.06MB, 11652 blks, 23040 inodes.
686: super-block backups (for fsck -b #) at:
687: 32, 372896, 745760, 1118624, 1491488, 1864352, 2237216, 2610080, 2982944,
688: ...............................................................................
689:
690: # fsck -fy /dev/rraid0a
691: ** /dev/rraid0a
692: ** File system is already clean
693: ** Last Mounted on
694: ** Phase 1 - Check Blocks and Sizes
695: ** Phase 2 - Check Pathnames
696: ** Phase 3 - Check Connectivity
697: ** Phase 4 - Check Reference Counts
698: ** Phase 5 - Check Cyl groups
699: 1 files, 1 used, 4679654 free (14 frags, 584955 blocks, 0.0% fragmentation)
700:
701: ### Migrating System to RAID
702:
703: The new RAID filesystems are now ready for use. We mount them under `/mnt` an
704: copy all files from the old system. This can be done usin
705: [dump(8)](http://netbsd.gw.com/cgi-bin/man-cgi?dump+8+NetBSD-5.0.1+i386) o
706: [pax(1)](http://netbsd.gw.com/cgi-bin/man-cgi?pax+1+NetBSD-5.0.1+i386).
707:
708: # mount /dev/raid0a /mnt
709: # df -h /mnt
710: Filesystem Size Used Avail %Cap Mounted on
711: /dev/raid0a 8.9G 2.0K 8.5G 0% /mnt
712: # cd /; pax -v -X -rw -pe . /mnt
713: [...snip...]
714:
715: The NetBSD install now exists on the RAID filesystem. We need to fix th
716: mount-points in the new copy of `/etc/fstab` or the system will not come u
717: correctly. Replace instances of `wd0` with `raid0`.
718:
719: The swap should be unconfigured upon shutdown to avoid parity errors on the RAI
720: device. This can be done with a simple, one-line setting in `/etc/rc.conf`.
721:
722: # vi /mnt/etc/rc.conf
723: swapoff=YES
724:
725: Next, the boot loader must be installed on Disk1/wd1. Failure to install th
726: loader on Disk1/wd1 will render the system un-bootable if Disk0/wd0 fails. You
727: should hope your system won't have to reboot when wd0 fails.
728:
729: *Tip*: Because the BIOS/CMOS menus in many i386 based systems are misleadin
730: with regard to device boot order. I highly recommend utilizing the `-
731: timeout=X` option supported by the i386 1st stage boot loader. Setup uniqu
732: values for each disk as a point of reference so that you can easily determin
733: from which disk the system is booting.
734:
735: *Caution*: Although it may seem logical to install the 1st stage boot block int
736: `/dev/rwd1{c,d}` (which is historically correct with NetBSD 1.6.
737: [installboot(8)](http://netbsd.gw.com/cgi-bin/man-cgi?installboot+8+NetBSD-5.0.1+i386
738: , this is no longer the case. If you make this mistake, the boot sector wil
739: become irrecoverably damaged and you will need to start the process over again.
740:
741: On i386, install the boot loader into `/dev/rwd1a`:
742:
743: # /usr/sbin/installboot -o timeout=30 -v /dev/rwd1a /usr/mdec/bootxx_ffsv1
744: File system: /dev/rwd1a
745: Primary bootstrap: /usr/mdec/bootxx_ffsv1
746: Ignoring PBR with invalid magic in sector 0 of `/dev/rwd1a'
747: Boot options: timeout 30, flags 0, speed 9600, ioaddr 0, console pc
748:
749: On sparc64, install the boot loader into `/dev/rwd1a` as well, however the `-o
750: flag is unsupported (and un-needed thanks to OpenBoot):
751:
752: # /usr/sbin/installboot -v /dev/rwd1a /usr/mdec/bootblk
753: File system: /dev/rwd1a
754: Primary bootstrap: /usr/mdec/bootblk
755: Bootstrap start sector: 1
756: Bootstrap byte count: 5140
757: Writing bootstrap
758:
759: Finally the RAID set must be made auto-configurable and the system should b
760: rebooted. After the reboot everything is mounted from the RAID devices.
761:
762: # raidctl -v -A root raid0
763: raid0: Autoconfigure: Yes
764: raid0: Root: Yes
765: # tail -2 /var/log/messages
766: raid0: New autoconfig value is: 1
767: raid0: New rootpartition value is: 1
768: # raidctl -v -s raid0
769: [...snip...]
770: Autoconfig: Yes
771: Root partition: Yes
772: Last configured as: raid0
773: [...snip...]
774: # shutdown -r now
775:
776: ### Warning
777:
778: Always us
779: [shutdown(8)](http://netbsd.gw.com/cgi-bin/man-cgi?shutdown+8+NetBSD-5.0.1+i386
780: when shutting down. Never simply us
781: [reboot(8)](http://netbsd.gw.com/cgi-bin/man-cgi?reboot+8+NetBSD-5.0.1+i386)
782: [reboot(8)](http://netbsd.gw.com/cgi-bin/man-cgi?reboot+8+NetBSD-5.0.1+i386
783: will not properly run shutdown RC scripts and will not safely disable swap. Thi
784: will cause dirty parity at every reboot.
785:
786: ### The first boot with RAID
787:
788: At this point, temporarily configure your system to boot Disk1/wd1. See notes i
789: [[Testing Boot Blocks|guide/rf#adding-text-boot]] for details on this process
790: The system should boot now and all filesystems should be on the RAID devices
791: The RAID will be functional with a single component, however the set is no
792: fully functional because the bogus drive (wd9) has failed.
793:
794: # egrep -i "raid|root" /var/run/dmesg.boot
795: raid0: RAID Level 1
796: raid0: Components: component0[**FAILED**] /dev/wd1a
797: raid0: Total Sectors: 19540864 (9541 MB)
798: boot device: raid0
799: root on raid0a dumps on raid0b
800: root file system type: ffs
801:
802: # df -h
803: Filesystem Size Used Avail Capacity Mounted on
804: /dev/raid0a 8.9G 196M 8.3G 2% /
805: kernfs 1.0K 1.0K 0B 100% /kern
806:
807: # swapctl -l
808: Device 1K-blocks Used Avail Capacity Priority
809: /dev/raid0b 262592 0 262592 0% 0
810: # raidctl -s raid0
811: Components:
812: component0: failed
813: /dev/wd1a: optimal
814: No spares.
815: component0 status is: failed. Skipping label.
816: Component label for /dev/wd1a:
817: Row: 0, Column: 1, Num Rows: 1, Num Columns: 2
818: Version: 2, Serial Number: 2009122601, Mod Counter: 65
819: Clean: No, Status: 0
820: sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
821: Queue size: 100, blocksize: 512, numBlocks: 19540864
822: RAID Level: 1
823: Autoconfig: Yes
824: Root partition: Yes
825: Last configured as: raid0
826: Parity status: DIRTY
827: Reconstruction is 100% complete.
828: Parity Re-write is 100% complete.
829: Copyback is 100% complete.
830:
831: ### Adding Disk0/wd0 to RAID
832:
833: We will now add Disk0/wd0 as a component of the RAID. This will destroy th
834: original file system structure. On i386, the MBR disklabel will be unaffecte
835: (remember we copied wd0's label to wd1 anyway) , therefore there is no need t
836: "zero" Disk0/wd0. However, we need to relabel Disk0/wd0 to have an identica
837: NetBSD disklabel layout as Disk1/wd1. Then we add Disk0/wd0 as "hot-spare" t
838: the RAID set and initiate the parity reconstruction for all RAID devices
839: effectively bringing Disk0/wd0 into the RAID-1 set and "syncing up" both disks.
840:
841: # disklabel -r wd1 > /tmp/disklabel.wd1
842: # disklabel -R -r wd0 /tmp/disklabel.wd1
843:
844: As a last-minute sanity check, you might want to us
845: [diff(1)](http://netbsd.gw.com/cgi-bin/man-cgi?diff+1+NetBSD-5.0.1+i386) t
846: ensure that the disklabels of Disk0/wd0 match Disk1/wd1. You should also backu
847: these files for reference in the event of an emergency.
848:
849: # disklabel -r wd0 > /tmp/disklabel.wd0
850: # disklabel -r wd1 > /tmp/disklabel.wd1
851: # diff /tmp/disklabel.wd0 /tmp/disklabel.wd1
852: # fdisk /dev/rwd0 > /tmp/fdisk.wd0
853: # fdisk /dev/rwd1 > /tmp/fdisk.wd1
854: # diff /tmp/fdisk.wd0 /tmp/fdisk.wd1
855: # mkdir /root/RFbackup
856: # cp -p /tmp/{disklabel,fdisk}* /root/RFbackup
857:
858: Once you are sure, add Disk0/wd0 as a spare component, and start reconstruction:
859:
860: # raidctl -v -a /dev/wd0a raid0
861: /netbsd: Warning: truncating spare disk /dev/wd0a to 241254528 blocks
862: # raidctl -v -s raid0
863: Components:
864: component0: failed
865: /dev/wd1a: optimal
866: Spares:
867: /dev/wd0a: spare
868: [...snip...]
869: # raidctl -F component0 raid0
870: RECON: initiating reconstruction on col 0 -> spare at col 2
871: 11% |**** | ETA: 04:26 \
872:
873: Depending on the speed of your hardware, the reconstruction time will vary. Yo
874: may wish to watch it on another terminal (note that you can interrupt
875: `raidctl -S` any time without stopping the synchronisation):
876:
877: # raidctl -S raid0
878: Reconstruction is 0% complete.
879: Parity Re-write is 100% complete.
880: Copyback is 100% complete.
881: Reconstruction status:
882: 17% |****** | ETA: 03:08 -
883:
884: After reconstruction, both disks should be *optimal*.
885:
886: # tail -f /var/log/messages
887: raid0: Reconstruction of disk at col 0 completed
888: raid0: Recon time was 1290.625033 seconds, accumulated XOR time was 0 us (0.000000)
889: raid0: (start time 1093407069 sec 145393 usec, end time 1093408359 sec 770426 usec)
890: raid0: Total head-sep stall count was 0
891: raid0: 305318 recon event waits, 1 recon delays
892: raid0: 1093407069060000 max exec ticks
893:
894: # raidctl -v -s raid0
895: Components:
896: component0: spared
897: /dev/wd1a: optimal
898: Spares:
899: /dev/wd0a: used_spare
900: [...snip...]
901:
902: When the reconstruction is finished we need to install the boot loader on th
903: Disk0/wd0. On i386, install the boot loader into `/dev/rwd0a`:
904:
905: # /usr/sbin/installboot -o timeout=15 -v /dev/rwd0a /usr/mdec/bootxx_ffsv1
906: File system: /dev/rwd0a
907: Primary bootstrap: /usr/mdec/bootxx_ffsv1
908: Boot options: timeout 15, flags 0, speed 9600, ioaddr 0, console pc
909:
910: On sparc64:
911:
912: # /usr/sbin/installboot -v /dev/rwd0a /usr/mdec/bootblk
913: File system: /dev/rwd0a
914: Primary bootstrap: /usr/mdec/bootblk
915: Bootstrap start sector: 1
916: Bootstrap byte count: 5140
917: Writing bootstrap
918:
919: And finally, reboot the machine one last time before proceeding. This i
920: required to migrate Disk0/wd0 from status "used\_spare" as "Component0" to stat
921: "optimal". Refer to notes in the next section regarding verification of clea
922: parity after each reboot.
923:
924: # shutdown -r now
925:
926: ### Testing Boot Blocks
927:
928: At this point, you need to ensure that your system's hardware can properly boo
929: using the boot blocks on either disk. On i386, this is a hardware-dependen
930: process that may be done via your motherboard CMOS/BIOS menu or your controlle
931: card's configuration menu.
932:
933: On i386, use the menu system on your machine to set the boot device order
934: priority to Disk1/wd1 before Disk0/wd0. The examples here depict a generic Awar
935: BIOS.
936:
937: 
938:
939: **Award BIOS i386 Boot Disk1/wd1**
940:
941: Save changes and exit:
942:
943: >> NetBSD/i386 BIOS Boot, Revision 5.2 (from NetBSD 5.0.2)
944: >> (builds@b7, Sun Feb 7 00:30:50 UTC 2010)
945: >> Memory: 639/130048 k
946: Press return to boot now, any other key for boot menu
947: booting hd0a:netbsd - starting in 30
948:
949: You can determine that the BIOS is reading Disk1/wd1 because the timeout of th
950: boot loader is 30 seconds instead of 15. After the reboot, re-enter the BIOS an
951: configure the drive boot order back to the default:
952:
953: 
954:
955: **Award BIOS i386 Boot Disk0/wd0**
956:
957: Save changes and exit:
958:
959: >> NetBSD/i386 BIOS Boot, Revision 5.2 (from NetBSD 5.0.2)
960: >> Memory: 639/130048 k
961: Press return to boot now, any other key for boot menu
962: booting hd0a:netbsd - starting in 15
963:
964: Notice how your custom kernel detects controller/bus/drive assignment
965: independent of what the BIOS assigns as the boot disk. This is the expecte
966: behavior.
967:
968: On sparc64, use the Sun OpenBoot **devalias** to confirm that both disks are bootable:
969:
970: Sun Ultra 5/10 UPA/PCI (UltraSPARC-IIi 400MHz), No Keyboard
971: OpenBoot 3.15, 128 MB memory installed, Serial #nnnnnnnn.
972: Ethernet address 8:0:20:a5:d1:3b, Host ID: nnnnnnnn.
973:
974: ok devalias
975: [...snip...]
976: cdrom /pci@1f,0/pci@1,1/ide@3/cdrom@2,0:f
977: disk /pci@1f,0/pci@1,1/ide@3/disk@0,0
978: disk3 /pci@1f,0/pci@1,1/ide@3/disk@3,0
979: disk2 /pci@1f,0/pci@1,1/ide@3/disk@2,0
980: disk1 /pci@1f,0/pci@1,1/ide@3/disk@1,0
981: disk0 /pci@1f,0/pci@1,1/ide@3/disk@0,0
982: [...snip...]
983:
984: ok boot disk0 netbsd
985: Initializing Memory [...]
986: Boot device /pci/pci/ide@3/disk@0,0 File and args: netbsd
987: NetBSD IEEE 1275 Bootblock
988: >> NetBSD/sparc64 OpenFirmware Boot, Revision 1.13
989: >> (builds@b7.netbsd.org, Wed Jul 29 23:43:42 UTC 2009)
990: loadfile: reading header
991: elf64_exec: Booting [...]
992: symbols @ [....]
993: Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
994: 2006, 2007, 2008, 2009
995: The NetBSD Foundation, Inc. All rights reserved.
996: Copyright (c) 1982, 1986, 1989, 1991, 1993
997: The Regents of the University of California. All rights reserved.
998: [...snip...]
999:
1000: And the second disk:
1001:
1002: ok boot disk2 netbsd
1003: Initializing Memory [...]
1004: Boot device /pci/pci/ide@3/disk@2,0: File and args:netbsd
1005: NetBSD IEEE 1275 Bootblock
1006: >> NetBSD/sparc64 OpenFirmware Boot, Revision 1.13
1007: >> (builds@b7.netbsd.org, Wed Jul 29 23:43:42 UTC 2009)
1008: loadfile: reading header
1009: elf64_exec: Booting [...]
1010: symbols @ [....]
1011: Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
1012: 2006, 2007, 2008, 2009
1013: The NetBSD Foundation, Inc. All rights reserved.
1014: Copyright (c) 1982, 1986, 1989, 1991, 1993
1015: The Regents of the University of California. All rights reserved.
1016: [...snip...]
1017:
1018: At each boot, the following should appear in the NetBSD kerne
1019: [dmesg(8)](http://netbsd.gw.com/cgi-bin/man-cgi?dmesg+8+NetBSD-5.0.1+i386) :
1020:
1021: Kernelized RAIDframe activated
1022: raid0: RAID Level 1
1023: raid0: Components: /dev/wd0a /dev/wd1a
1024: raid0: Total Sectors: 19540864 (9541 MB)
1025: boot device: raid0
1026: root on raid0a dumps on raid0b
1027: root file system type: ffs
1028:
1029: Once you are certain that both disks are bootable, verify the RAID parity i
1030: clean after each reboot:
1031:
1032: # raidctl -v -s raid0
1033: Components:
1034: /dev/wd0a: optimal
1035: /dev/wd1a: optimal
1036: No spares.
1037: [...snip...]
1038: Component label for /dev/wd0a:
1039: Row: 0, Column: 0, Num Rows: 1, Num Columns: 2
1040: Version: 2, Serial Number: 2009122601, Mod Counter: 67
1041: Clean: No, Status: 0
1042: sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
1043: Queue size: 100, blocksize: 512, numBlocks: 19540864
1044: RAID Level: 1
1045: Autoconfig: Yes
1046: Root partition: Yes
1047: Last configured as: raid0
1048: Component label for /dev/wd1a:
1049: Row: 0, Column: 1, Num Rows: 1, Num Columns: 2
1050: Version: 2, Serial Number: 2009122601, Mod Counter: 67
1051: Clean: No, Status: 0
1052: sectPerSU: 128, SUsPerPU: 1, SUsPerRU: 1
1053: Queue size: 100, blocksize: 512, numBlocks: 19540864
1054: RAID Level: 1
1055: Autoconfig: Yes
1056: Root partition: Yes
1057: Last configured as: raid0
1058: Parity status: clean
1059: Reconstruction is 100% complete.
1060: Parity Re-write is 100% complete.
1061: Copyback is 100% complete.
1062:
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb
![[BACK]](/icons/back.gif){kind=icon}
Return to raidframe.mdwn CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [NetBSD Developer Wiki] / wikisrc / guide