Annotation of wikisrc/users/mlelstv/using-large-disks.mdwn, revision 1.26
1.3 mlelstv 1: **Contents**
2:
3: [[!toc levels=2]]
4:
1.1 mlelstv 5: # What do you do to use a large disk with NetBSD?
6:
7: Here are two:
1.2 mlelstv 8: <pre><code>
1.1 mlelstv 9: wd1 at atabus3 drive 0
10: wd1: <WDC WD60EFRX-68MYMN1>
11: wd1: drive supports 16-sector PIO transfers, LBA48 addressing
12: wd1: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
13: wd1: GPT GUID: 8ee69292-5099-11e4-833b-001cc4d779ed
14: wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
15: wd1(ahcisata0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
16:
17: wd2 at atabus4 drive 0
18: wd2: <WDC WD60EFRX-68MYMN1>
19: wd2: drive supports 16-sector PIO transfers, LBA48 addressing
20: wd2: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
21: wd2: GPT GUID: 902717d6-5099-11e4-833b-001cc4d779ed
22: wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
23: wd2(ahcisata0:2:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
1.2 mlelstv 24: </code></pre>
1.1 mlelstv 25:
26: # Disklabel
1.2 mlelstv 27: The NetBSD kernel seems to recgonize the disk size fine. But when running
1.1 mlelstv 28: the disklabel program to partition the disk, you get this:
1.2 mlelstv 29: <pre><code>
1.13 mlelstv 30: % disklabel wd1
1.1 mlelstv 31: # /dev/rwd1d:
32: type: ESDI
33: disk: WDC WD60EFRX-68M
34: label: fictitious
35: flags:
36: bytes/sector: 512
37: sectors/track: 63
38: tracks/cylinder: 16
39: sectors/cylinder: 1008
40: cylinders: 11628021
41: total sectors: 4294967295
42: rpm: 3600
43: interleave: 1
44: trackskew: 0
45: cylinderskew: 0
46: headswitch: 0 # microseconds
47: track-to-track seek: 0 # microseconds
48: drivedata: 0
49:
50: 4 partitions:
51: # size offset fstype [fsize bsize cpg/sgs]
1.7 mlelstv 52: a: 4294967295 0 4.2BSD 0 0 0 # (Cyl. 0 - 4294967295*)
53: d: 4294967295 0 unused 0 0 # (Cyl. 0 - 4294967295*)
54: disklabel: boot block size 0
55: disklabel: super block size 0
1.2 mlelstv 56: </code></pre>
1.1 mlelstv 57:
58: While the disk geometry seems to be correct, the sectors and partition
59: sizes are limited to 4294967295 which is 2^32-1. Older NetBSD kernels
60: even produce a size from the lower 32bit of the real sector count. This
1.2 mlelstv 61: is because the disklabel data structure has only 32bit fields to store
1.1 mlelstv 62: partition sizes and offsets and the total sector count. This is good
1.2 mlelstv 63: for up to 2TB disks with the standard sector size of 512 bytes.
1.1 mlelstv 64:
65: The disklabel is part of the original disk driver interface and
66: the driver will use that data inside its strategy routine to
67: validate disk accesses.
68:
69: You can still use the disk somewhat. While normal partitions are
70: limited by the 32bit values in the disklabel (or even the truncated
71: values set by older kernels), there is the "raw" partition (on
72: i386/amd64 that's partition 'd', other archs mostly use 'c') where
73: the driver ignores the disklabel and validates the access against
1.2 mlelstv 74: an internal value which is not truncated.
1.1 mlelstv 75:
76: There are a few obstacles though.
77:
1.6 mlelstv 78: ## formatting
1.2 mlelstv 79: <pre><code>
1.13 mlelstv 80: % sudo newfs /dev/rwd1d
1.1 mlelstv 81: newfs: /dev/rwd1d partition type is not `4.2BSD'
1.2 mlelstv 82: </code></pre>
1.1 mlelstv 83:
84: The raw partition has no correct type and newfs refuses to work, we
85: have to persuade it a little bit:
1.2 mlelstv 86: <pre><code>
1.13 mlelstv 87: % sudo newfs -O2 -F -s 11721045168 /dev/rwd1d
1.1 mlelstv 88: /dev/rwd1d: 5723166.5MB (11721045168 sectors) block size 32768, fragment size 4096
89: using 7710 cylinder groups of 742.31MB, 23754 blks, 46848 inodes.
90: super-block backups (for fsck_ffs -b #) at:
91: 192, 1520448, 3040704, 4560960, 6081216, 7601472, 9121728, 10641984, 12162240, 13682496, 15202752,
92: .......................................................................................................
1.2 mlelstv 93: </code></pre>
1.1 mlelstv 94:
95: We need -O2 for a filesystem > 1TB, -F to prevent newfs from checking
96: a disklabel and -s to tell it the disk size.
97:
1.6 mlelstv 98: ## using
1.1 mlelstv 99: Then we can mount and use the disk:
1.2 mlelstv 100: <pre><code>
1.13 mlelstv 101: % sudo mount /dev/wd1d /mnt
102: % sudo mkdir -m 1777 /mnt/scratch
103: % dd if=/dev/zero bs=1024k count=1000 of=/mnt/scratch/testfile
1.1 mlelstv 104: 1000+0 records in
105: 1000+0 records out
106: 1048576000 bytes transferred in 10.350 secs (101311690 bytes/sec)
1.13 mlelstv 107: % ls -la /mnt/scratch/
1.1 mlelstv 108: total 2048592
109: drwxrwxrwt 2 root wheel 512 Oct 25 17:34 .
110: drwxr-xr-x 3 root wheel 512 Oct 25 17:34 ..
111: -rw-r--r-- 1 mlelstv wheel 1048576000 Oct 25 17:35 testfile
1.13 mlelstv 112: % sudo umount /mnt
1.2 mlelstv 113: </code></pre>
1.5 mlelstv 114:
1.6 mlelstv 115: ## checking
1.1 mlelstv 116: Filesystem checking needs extra effort:
1.2 mlelstv 117: <pre><code>
1.13 mlelstv 118: % sudo fsck /dev/rwd1d
1.1 mlelstv 119: fsck: vfstype `unused' on partition `/dev/rwd1d' is not supported
1.2 mlelstv 120: </code></pre>
1.1 mlelstv 121:
122: Again you need to augment information from the disklabel for the raw
123: partition:
1.2 mlelstv 124: <pre><code>
1.13 mlelstv 125: % sudo fsck -t ffs /dev/rwd1d
1.1 mlelstv 126: ** /dev/rwd1d
127: ** File system is clean; not checking
1.13 mlelstv 128: % sudo fsck -t ffs -f /dev/rwd1d
1.1 mlelstv 129: ** /dev/rwd1d
130: ** File system is already clean
131: ** Last Mounted on /mnt
132: ** Phase 1 - Check Blocks and Sizes
133: ** Phase 2 - Check Pathnames
134: ** Phase 3 - Check Connectivity
135: ** Phase 4 - Check Reference Counts
136: ** Phase 5 - Check Cyl groups
137: 3 files, 256074 used, 1442176277 free (21 frags, 180272032 blocks, 0.0% fragmentation)
1.2 mlelstv 138: </code></pre>
1.1 mlelstv 139:
1.5 mlelstv 140: ## and more
1.1 mlelstv 141: You can also modify the disklabel, set the raw partition to type 4.2BSD
142: (including fsize+bsize parameters) and delete the 'a' partition to avoid
143: overlap warnings. The result looks like:
1.2 mlelstv 144: <pre><code>
1.1 mlelstv 145: 4 partitions:
146: # size offset fstype [fsize bsize cpg/sgs]
147: d: 4294967295 0 4.2BSD 4096 65536 0 # (Cyl. 0 - 4294967295*)
1.2 mlelstv 148: </code></pre>
1.1 mlelstv 149:
150: This makes fsck recognize the disk correctly, newfs still requires the
151: disksize parameter.
152:
1.3 mlelstv 153: One more caveat, when you write the disklabel to the disk, it has to
154: coexist with the filesystem data. Since the raw partition starts at
155: offset 0 (the disklabel data is ignored by the driver!), this does not
156: work with every filesystem, but FFS is safe.
157:
1.1 mlelstv 158: # Wedges
1.3 mlelstv 159:
160: Wedges solve two problems in NetBSD. They support larger disks without
161: the 32bit limitation of the disklabel, and they can be used with any
162: partition information on the disk which makes it possible to exchange
163: disks between different platforms.
164:
165: Here is a disk with a wedge:
166: <pre><code>
1.13 mlelstv 167: % sudo dkctl wd2 listwedges
1.3 mlelstv 168: /dev/rwd2d: 1 wedge:
1.4 mlelstv 169: dk6: hugedisk2, 11721043968 blocks at 1024, type: ccd
1.3 mlelstv 170: </code></pre>
171:
172: Wedges can be created and removed at any time with the dkctl command
173: by specifying a name, start offset on the disk, the wedge size and
174: a type.
175: <pre><code>
1.13 mlelstv 176: % sudo dkctl wd2 addwedge testwedge 34 500 ffs
1.3 mlelstv 177: dk5 created successfully.
1.13 mlelstv 178: % sudo dkctl wd2 addwedge testwedge 500 100 unused
1.3 mlelstv 179: dkctl: /dev/rwd2d: addwedge: Invalid argument
1.13 mlelstv 180: % sudo dkctl wd2 addwedge testwedge 534 100 unused
1.3 mlelstv 181: dkctl: /dev/rwd2d: addwedge: File exists
1.13 mlelstv 182: % sudo dkctl wd2 addwedge testwedge2 534 100 unused
1.3 mlelstv 183: dk10 created successfully.
1.13 mlelstv 184: % sudo dkctl wd2 listwedges
1.3 mlelstv 185: /dev/rwd2d: 3 wedges:
186: dk5: testwedge, 500 blocks at 34, type: ffs
187: dk10: testwedge2, 100 blocks at 534, type: unused
1.4 mlelstv 188: dk6: hugedisk2, 11721043968 blocks at 1024, type: ccd
1.3 mlelstv 189: </code></pre>
190:
191: You can see that creation of a wedge is validated, it requires a unique
192: name and the wedges must not overlap.
193: <pre><code>
1.13 mlelstv 194: % sudo newfs NAME=testwedge
1.3 mlelstv 195: /dev/rdk5: 0.2MB (500 sectors) block size 4096, fragment size 512
196: using 3 cylinder groups of 0.08MB, 21 blks, 32 inodes.
197: super-block backups (for fsck_ffs -b #) at:
198: 32, 200, 368,
199: </code></pre>
200:
1.4 mlelstv 201: Older versions of newfs do not support wedgenames, you need to
202: specify the device, e.g. /dev/rdk5.
203:
204:
205: # GPT
206:
207: If wedges could only be created by running a command, they wouldn't be
208: useful. But the NetBSD kernel can generate wedges automatically when
209: a disk is attached. Most NetBSD architectures will scan for MBR (PC BIOS
210: Master Boot Record) and GPT (GUID Partition Table). There is also support
211: for standard BSD disklabels but which is currently disabled.
212:
213: A GPT can be displayed, created and edited with the GPT command:
214: <pre><code>
1.13 mlelstv 215: % sudo gpt show wd2
1.4 mlelstv 216: start size index contents
217: 0 1 PMBR
218: 1 1 Pri GPT header
219: 2 32 Pri GPT table
220: 34 990
221: 1024 11721043968 1 GPT part - NetBSD ccd component
222: 11721044992 143
223: 11721045135 32 Sec GPT table
224: 11721045167 1 Sec GPT header
1.13 mlelstv 225: % sudo gpt show wd1
1.4 mlelstv 226: start size index contents
227: 0 11721045168
1.13 mlelstv 228: % sudo gpt create wd1
229: % sudo gpt show wd1
1.4 mlelstv 230: start size index contents
231: 0 1 PMBR
232: 1 1 Pri GPT header
233: 2 32 Pri GPT table
234: 34 11721045101
235: 11721045135 32 Sec GPT table
236: 11721045167 1 Sec GPT header
1.13 mlelstv 237: % sudo gpt add -a 512k -l hugedisk1 -t ccd wd1
1.4 mlelstv 238: Partition 1 added, use:
239: dkctl wd1 addwedge <wedgename> 1024 11721043968 <type>
240: to create a wedge for it
1.13 mlelstv 241: % sudo gpt show wd1
1.4 mlelstv 242: start size index contents
243: 0 1 PMBR
244: 1 1 Pri GPT header
245: 2 32 Pri GPT table
246: 34 990
247: 1024 11721043968 1 GPT part - NetBSD ccd component
248: 11721044992 143
249: 11721045135 32 Sec GPT table
250: 11721045167 1 Sec GPT header
1.16 mlelstv 251: </code></pre>
1.4 mlelstv 252:
253: Since wedges are currently only created when a device is attached, we
254: need either a reboot or some magic using drvctl.
255: <pre><code>
1.13 mlelstv 256: % sudo drvctl -d wd1
257: % sudo drvctl -d wd2
258: % sudo drvctl -a ata_hl -r atabus3
259: % sudo drvctl -a ata_hl -r atabus4
1.16 mlelstv 260: </code></pre>
1.4 mlelstv 261:
1.22 mlelstv 262: NetBSD-current as a of 20141104 can rescan a device for wedges instead.
263: This will delete all unused wedges for a device and readd them according
264: to the label.
265: <pre><code>
266: % sudo dkctl wd1 makewedges
267: successfully scanned /dev/rwd1d.
268: % sudo dkctl wd2 makewedges
269: successfully scanned /dev/rwd2d.
270: </code></pre>
271:
1.4 mlelstv 272: The console or dmesg will reveal that both disks have been reattached
273: and wedges have been created:
274: <pre><code>
275: wd1 at atabus3 drive 0
276: wd1: <WDC WD60EFRX-68MYMN1>
277: wd1: drive supports 16-sector PIO transfers, LBA48 addressing
278: wd1: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
279: wd1: GPT GUID: 38ba6ff4-e48a-42e4-a513-fe217d7fa013
280: dk5 at wd1: hugedisk1
281: dk5: 11721043968 blocks at 1024, type: ccd
282: wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
283: wd1(ahcisata0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
284: wd2 at atabus4 drive 0
285: wd2: <WDC WD60EFRX-68MYMN1>
286: wd2: drive supports 16-sector PIO transfers, LBA48 addressing
287: wd2: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
288: wd2: GPT GUID: 902717d6-5099-11e4-833b-001cc4d779ed
289: dk6 at wd2: hugedisk2
290: dk6: 11721043968 blocks at 1024, type: ccd
291: wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
292: wd2(ahcisata0:2:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
1.16 mlelstv 293: </code></pre>
1.4 mlelstv 294:
1.8 mlelstv 295: # CCD
296:
297: If the partitions (and wedges) were created with type ffs you could just
298: use them to format filesystems and mount them but I created them with
299: type ccd to be used by the concatenating disk driver.
300:
301: ccd is a very old driver and has some deficiencies. For one, the ccdconfig
1.24 mlelstv 302: tool didn't understand wedge names until very recently.
1.8 mlelstv 303: <pre><code>
1.13 mlelstv 304: % sudo ccdconfig -c ccd0 16 none NAME=hugedisk1 NAME=hugedisk2
1.8 mlelstv 305: ccdconfig: NAME=hugedisk1: No such file or directory
1.16 mlelstv 306: </code></pre>
1.8 mlelstv 307:
1.14 mlelstv 308: ## configuring
1.8 mlelstv 309: For now we use the wedge device directly.
310: <pre><code>
1.13 mlelstv 311: % sudo ccdconfig -c ccd0 16 none /dev/dk5 /dev/dk6
312: % disklabel ccd0
1.8 mlelstv 313: # /dev/rccd0d:
314: type: ccd
315: disk: ccd
316: label: fictitious
317: flags:
318: bytes/sector: 512
319: sectors/track: 2048
320: tracks/cylinder: 1
321: sectors/cylinder: 2048
322: cylinders: 11446332
323: total sectors: 4294967295
324: rpm: 3600
325: interleave: 1
326: trackskew: 0
327: cylinderskew: 0
328: headswitch: 0 # microseconds
329: track-to-track seek: 0 # microseconds
330: drivedata: 0
331:
332: 4 partitions:
333: # size offset fstype [fsize bsize cpg/sgs]
1.17 mlelstv 334: a: 4294967295 0 4.2BSD 0 0 0 # (Cyl. 0 - 4294967295*)
335: d: 4294967295 0 unused 0 0 # (Cyl. 0 - 4294967295*)
336: disklabel: boot block size 0
337: disklabel: super block size 0
1.16 mlelstv 338: </code></pre>
1.8 mlelstv 339:
1.14 mlelstv 340: ## using
1.8 mlelstv 341: Obviously the device that spans two disks is also too large for disklabel.
342: What about wedges?
343: <pre><code>
1.13 mlelstv 344: % sudo dkctl ccd0 listwedges
1.8 mlelstv 345: dkctl: /dev/rccd0d: listwedges: Inappropriate ioctl for device
1.16 mlelstv 346: </code></pre>
1.8 mlelstv 347:
348: The ccd driver does not support wedges at all. But we can still use
349: the raw partition.
350: <pre><code>
1.13 mlelstv 351: % sudo newfs -O2 -F -s 23442087936 /dev/rccd0d
1.8 mlelstv 352: /dev/rccd0d: 11446332.0MB (23442087936 sectors) block size 32768, fragment size 4096
353: using 15420 cylinder groups of 742.31MB, 23754 blks, 46848 inodes.
354: super-block backups (for fsck_ffs -b #) at:
355: 192, 1520448, 3040704, 4560960, 6081216, 7601472, 9121728, 10641984, 12162240, 13682496, 15202752, 16723008, 18243264,
356: ......................................................................................................................
1.13 mlelstv 357: % sudo mount /dev/ccd0d /mnt
358: % df -h /mnt
1.8 mlelstv 359: Filesystem Size Used Avail %Cap Mounted on
360: /dev/ccd0d 11T 4.0K 10T 0% /mnt
1.13 mlelstv 361: % sudo umount /mnt
362: % sudo ccdconfig -u ccd0
1.16 mlelstv 363: </code></pre>
1.8 mlelstv 364:
1.9 mlelstv 365: # Raidframe
366: Since ccd wasn't that good, lets create a RAID0 using the raidframe driver.
367:
368: First, change the partition types:
369: <pre><code>
1.13 mlelstv 370: % sudo gpt type -i 1 -t ccd -T raid wd1
1.9 mlelstv 371: partition 1 type changed
1.13 mlelstv 372: % sudo gpt type -i 1 -t ccd -T raid wd2
1.9 mlelstv 373: partition 1 type changed
1.13 mlelstv 374: % sudo gpt show wd1
1.9 mlelstv 375: start size index contents
376: 0 1 PMBR
377: 1 1 Pri GPT header
378: 2 32 Pri GPT table
379: 34 990
380: 1024 11721043968 1 GPT part - NetBSD RAIDFrame component
381: 11721044992 143
382: 11721045135 32 Sec GPT table
383: 11721045167 1 Sec GPT header
1.13 mlelstv 384: % sudo gpt show wd2
1.9 mlelstv 385: start size index contents
386: 0 1 PMBR
387: 1 1 Pri GPT header
388: 2 32 Pri GPT table
389: 34 990
390: 1024 11721043968 1 GPT part - NetBSD RAIDFrame component
391: 11721044992 143
392: 11721045135 32 Sec GPT table
393: 11721045167 1 Sec GPT header
1.16 mlelstv 394: </code></pre>
1.9 mlelstv 395:
1.22 mlelstv 396: And the reattachment magic (or remake the wedges on recent NetBSD):
1.9 mlelstv 397: <pre><code>
1.13 mlelstv 398: % sudo drvctl -d wd1
399: % sudo drvctl -d wd2
400: % sudo drvctl -a ata_hl -r atabus3
401: % sudo drvctl -a ata_hl -r atabus4
402: % sudo dkctl wd1 listwedges
1.9 mlelstv 403: /dev/rwd1d: 1 wedge:
404: dk5: hugedisk1, 11721043968 blocks at 1024, type: raidframe
1.13 mlelstv 405: % sudo dkctl wd2 listwedges
1.9 mlelstv 406: /dev/rwd2d: 1 wedge:
407: dk6: hugedisk2, 11721043968 blocks at 1024, type: raidframe
1.16 mlelstv 408: </code></pre>
1.9 mlelstv 409:
1.10 mlelstv 410: ## configuring
1.9 mlelstv 411: For raidframe we need a configuration file:
412: <pre><code>
1.13 mlelstv 413: % cat raid0.conf
1.9 mlelstv 414: START array
415: # numRow numCol numSpare
416: 1 2 0
417:
418: START disks
419: /dev/dk5
420: /dev/dk6
421:
422: START layout
423: # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_0
424: 16 1 1 0
425:
426: START queue
427: fifo 100
1.13 mlelstv 428: % sudo raidctl -C raid0.conf raid0
429: % sudo raidctl -I 7894737 raid0
430: % sudo raidctl -A yes raid0
1.9 mlelstv 431: raid0: Autoconfigure: Yes
432: raid0: Root: No
1.13 mlelstv 433: % disklabel raid0
1.9 mlelstv 434: # /dev/rraid0d:
435: type: RAID
436: disk: raid
437: label: fictitious
438: flags:
439: bytes/sector: 512
440: sectors/track: 32
441: tracks/cylinder: 8
442: sectors/cylinder: 256
443: cylinders: 91570655
444: total sectors: 4294967295
445: rpm: 3600
446: interleave: 1
447: trackskew: 0
448: cylinderskew: 0
449: headswitch: 0 # microseconds
450: track-to-track seek: 0 # microseconds
451: drivedata: 0
452:
453: 4 partitions:
454: # size offset fstype [fsize bsize cpg/sgs]
1.17 mlelstv 455: a: 4294967295 0 4.2BSD 0 0 0 # (Cyl. 0 - 4294967295*)
456: d: 4294967295 0 unused 0 0 # (Cyl. 0 - 4294967295*)
1.9 mlelstv 457: disklabel: boot block size 0
458: disklabel: super block size 0
1.13 mlelstv 459: % sudo dkctl raid0 listwedges
1.9 mlelstv 460: /dev/rraid0d: no wedges configured
1.16 mlelstv 461: </code></pre>
1.9 mlelstv 462:
1.10 mlelstv 463: ## formatting
1.9 mlelstv 464: The raidframe driver has no problems with wedges. We can create a GPT
465: on the raid device and create the wedge and format a filesystem.
466: This time manually but the reattachment magic would do the same.
467: <pre><code>
1.13 mlelstv 468: % sudo gpt create raid0
469: % sudo gpt add -a 1024 -t ffs -l stripes raid0
1.9 mlelstv 470: Partition 1 added, use:
471: dkctl raid0 addwedge <wedgename> 34 23442087740 <type>
472: to create a wedge for it
1.13 mlelstv 473: % sudo dkctl raid0 addwedge stripes 34 23442087740 ffs
1.9 mlelstv 474: dk10 created successfully.
1.13 mlelstv 475: % sudo newfs -O2 NAME=stripes
1.9 mlelstv 476: /dev/rdk10: 11446332.0MB (23442087736 sectors) block size 32768, fragment size 4096
477: using 15420 cylinder groups of 742.31MB, 23754 blks, 46848 inodes.
478: super-block backups (for fsck_ffs -b #) at:
479: 192, 1520448, 3040704, 4560960, 6081216, 7601472, 9121728, 10641984, 12162240, 13682496, 15202752, 16723008, 18243264,
480: ......................................................................................................................
1.10 mlelstv 481: </code></pre>
482:
483: ## using
1.11 mlelstv 484: Wedges can be mounted by name.
1.10 mlelstv 485: <pre><code>
1.13 mlelstv 486: % sudo mount NAME=stripes /mnt
487: % df -h /mnt
1.9 mlelstv 488: Filesystem Size Used Avail %Cap Mounted on
489: /dev/dk10 11T 4.0K 10T 0% /mnt
1.13 mlelstv 490: % sudo mkdir -m 1777 /mnt/scratch
491: % dd if=/dev/zero bs=1024k count=1000 of=/mnt/scratch/testfile
1.9 mlelstv 492: 1000+0 records in
493: 1000+0 records out
494: 1048576000 bytes transferred in 4.961 secs (211363837 bytes/sec)
1.13 mlelstv 495: % sudo umount /mnt
1.16 mlelstv 496: </code></pre>
1.9 mlelstv 497:
1.11 mlelstv 498: ## checking
1.12 mlelstv 499: And also be checked.
1.11 mlelstv 500: <pre><code>
1.15 mlelstv 501: % sudo fsck NAME=stripes
502: ** /dev/rdk10
503: ** File system is clean; not checking
1.16 mlelstv 504: </code></pre>
1.11 mlelstv 505:
1.18 mlelstv 506: Older versions of fsck do not support wedgenames, you need to
1.12 mlelstv 507: specify the device, e.g. /dev/rdk10.
508:
1.10 mlelstv 509: ## and more
1.12 mlelstv 510: Verify that this would reconfigure automatically after a reboot:
1.9 mlelstv 511: <pre><code>
1.13 mlelstv 512: % sudo raidctl -u raid0
513: % sudo drvctl -d wd1
514: % sudo drvctl -d wd2
515: % sudo drvctl -a ata_hl -r atabus3
516: % sudo drvctl -a ata_hl -r atabus4
517: % sudo raidctl -c raid0.conf raid0
518: % sudo mount NAME=stripes /mnt
519: % df -h /mnt
1.9 mlelstv 520: Filesystem Size Used Avail %Cap Mounted on
521: /dev/dk10 11T 1.0G 10T 0% /mnt
1.13 mlelstv 522: % ls -l /mnt/scratch/testfile
1.9 mlelstv 523: -rw-r--r-- 1 mlelstv wheel 1048576000 Oct 25 21:47 /mnt/scratch/testfile
1.16 mlelstv 524: </code></pre>
1.8 mlelstv 525:
1.12 mlelstv 526: Unlike wedges, the raidframe devices are not automatically created when a disk attaches.
527: That's why we needed the raidctl -c command. The raidframe driver however scans all disks
1.19 mlelstv 528: available when booting and autoconfigures devices for all raidsets found that have the
529: autoconfig flag set.
1.12 mlelstv 530:
1.21 mlelstv 531: # LVM
532:
533: Linux LVM is a different scheme to manage disk space, it uses its own label to group
534: multiple disks together and to carve out blocks using the device mapper driver to
535: form logical volumes.
536:
537: The device mapper provides logical block and character devices that route I/O to
538: physical disks or some other logical devices. Such devices can be used for filesystems
539: like a disk partition or wedge.
540:
541: LVM is mostly used on raw disks as there is rarely a necessity to partition a disk first.
542: But it can also be used on disk partitions or wedges, this has advantages if the disks are used
543: for booting or are moved between different systems or platforms.
544:
545: ## LVM on raw disks
546:
547: LVM disks are labeled first with the pvcreate command and then coalesced into a volume group.
548: <pre><code>
549: % sudo lvm pvcreate /dev/rwd1d
550: Physical volume "/dev/rwd1d" successfully created
551: % sudo lvm pvcreate /dev/rwd2d
552: Physical volume "/dev/rwd2d" successfully created
553: % sudo lvm vgcreate vg0 /dev/rwd1d /dev/rwd2d
554: Volume group "vg0" successfully created
555: % sudo lvm pvs
556: PV VG Fmt Attr PSize PFree
557: /dev/rwd1d vg0 lvm2 a- 2.00t 2.00t
558: /dev/rwd2d vg0 lvm2 a- 2.00t 2.00t
559: </code></pre>
560:
1.25 mlelstv 561: This shows a problem with large disks, the LVM tools only understand
562: conventional disk partitions that are limited to 2TB each. However,
563: neither LVM, the device mapper driver nor the disk drivers when
564: using the raw partition are bound to the disklabel information.
565: But you need to tell LVM the real size to override the synthesized
566: disklabel. The real size is 11721045168 sectors by 512 bytes giving
567: 5723166 Megabytes.
1.21 mlelstv 568: <pre><code>
569: % sudo lvm vgremove vg0
570: Volume group "vg0" successfully removed
571: % sudo lvm pvcreate --setphysicalvolumesize=5723166 /dev/rwd1d
572: WARNING: /dev/rwd1d: Overriding real size. You could lose data.
573: Physical volume "/dev/rwd1d" successfully created
574: % sudo lvm pvcreate --setphysicalvolumesize=5723166 /dev/rwd2d
575: WARNING: /dev/rwd2d: Overriding real size. You could lose data.
576: Physical volume "/dev/rwd2d" successfully created
577: % sudo lvm vgcreate vg0 /dev/rwd1d /dev/rwd2d
578: Volume group "vg0" successfully created
579: % sudo lvm pvs
580: PV VG Fmt Attr PSize PFree
581: /dev/rwd1d vg0 lvm2 a- 5.46t 5.46t
582: /dev/rwd2d vg0 lvm2 a- 5.46t 5.46t
583: </code></pre>
584:
585: ## LVM on wedges
586:
587: Here are again the two disks with a large wedge each. The wedge type is unknown
588: because the GPT lists the partition as type linux-lvm which has no well-known
589: wedge type.
590: <pre><code>
591: % sudo dkctl wd1 listwedges
592: /dev/rwd1d: 1 wedge:
593: dk8: hugedisk1, 11721045100 blocks at 34, type:
594: % sudo dkctl wd2 listwedges
595: /dev/rwd2d: 1 wedge:
596: dk9: hugedisk2, 11721045100 blocks at 34, type:
597: </code></pre>
598:
599: Now label the disks, form a volume group, create a logical partition and a filesystem:
600: <pre><code>
601: % sudo lvm pvcreate /dev/rdk8
602: Physical volume "/dev/rdk8" successfully created
603: % sudo lvm pvcreate /dev/rdk9
604: Physical volume "/dev/rdk9" successfully created
605: % sudo lvm vgcreate vg0 /dev/rdk8 /dev/rdk9
606: Volume group "vg0" successfully created
607: % sudo lvm pvs
608: PV VG Fmt Attr PSize PFree
609: /dev/rdk8 vg0 lvm2 a- 5.46t 5.46t
610: /dev/rdk9 vg0 lvm2 a- 5.46t 5.46t
611: % sudo lvm lvcreate -L 500m -n lvtest vg0
612: Logical volume "lvtest" created
613: % sudo newfs -O2 /dev/vg0/lvtest
614: /dev/mapper/rvg0-lvtest: 500.0MB (1024000 sectors) block size 8192, fragment size 1024
615: using 11 cylinder groups of 45.46MB, 5819 blks, 10976 inodes.
616: super-block backups (for fsck_ffs -b #) at:
617: 144, 93248, 186352, 279456, 372560, 465664, 558768, 651872, 744976, 838080,
618: ...............................................................................
619: % sudo mount /dev/vg0/lvtest /mnt
620: mount_ffs: "/dev/vg0/lvtest" is a non-resolved or relative path.
621: mount_ffs: using "/dev/mapper/vg0-lvtest" instead.
622: % df -h /mnt
623: Filesystem Size Used Avail %Cap Mounted on
624: /dev/mapper/vg0-lvtest 470M 1.0K 447M 0% /mnt
625: </code></pre>
626:
1.25 mlelstv 627: There is one issue with LVM on wedges. LVM scans disks for its
628: label to coalesce them into volume groups and to find the logical
629: volumes. You can restrict the search in the LVM configuration, but
630: if you use wedges, you must scan all wedges as the name of the
631: device may change. Also, the result of a scan is saved to optimize
632: subsequent scans. If your disk configuration changes, you either
633: need to remove the cached result or configure LVM to not save it
634: with the write_cache_state option.
1.21 mlelstv 635:
CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb
![[BACK]](/icons/back.gif){kind=icon}
Return to using-large-disks.mdwn CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [NetBSD Developer Wiki] / wikisrc / users / mlelstv