Annotation of wikisrc/users/mlelstv/using-large-disks.mdwn, revision 1.26

1.3       mlelstv     1: **Contents**
                      2: 
                      3: [[!toc levels=2]]
                      4: 
1.1       mlelstv     5: # What do you do to use a large disk with NetBSD?
                      6: 
                      7: Here are two:
1.2       mlelstv     8: <pre><code>
1.1       mlelstv     9:     wd1 at atabus3 drive 0
                     10:     wd1: <WDC WD60EFRX-68MYMN1>
                     11:     wd1: drive supports 16-sector PIO transfers, LBA48 addressing
                     12:     wd1: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
                     13:     wd1: GPT GUID: 8ee69292-5099-11e4-833b-001cc4d779ed
                     14:     wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
                     15:     wd1(ahcisata0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
                     16:     
                     17:     wd2 at atabus4 drive 0
                     18:     wd2: <WDC WD60EFRX-68MYMN1>
                     19:     wd2: drive supports 16-sector PIO transfers, LBA48 addressing
                     20:     wd2: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
                     21:     wd2: GPT GUID: 902717d6-5099-11e4-833b-001cc4d779ed
                     22:     wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
                     23:     wd2(ahcisata0:2:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
1.2       mlelstv    24: </code></pre>
1.1       mlelstv    25:     
                     26: # Disklabel
1.2       mlelstv    27: The NetBSD kernel seems to recgonize the disk size fine. But when running
1.1       mlelstv    28: the disklabel program to partition the disk, you get this:
1.2       mlelstv    29: <pre><code>
1.13      mlelstv    30:     % disklabel wd1
1.1       mlelstv    31:     # /dev/rwd1d:
                     32:     type: ESDI
                     33:     disk: WDC WD60EFRX-68M
                     34:     label: fictitious
                     35:     flags:
                     36:     bytes/sector: 512
                     37:     sectors/track: 63
                     38:     tracks/cylinder: 16
                     39:     sectors/cylinder: 1008
                     40:     cylinders: 11628021
                     41:     total sectors: 4294967295
                     42:     rpm: 3600
                     43:     interleave: 1
                     44:     trackskew: 0
                     45:     cylinderskew: 0
                     46:     headswitch: 0           # microseconds
                     47:     track-to-track seek: 0  # microseconds
                     48:     drivedata: 0 
                     49: 
                     50:     4 partitions:
                     51:     #        size    offset     fstype [fsize bsize cpg/sgs]
1.7       mlelstv    52:     a: 4294967295         0     4.2BSD      0     0     0  # (Cyl.      0 - 4294967295*)
                     53:     d: 4294967295         0     unused      0     0        # (Cyl.      0 - 4294967295*)
                     54:     disklabel: boot block size 0
                     55:     disklabel: super block size 0
1.2       mlelstv    56: </code></pre>
1.1       mlelstv    57: 
                     58: While the disk geometry seems to be correct, the sectors and partition
                     59: sizes are limited to 4294967295 which is 2^32-1. Older NetBSD kernels
                     60: even produce a size from the lower 32bit of the real sector count. This
1.2       mlelstv    61: is because the disklabel data structure has only 32bit fields to store
1.1       mlelstv    62: partition sizes and offsets and the total sector count. This is good
1.2       mlelstv    63: for up to 2TB disks with the standard sector size of 512 bytes.
1.1       mlelstv    64: 
                     65: The disklabel is part of the original disk driver interface and
                     66: the driver will use that data inside its strategy routine to
                     67: validate disk accesses.
                     68: 
                     69: You can still use the disk somewhat. While normal partitions are
                     70: limited by the 32bit values in the disklabel (or even the truncated
                     71: values set by older kernels), there is the "raw" partition (on
                     72: i386/amd64 that's partition 'd', other archs mostly use 'c') where
                     73: the driver ignores the disklabel and validates the access against
1.2       mlelstv    74: an internal value which is not truncated.
1.1       mlelstv    75: 
                     76: There are a few obstacles though.
                     77: 
1.6       mlelstv    78: ## formatting
1.2       mlelstv    79: <pre><code>
1.13      mlelstv    80:     % sudo newfs /dev/rwd1d
1.1       mlelstv    81:     newfs: /dev/rwd1d partition type is not `4.2BSD'
1.2       mlelstv    82: </code></pre>
1.1       mlelstv    83: 
                     84: The raw partition has no correct type and newfs refuses to work, we
                     85: have to persuade it a little bit:
1.2       mlelstv    86: <pre><code>
1.13      mlelstv    87:     % sudo newfs -O2 -F -s 11721045168 /dev/rwd1d
1.1       mlelstv    88:     /dev/rwd1d: 5723166.5MB (11721045168 sectors) block size 32768, fragment size 4096
                     89:             using 7710 cylinder groups of 742.31MB, 23754 blks, 46848 inodes.
                     90:     super-block backups (for fsck_ffs -b #) at:
                     91:     192, 1520448, 3040704, 4560960, 6081216, 7601472, 9121728, 10641984, 12162240, 13682496, 15202752,
                     92:     .......................................................................................................
1.2       mlelstv    93: </code></pre>
1.1       mlelstv    94: 
                     95: We need -O2 for a filesystem > 1TB, -F to prevent newfs from checking
                     96: a disklabel and -s to tell it the disk size.
                     97: 
1.6       mlelstv    98: ## using
1.1       mlelstv    99: Then we can mount and use the disk:
1.2       mlelstv   100: <pre><code>
1.13      mlelstv   101:     % sudo mount /dev/wd1d /mnt
                    102:     % sudo mkdir -m 1777 /mnt/scratch
                    103:     % dd if=/dev/zero bs=1024k count=1000 of=/mnt/scratch/testfile
1.1       mlelstv   104:     1000+0 records in
                    105:     1000+0 records out
                    106:     1048576000 bytes transferred in 10.350 secs (101311690 bytes/sec)
1.13      mlelstv   107:     % ls -la /mnt/scratch/
1.1       mlelstv   108:     total 2048592
                    109:     drwxrwxrwt  2 root     wheel         512 Oct 25 17:34 .
                    110:     drwxr-xr-x  3 root     wheel         512 Oct 25 17:34 ..
                    111:     -rw-r--r--  1 mlelstv  wheel  1048576000 Oct 25 17:35 testfile
1.13      mlelstv   112:     % sudo umount /mnt
1.2       mlelstv   113: </code></pre>
1.5       mlelstv   114: 
1.6       mlelstv   115: ## checking
1.1       mlelstv   116: Filesystem checking needs extra effort:
1.2       mlelstv   117: <pre><code>
1.13      mlelstv   118:     % sudo fsck /dev/rwd1d
1.1       mlelstv   119:     fsck: vfstype `unused' on partition `/dev/rwd1d' is not supported
1.2       mlelstv   120: </code></pre>
1.1       mlelstv   121: 
                    122: Again you need to augment information from the disklabel for the raw
                    123: partition:
1.2       mlelstv   124: <pre><code>
1.13      mlelstv   125:     % sudo fsck -t ffs /dev/rwd1d
1.1       mlelstv   126:     ** /dev/rwd1d
                    127:     ** File system is clean; not checking
1.13      mlelstv   128:     % sudo fsck -t ffs -f /dev/rwd1d
1.1       mlelstv   129:     ** /dev/rwd1d
                    130:     ** File system is already clean
                    131:     ** Last Mounted on /mnt
                    132:     ** Phase 1 - Check Blocks and Sizes
                    133:     ** Phase 2 - Check Pathnames
                    134:     ** Phase 3 - Check Connectivity
                    135:     ** Phase 4 - Check Reference Counts
                    136:     ** Phase 5 - Check Cyl groups
                    137:     3 files, 256074 used, 1442176277 free (21 frags, 180272032 blocks, 0.0% fragmentation)
1.2       mlelstv   138: </code></pre>
1.1       mlelstv   139: 
1.5       mlelstv   140: ## and more
1.1       mlelstv   141: You can also modify the disklabel, set the raw partition to type 4.2BSD
                    142: (including fsize+bsize parameters) and delete the 'a' partition to avoid
                    143: overlap warnings. The result looks like:
1.2       mlelstv   144: <pre><code>
1.1       mlelstv   145:     4 partitions:
                    146:     #        size    offset     fstype [fsize bsize cpg/sgs]
                    147:      d: 4294967295         0     4.2BSD   4096 65536     0  # (Cyl.      0 - 4294967295*)
1.2       mlelstv   148: </code></pre>
1.1       mlelstv   149: 
                    150: This makes fsck recognize the disk correctly, newfs still requires the
                    151: disksize parameter.
                    152: 
1.3       mlelstv   153: One more caveat, when you write the disklabel to the disk, it has to
                    154: coexist with the filesystem data. Since the raw partition starts at
                    155: offset 0 (the disklabel data is ignored by the driver!), this does not
                    156: work with every filesystem, but FFS is safe.
                    157: 
1.1       mlelstv   158: # Wedges
1.3       mlelstv   159: 
                    160: Wedges solve two problems in NetBSD. They support larger disks without
                    161: the 32bit limitation of the disklabel, and they can be used with any
                    162: partition information on the disk which makes it possible to exchange
                    163: disks between different platforms.
                    164: 
                    165: Here is a disk with a wedge:
                    166: <pre><code>
1.13      mlelstv   167:     % sudo dkctl wd2 listwedges
1.3       mlelstv   168:     /dev/rwd2d: 1 wedge:
1.4       mlelstv   169:     dk6: hugedisk2, 11721043968 blocks at 1024, type: ccd
1.3       mlelstv   170: </code></pre>
                    171: 
                    172: Wedges can be created and removed at any time with the dkctl command
                    173: by specifying a name, start offset on the disk, the wedge size and
                    174: a type.
                    175: <pre><code>
1.13      mlelstv   176:     % sudo dkctl wd2 addwedge testwedge 34 500 ffs
1.3       mlelstv   177:     dk5 created successfully.
1.13      mlelstv   178:     % sudo dkctl wd2 addwedge testwedge 500 100 unused
1.3       mlelstv   179:     dkctl: /dev/rwd2d: addwedge: Invalid argument
1.13      mlelstv   180:     % sudo dkctl wd2 addwedge testwedge 534 100 unused
1.3       mlelstv   181:     dkctl: /dev/rwd2d: addwedge: File exists
1.13      mlelstv   182:     % sudo dkctl wd2 addwedge testwedge2 534 100 unused
1.3       mlelstv   183:     dk10 created successfully.
1.13      mlelstv   184:     % sudo dkctl wd2 listwedges
1.3       mlelstv   185:     /dev/rwd2d: 3 wedges:
                    186:     dk5: testwedge, 500 blocks at 34, type: ffs
                    187:     dk10: testwedge2, 100 blocks at 534, type: unused
1.4       mlelstv   188:     dk6: hugedisk2, 11721043968 blocks at 1024, type: ccd
1.3       mlelstv   189: </code></pre>
                    190: 
                    191: You can see that creation of a wedge is validated, it requires a unique
                    192: name and the wedges must not overlap.
                    193: <pre><code>
1.13      mlelstv   194:     % sudo newfs NAME=testwedge
1.3       mlelstv   195:     /dev/rdk5: 0.2MB (500 sectors) block size 4096, fragment size 512
                    196:             using 3 cylinder groups of 0.08MB, 21 blks, 32 inodes.
                    197:            super-block backups (for fsck_ffs -b #) at:
                    198:            32, 200, 368,
                    199: </code></pre>
                    200: 
1.4       mlelstv   201: Older versions of newfs do not support wedgenames, you need to
                    202: specify the device, e.g. /dev/rdk5.
                    203: 
                    204: 
                    205: # GPT
                    206: 
                    207: If wedges could only be created by running a command, they wouldn't be
                    208: useful. But the NetBSD kernel can generate wedges automatically when
                    209: a disk is attached. Most NetBSD architectures will scan for MBR (PC BIOS
                    210: Master Boot Record) and GPT (GUID Partition Table). There is also support
                    211: for standard BSD disklabels but which is currently disabled.
                    212: 
                    213: A GPT can be displayed, created and edited with the GPT command:
                    214: <pre><code>
1.13      mlelstv   215:     % sudo gpt show wd2
1.4       mlelstv   216:             start         size  index  contents
                    217:                 0            1         PMBR
                    218:                 1            1         Pri GPT header
                    219:                 2           32         Pri GPT table
                    220:                34          990         
                    221:              1024  11721043968      1  GPT part - NetBSD ccd component
                    222:       11721044992          143         
                    223:       11721045135           32         Sec GPT table
                    224:       11721045167            1         Sec GPT header
1.13      mlelstv   225:     % sudo gpt show wd1
1.4       mlelstv   226:             start         size  index  contents
                    227:                 0  11721045168         
1.13      mlelstv   228:     % sudo gpt create wd1
                    229:     % sudo gpt show wd1
1.4       mlelstv   230:             start         size  index  contents
                    231:                 0            1         PMBR
                    232:                 1            1         Pri GPT header
                    233:                 2           32         Pri GPT table
                    234:                34  11721045101         
                    235:       11721045135           32         Sec GPT table
                    236:       11721045167            1         Sec GPT header
1.13      mlelstv   237:     % sudo gpt add -a 512k -l hugedisk1 -t ccd wd1
1.4       mlelstv   238:     Partition 1 added, use:
                    239:             dkctl wd1 addwedge <wedgename> 1024 11721043968 <type>
                    240:     to create a wedge for it
1.13      mlelstv   241:     % sudo gpt show wd1
1.4       mlelstv   242:             start         size  index  contents
                    243:                 0            1         PMBR
                    244:                 1            1         Pri GPT header
                    245:                 2           32         Pri GPT table
                    246:                34          990         
                    247:              1024  11721043968      1  GPT part - NetBSD ccd component
                    248:       11721044992          143         
                    249:       11721045135           32         Sec GPT table
                    250:       11721045167            1         Sec GPT header
1.16      mlelstv   251: </code></pre>
1.4       mlelstv   252: 
                    253: Since wedges are currently only created when a device is attached, we
                    254: need either a reboot or some magic using drvctl.
                    255: <pre><code>
1.13      mlelstv   256:     % sudo drvctl -d wd1
                    257:     % sudo drvctl -d wd2
                    258:     % sudo drvctl -a ata_hl -r atabus3
                    259:     % sudo drvctl -a ata_hl -r atabus4
1.16      mlelstv   260: </code></pre>
1.4       mlelstv   261: 
1.22      mlelstv   262: NetBSD-current as a of 20141104 can rescan a device for wedges instead.
                    263: This will delete all unused wedges for a device and readd them according
                    264: to the label.
                    265: <pre><code>
                    266:     % sudo dkctl wd1 makewedges
                    267:     successfully scanned /dev/rwd1d.
                    268:     % sudo dkctl wd2 makewedges
                    269:     successfully scanned /dev/rwd2d.
                    270: </code></pre>
                    271: 
1.4       mlelstv   272: The console or dmesg will reveal that both disks have been reattached
                    273: and wedges have been created:
                    274: <pre><code>
                    275:     wd1 at atabus3 drive 0
                    276:     wd1: <WDC WD60EFRX-68MYMN1>
                    277:     wd1: drive supports 16-sector PIO transfers, LBA48 addressing
                    278:     wd1: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
                    279:     wd1: GPT GUID: 38ba6ff4-e48a-42e4-a513-fe217d7fa013
                    280:     dk5 at wd1: hugedisk1
                    281:     dk5: 11721043968 blocks at 1024, type: ccd
                    282:     wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
                    283:     wd1(ahcisata0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
                    284:     wd2 at atabus4 drive 0
                    285:     wd2: <WDC WD60EFRX-68MYMN1>
                    286:     wd2: drive supports 16-sector PIO transfers, LBA48 addressing
                    287:     wd2: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
                    288:     wd2: GPT GUID: 902717d6-5099-11e4-833b-001cc4d779ed
                    289:     dk6 at wd2: hugedisk2
                    290:     dk6: 11721043968 blocks at 1024, type: ccd
                    291:     wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
                    292:     wd2(ahcisata0:2:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
1.16      mlelstv   293: </code></pre>
1.4       mlelstv   294: 
1.8       mlelstv   295: # CCD
                    296: 
                    297: If the partitions (and wedges) were created with type ffs you could just
                    298: use them to format filesystems and mount them but I created them with
                    299: type ccd to be used by the concatenating disk driver.
                    300: 
                    301: ccd is a very old driver and has some deficiencies. For one, the ccdconfig
1.24      mlelstv   302: tool didn't understand wedge names until very recently.
1.8       mlelstv   303: <pre><code>
1.13      mlelstv   304:    % sudo ccdconfig -c ccd0 16 none NAME=hugedisk1 NAME=hugedisk2
1.8       mlelstv   305:    ccdconfig: NAME=hugedisk1: No such file or directory
1.16      mlelstv   306: </code></pre>
1.8       mlelstv   307: 
1.14      mlelstv   308: ## configuring
1.8       mlelstv   309: For now we use the wedge device directly.
                    310: <pre><code>
1.13      mlelstv   311:     % sudo ccdconfig -c ccd0 16 none /dev/dk5 /dev/dk6
                    312:     % disklabel ccd0
1.8       mlelstv   313:     # /dev/rccd0d:
                    314:     type: ccd
                    315:     disk: ccd
                    316:     label: fictitious
                    317:     flags:
                    318:     bytes/sector: 512
                    319:     sectors/track: 2048
                    320:     tracks/cylinder: 1
                    321:     sectors/cylinder: 2048
                    322:     cylinders: 11446332
                    323:     total sectors: 4294967295
                    324:     rpm: 3600
                    325:     interleave: 1
                    326:     trackskew: 0
                    327:     cylinderskew: 0
                    328:     headswitch: 0           # microseconds
                    329:     track-to-track seek: 0  # microseconds
                    330:     drivedata: 0 
                    331:     
                    332:     4 partitions:
                    333:     #        size    offset     fstype [fsize bsize cpg/sgs]
1.17      mlelstv   334:     a: 4294967295         0     4.2BSD      0     0     0  # (Cyl.      0 - 4294967295*)
                    335:     d: 4294967295         0     unused      0     0        # (Cyl.      0 - 4294967295*)
                    336:     disklabel: boot block size 0
                    337:     disklabel: super block size 0
1.16      mlelstv   338: </code></pre>
1.8       mlelstv   339: 
1.14      mlelstv   340: ## using
1.8       mlelstv   341: Obviously the device that spans two disks is also too large for disklabel.
                    342: What about wedges?
                    343: <pre><code>
1.13      mlelstv   344:     % sudo dkctl ccd0 listwedges
1.8       mlelstv   345:     dkctl: /dev/rccd0d: listwedges: Inappropriate ioctl for device
1.16      mlelstv   346: </code></pre>
1.8       mlelstv   347: 
                    348: The ccd driver does not support wedges at all. But we can still use
                    349: the raw partition.
                    350: <pre><code>
1.13      mlelstv   351:     % sudo newfs -O2 -F -s 23442087936 /dev/rccd0d
1.8       mlelstv   352:     /dev/rccd0d: 11446332.0MB (23442087936 sectors) block size 32768, fragment size 4096
                    353:             using 15420 cylinder groups of 742.31MB, 23754 blks, 46848 inodes.
                    354:     super-block backups (for fsck_ffs -b #) at:
                    355:     192, 1520448, 3040704, 4560960, 6081216, 7601472, 9121728, 10641984, 12162240, 13682496, 15202752, 16723008, 18243264,
                    356:     ......................................................................................................................
1.13      mlelstv   357:     % sudo mount /dev/ccd0d /mnt
                    358:     % df -h /mnt
1.8       mlelstv   359:     Filesystem         Size       Used      Avail %Cap Mounted on
                    360:     /dev/ccd0d          11T       4.0K        10T   0% /mnt
1.13      mlelstv   361:     % sudo umount /mnt
                    362:     % sudo ccdconfig -u ccd0
1.16      mlelstv   363: </code></pre>
1.8       mlelstv   364: 
1.9       mlelstv   365: # Raidframe
                    366: Since ccd wasn't that good, lets create a RAID0 using the raidframe driver.
                    367: 
                    368: First, change the partition types:
                    369: <pre><code>
1.13      mlelstv   370: % sudo gpt type -i 1 -t ccd -T raid wd1
1.9       mlelstv   371: partition 1 type changed
1.13      mlelstv   372: % sudo gpt type -i 1 -t ccd -T raid wd2
1.9       mlelstv   373: partition 1 type changed
1.13      mlelstv   374: % sudo gpt show wd1
1.9       mlelstv   375:         start         size  index  contents
                    376:             0            1         PMBR
                    377:             1            1         Pri GPT header
                    378:             2           32         Pri GPT table
                    379:            34          990         
                    380:          1024  11721043968      1  GPT part - NetBSD RAIDFrame component
                    381:   11721044992          143         
                    382:   11721045135           32         Sec GPT table
                    383:   11721045167            1         Sec GPT header
1.13      mlelstv   384: % sudo gpt show wd2
1.9       mlelstv   385:         start         size  index  contents
                    386:             0            1         PMBR
                    387:             1            1         Pri GPT header
                    388:             2           32         Pri GPT table
                    389:            34          990         
                    390:          1024  11721043968      1  GPT part - NetBSD RAIDFrame component
                    391:   11721044992          143         
                    392:   11721045135           32         Sec GPT table
                    393:   11721045167            1         Sec GPT header
1.16      mlelstv   394: </code></pre>
1.9       mlelstv   395: 
1.22      mlelstv   396: And the reattachment magic (or remake the wedges on recent NetBSD):
1.9       mlelstv   397: <pre><code>
1.13      mlelstv   398:     % sudo drvctl -d wd1
                    399:     % sudo drvctl -d wd2
                    400:     % sudo drvctl -a ata_hl -r atabus3
                    401:     % sudo drvctl -a ata_hl -r atabus4
                    402:     % sudo dkctl wd1 listwedges
1.9       mlelstv   403:     /dev/rwd1d: 1 wedge:
                    404:     dk5: hugedisk1, 11721043968 blocks at 1024, type: raidframe
1.13      mlelstv   405:     % sudo dkctl wd2 listwedges
1.9       mlelstv   406:     /dev/rwd2d: 1 wedge:
                    407:     dk6: hugedisk2, 11721043968 blocks at 1024, type: raidframe
1.16      mlelstv   408: </code></pre>
1.9       mlelstv   409: 
1.10      mlelstv   410: ## configuring
1.9       mlelstv   411: For raidframe we need a configuration file:
                    412: <pre><code>
1.13      mlelstv   413:     % cat raid0.conf
1.9       mlelstv   414:     START array
                    415:     # numRow numCol numSpare
                    416:     1 2 0
                    417:     
                    418:     START disks
                    419:     /dev/dk5
                    420:     /dev/dk6
                    421:     
                    422:     START layout
                    423:     # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_0
                    424:     16 1 1 0
                    425:     
                    426:     START queue
                    427:     fifo 100
1.13      mlelstv   428:     % sudo raidctl -C raid0.conf raid0
                    429:     % sudo raidctl -I 7894737 raid0
                    430:     % sudo raidctl -A yes raid0
1.9       mlelstv   431:     raid0: Autoconfigure: Yes
                    432:     raid0: Root: No
1.13      mlelstv   433:     % disklabel raid0
1.9       mlelstv   434:     # /dev/rraid0d:
                    435:     type: RAID
                    436:     disk: raid
                    437:     label: fictitious
                    438:     flags:
                    439:     bytes/sector: 512
                    440:     sectors/track: 32
                    441:     tracks/cylinder: 8
                    442:     sectors/cylinder: 256
                    443:     cylinders: 91570655
                    444:     total sectors: 4294967295
                    445:     rpm: 3600
                    446:     interleave: 1
                    447:     trackskew: 0
                    448:     cylinderskew: 0
                    449:     headswitch: 0           # microseconds
                    450:     track-to-track seek: 0  # microseconds
                    451:     drivedata: 0 
                    452:     
                    453:     4 partitions:
                    454:     #        size    offset     fstype [fsize bsize cpg/sgs]
1.17      mlelstv   455:     a: 4294967295         0     4.2BSD      0     0     0  # (Cyl.      0 - 4294967295*)
                    456:     d: 4294967295         0     unused      0     0        # (Cyl.      0 - 4294967295*)
1.9       mlelstv   457:     disklabel: boot block size 0
                    458:     disklabel: super block size 0
1.13      mlelstv   459:     % sudo dkctl raid0 listwedges
1.9       mlelstv   460:     /dev/rraid0d: no wedges configured
1.16      mlelstv   461: </code></pre>
1.9       mlelstv   462: 
1.10      mlelstv   463: ## formatting
1.9       mlelstv   464: The raidframe driver has no problems with wedges. We can create a GPT
                    465: on the raid device and create the wedge and format a filesystem.
                    466: This time manually but the reattachment magic would do the same.
                    467: <pre><code>
1.13      mlelstv   468:     % sudo gpt create raid0
                    469:     % sudo gpt add -a 1024 -t ffs -l stripes raid0
1.9       mlelstv   470:     Partition 1 added, use:
                    471:             dkctl raid0 addwedge <wedgename> 34 23442087740 <type>
                    472:     to create a wedge for it
1.13      mlelstv   473:     % sudo dkctl raid0 addwedge stripes 34 23442087740 ffs
1.9       mlelstv   474:     dk10 created successfully.
1.13      mlelstv   475:     % sudo newfs -O2 NAME=stripes
1.9       mlelstv   476:     /dev/rdk10: 11446332.0MB (23442087736 sectors) block size 32768, fragment size 4096
                    477:             using 15420 cylinder groups of 742.31MB, 23754 blks, 46848 inodes.
                    478:     super-block backups (for fsck_ffs -b #) at:
                    479:     192, 1520448, 3040704, 4560960, 6081216, 7601472, 9121728, 10641984, 12162240, 13682496, 15202752, 16723008, 18243264,
                    480:     ......................................................................................................................
1.10      mlelstv   481: </code></pre>
                    482: 
                    483: ## using
1.11      mlelstv   484: Wedges can be mounted by name.
1.10      mlelstv   485: <pre><code>
1.13      mlelstv   486:     % sudo mount NAME=stripes /mnt
                    487:     % df -h /mnt
1.9       mlelstv   488:     Filesystem         Size       Used      Avail %Cap Mounted on
                    489:     /dev/dk10           11T       4.0K        10T   0% /mnt
1.13      mlelstv   490:     % sudo mkdir -m 1777 /mnt/scratch
                    491:     % dd if=/dev/zero bs=1024k count=1000 of=/mnt/scratch/testfile
1.9       mlelstv   492:     1000+0 records in
                    493:     1000+0 records out
                    494:     1048576000 bytes transferred in 4.961 secs (211363837 bytes/sec)
1.13      mlelstv   495:     % sudo umount /mnt
1.16      mlelstv   496: </code></pre>
1.9       mlelstv   497: 
1.11      mlelstv   498: ## checking
1.12      mlelstv   499: And also be checked.
1.11      mlelstv   500: <pre><code>
1.15      mlelstv   501:     % sudo fsck NAME=stripes
                    502:     ** /dev/rdk10
                    503:     ** File system is clean; not checking
1.16      mlelstv   504: </code></pre>
1.11      mlelstv   505: 
1.18      mlelstv   506: Older versions of fsck do not support wedgenames, you need to
1.12      mlelstv   507: specify the device, e.g. /dev/rdk10.
                    508: 
1.10      mlelstv   509: ## and more
1.12      mlelstv   510: Verify that this would reconfigure automatically after a reboot:
1.9       mlelstv   511: <pre><code>
1.13      mlelstv   512:     % sudo raidctl -u raid0
                    513:     % sudo drvctl -d wd1
                    514:     % sudo drvctl -d wd2
                    515:     % sudo drvctl -a ata_hl -r atabus3
                    516:     % sudo drvctl -a ata_hl -r atabus4
                    517:     % sudo raidctl -c raid0.conf raid0
                    518:     % sudo mount NAME=stripes /mnt
                    519:     % df -h /mnt
1.9       mlelstv   520:     Filesystem         Size       Used      Avail %Cap Mounted on
                    521:     /dev/dk10           11T       1.0G        10T   0% /mnt
1.13      mlelstv   522:     % ls -l /mnt/scratch/testfile
1.9       mlelstv   523:     -rw-r--r--  1 mlelstv  wheel  1048576000 Oct 25 21:47 /mnt/scratch/testfile
1.16      mlelstv   524: </code></pre>
1.8       mlelstv   525: 
1.12      mlelstv   526: Unlike wedges, the raidframe devices are not automatically created when a disk attaches.
                    527: That's why we needed the raidctl -c command. The raidframe driver however scans all disks
1.19      mlelstv   528: available when booting and autoconfigures devices for all raidsets found that have the
                    529: autoconfig flag set.
1.12      mlelstv   530: 
1.21      mlelstv   531: # LVM
                    532: 
                    533: Linux LVM is a different scheme to manage disk space, it uses its own label to group
                    534: multiple disks together and to carve out blocks using the device mapper driver to
                    535: form logical volumes.
                    536: 
                    537: The device mapper provides logical block and character devices that route I/O to
                    538: physical disks or some other logical devices. Such devices can be used for filesystems
                    539: like a disk partition or wedge.
                    540: 
                    541: LVM is mostly used on raw disks as there is rarely a necessity to partition a disk first.
                    542: But it can also be used on disk partitions or wedges, this has advantages if the disks are used
                    543: for booting or are moved between different systems or platforms.
                    544: 
                    545: ## LVM on raw disks
                    546: 
                    547: LVM disks are labeled first with the pvcreate command and then coalesced into a volume group.
                    548: <pre><code>
                    549:     % sudo lvm pvcreate /dev/rwd1d
                    550:     Physical volume "/dev/rwd1d" successfully created
                    551:     % sudo lvm pvcreate /dev/rwd2d
                    552:     Physical volume "/dev/rwd2d" successfully created
                    553:     % sudo lvm vgcreate vg0 /dev/rwd1d /dev/rwd2d
                    554:     Volume group "vg0" successfully created
                    555:     % sudo lvm pvs
                    556:     PV         VG   Fmt  Attr PSize PFree
                    557:     /dev/rwd1d vg0  lvm2 a-   2.00t 2.00t
                    558:     /dev/rwd2d vg0  lvm2 a-   2.00t 2.00t
                    559: </code></pre>
                    560: 
1.25      mlelstv   561: This shows a problem with large disks, the LVM tools only understand
                    562: conventional disk partitions that are limited to 2TB each. However,
                    563: neither LVM, the device mapper driver nor the disk drivers when
                    564: using the raw partition are bound to the disklabel information.
                    565: But you need to tell LVM the real size to override the synthesized
                    566: disklabel. The real size is 11721045168 sectors by 512 bytes giving
                    567: 5723166 Megabytes.
1.21      mlelstv   568: <pre><code>
                    569:     % sudo lvm vgremove vg0
                    570:     Volume group "vg0" successfully removed
                    571:     % sudo lvm pvcreate --setphysicalvolumesize=5723166 /dev/rwd1d
                    572:     WARNING: /dev/rwd1d: Overriding real size. You could lose data.
                    573:     Physical volume "/dev/rwd1d" successfully created
                    574:     % sudo lvm pvcreate --setphysicalvolumesize=5723166 /dev/rwd2d
                    575:     WARNING: /dev/rwd2d: Overriding real size. You could lose data.
                    576:     Physical volume "/dev/rwd2d" successfully created
                    577:     % sudo lvm vgcreate vg0 /dev/rwd1d /dev/rwd2d
                    578:     Volume group "vg0" successfully created
                    579:     % sudo lvm pvs
                    580:     PV         VG   Fmt  Attr PSize PFree
                    581:     /dev/rwd1d vg0  lvm2 a-   5.46t 5.46t
                    582:     /dev/rwd2d vg0  lvm2 a-   5.46t 5.46t
                    583: </code></pre>
                    584: 
                    585: ## LVM on wedges
                    586: 
                    587: Here are again the two disks with a large wedge each. The wedge type is unknown
                    588: because the GPT lists the partition as type linux-lvm which has no well-known
                    589: wedge type.
                    590: <pre><code>
                    591:     % sudo dkctl wd1 listwedges
                    592:     /dev/rwd1d: 1 wedge:
                    593:     dk8: hugedisk1, 11721045100 blocks at 34, type: 
                    594:     % sudo dkctl wd2 listwedges
                    595:     /dev/rwd2d: 1 wedge:
                    596:     dk9: hugedisk2, 11721045100 blocks at 34, type: 
                    597: </code></pre>
                    598: 
                    599: Now label the disks, form a volume group, create a logical partition and a filesystem:
                    600: <pre><code>
                    601:     % sudo lvm pvcreate /dev/rdk8
                    602:     Physical volume "/dev/rdk8" successfully created
                    603:     % sudo lvm pvcreate /dev/rdk9
                    604:     Physical volume "/dev/rdk9" successfully created
                    605:     % sudo lvm vgcreate vg0 /dev/rdk8 /dev/rdk9
                    606:     Volume group "vg0" successfully created
                    607:     % sudo lvm pvs
                    608:     PV         VG   Fmt  Attr PSize PFree
                    609:     /dev/rdk8  vg0  lvm2 a-   5.46t 5.46t
                    610:     /dev/rdk9  vg0  lvm2 a-   5.46t 5.46t
                    611:     % sudo lvm lvcreate -L 500m -n lvtest vg0
                    612:     Logical volume "lvtest" created
                    613:     % sudo newfs -O2 /dev/vg0/lvtest
                    614:     /dev/mapper/rvg0-lvtest: 500.0MB (1024000 sectors) block size 8192, fragment size 1024
                    615:     using 11 cylinder groups of 45.46MB, 5819 blks, 10976 inodes.
                    616:     super-block backups (for fsck_ffs -b #) at:
                    617:     144, 93248, 186352, 279456, 372560, 465664, 558768, 651872, 744976, 838080,
                    618:     ...............................................................................
                    619:     % sudo mount /dev/vg0/lvtest /mnt
                    620:     mount_ffs: "/dev/vg0/lvtest" is a non-resolved or relative path.
                    621:     mount_ffs: using "/dev/mapper/vg0-lvtest" instead.
                    622:     % df -h /mnt
                    623:     Filesystem                   Size       Used      Avail %Cap Mounted on
                    624:     /dev/mapper/vg0-lvtest       470M       1.0K       447M   0% /mnt
                    625: </code></pre>
                    626: 
1.25      mlelstv   627: There is one issue with LVM on wedges. LVM scans disks for its
                    628: label to coalesce them into volume groups and to find the logical
                    629: volumes. You can restrict the search in the LVM configuration, but
                    630: if you use wedges, you must scan all wedges as the name of the
                    631: device may change. Also, the result of a scan is saved to optimize
                    632: subsequent scans. If your disk configuration changes, you either
                    633: need to remove the cached result or configure LVM to not save it
                    634: with the write_cache_state option.
1.21      mlelstv   635: 

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb