File:  [NetBSD Developer Wiki] / wikisrc / users / mlelstv / using-large-disks.mdwn
Revision 1.26: download - view: text, annotated - select for diffs
Sun Dec 14 10:26:36 2014 UTC (6 years, 9 months ago) by mlelstv
Branches: MAIN
CVS tags: HEAD
strip comment about cgd

    1: **Contents**
    2: 
    3: [[!toc levels=2]]
    4: 
    5: # What do you do to use a large disk with NetBSD?
    6: 
    7: Here are two:
    8: <pre><code>
    9:     wd1 at atabus3 drive 0
   10:     wd1: <WDC WD60EFRX-68MYMN1>
   11:     wd1: drive supports 16-sector PIO transfers, LBA48 addressing
   12:     wd1: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
   13:     wd1: GPT GUID: 8ee69292-5099-11e4-833b-001cc4d779ed
   14:     wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
   15:     wd1(ahcisata0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
   16:     
   17:     wd2 at atabus4 drive 0
   18:     wd2: <WDC WD60EFRX-68MYMN1>
   19:     wd2: drive supports 16-sector PIO transfers, LBA48 addressing
   20:     wd2: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
   21:     wd2: GPT GUID: 902717d6-5099-11e4-833b-001cc4d779ed
   22:     wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
   23:     wd2(ahcisata0:2:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
   24: </code></pre>
   25:     
   26: # Disklabel
   27: The NetBSD kernel seems to recgonize the disk size fine. But when running
   28: the disklabel program to partition the disk, you get this:
   29: <pre><code>
   30:     % disklabel wd1
   31:     # /dev/rwd1d:
   32:     type: ESDI
   33:     disk: WDC WD60EFRX-68M
   34:     label: fictitious
   35:     flags:
   36:     bytes/sector: 512
   37:     sectors/track: 63
   38:     tracks/cylinder: 16
   39:     sectors/cylinder: 1008
   40:     cylinders: 11628021
   41:     total sectors: 4294967295
   42:     rpm: 3600
   43:     interleave: 1
   44:     trackskew: 0
   45:     cylinderskew: 0
   46:     headswitch: 0           # microseconds
   47:     track-to-track seek: 0  # microseconds
   48:     drivedata: 0 
   49: 
   50:     4 partitions:
   51:     #        size    offset     fstype [fsize bsize cpg/sgs]
   52:     a: 4294967295         0     4.2BSD      0     0     0  # (Cyl.      0 - 4294967295*)
   53:     d: 4294967295         0     unused      0     0        # (Cyl.      0 - 4294967295*)
   54:     disklabel: boot block size 0
   55:     disklabel: super block size 0
   56: </code></pre>
   57: 
   58: While the disk geometry seems to be correct, the sectors and partition
   59: sizes are limited to 4294967295 which is 2^32-1. Older NetBSD kernels
   60: even produce a size from the lower 32bit of the real sector count. This
   61: is because the disklabel data structure has only 32bit fields to store
   62: partition sizes and offsets and the total sector count. This is good
   63: for up to 2TB disks with the standard sector size of 512 bytes.
   64: 
   65: The disklabel is part of the original disk driver interface and
   66: the driver will use that data inside its strategy routine to
   67: validate disk accesses.
   68: 
   69: You can still use the disk somewhat. While normal partitions are
   70: limited by the 32bit values in the disklabel (or even the truncated
   71: values set by older kernels), there is the "raw" partition (on
   72: i386/amd64 that's partition 'd', other archs mostly use 'c') where
   73: the driver ignores the disklabel and validates the access against
   74: an internal value which is not truncated.
   75: 
   76: There are a few obstacles though.
   77: 
   78: ## formatting
   79: <pre><code>
   80:     % sudo newfs /dev/rwd1d
   81:     newfs: /dev/rwd1d partition type is not `4.2BSD'
   82: </code></pre>
   83: 
   84: The raw partition has no correct type and newfs refuses to work, we
   85: have to persuade it a little bit:
   86: <pre><code>
   87:     % sudo newfs -O2 -F -s 11721045168 /dev/rwd1d
   88:     /dev/rwd1d: 5723166.5MB (11721045168 sectors) block size 32768, fragment size 4096
   89:             using 7710 cylinder groups of 742.31MB, 23754 blks, 46848 inodes.
   90:     super-block backups (for fsck_ffs -b #) at:
   91:     192, 1520448, 3040704, 4560960, 6081216, 7601472, 9121728, 10641984, 12162240, 13682496, 15202752,
   92:     .......................................................................................................
   93: </code></pre>
   94: 
   95: We need -O2 for a filesystem > 1TB, -F to prevent newfs from checking
   96: a disklabel and -s to tell it the disk size.
   97: 
   98: ## using
   99: Then we can mount and use the disk:
  100: <pre><code>
  101:     % sudo mount /dev/wd1d /mnt
  102:     % sudo mkdir -m 1777 /mnt/scratch
  103:     % dd if=/dev/zero bs=1024k count=1000 of=/mnt/scratch/testfile
  104:     1000+0 records in
  105:     1000+0 records out
  106:     1048576000 bytes transferred in 10.350 secs (101311690 bytes/sec)
  107:     % ls -la /mnt/scratch/
  108:     total 2048592
  109:     drwxrwxrwt  2 root     wheel         512 Oct 25 17:34 .
  110:     drwxr-xr-x  3 root     wheel         512 Oct 25 17:34 ..
  111:     -rw-r--r--  1 mlelstv  wheel  1048576000 Oct 25 17:35 testfile
  112:     % sudo umount /mnt
  113: </code></pre>
  114: 
  115: ## checking
  116: Filesystem checking needs extra effort:
  117: <pre><code>
  118:     % sudo fsck /dev/rwd1d
  119:     fsck: vfstype `unused' on partition `/dev/rwd1d' is not supported
  120: </code></pre>
  121: 
  122: Again you need to augment information from the disklabel for the raw
  123: partition:
  124: <pre><code>
  125:     % sudo fsck -t ffs /dev/rwd1d
  126:     ** /dev/rwd1d
  127:     ** File system is clean; not checking
  128:     % sudo fsck -t ffs -f /dev/rwd1d
  129:     ** /dev/rwd1d
  130:     ** File system is already clean
  131:     ** Last Mounted on /mnt
  132:     ** Phase 1 - Check Blocks and Sizes
  133:     ** Phase 2 - Check Pathnames
  134:     ** Phase 3 - Check Connectivity
  135:     ** Phase 4 - Check Reference Counts
  136:     ** Phase 5 - Check Cyl groups
  137:     3 files, 256074 used, 1442176277 free (21 frags, 180272032 blocks, 0.0% fragmentation)
  138: </code></pre>
  139: 
  140: ## and more
  141: You can also modify the disklabel, set the raw partition to type 4.2BSD
  142: (including fsize+bsize parameters) and delete the 'a' partition to avoid
  143: overlap warnings. The result looks like:
  144: <pre><code>
  145:     4 partitions:
  146:     #        size    offset     fstype [fsize bsize cpg/sgs]
  147:      d: 4294967295         0     4.2BSD   4096 65536     0  # (Cyl.      0 - 4294967295*)
  148: </code></pre>
  149: 
  150: This makes fsck recognize the disk correctly, newfs still requires the
  151: disksize parameter.
  152: 
  153: One more caveat, when you write the disklabel to the disk, it has to
  154: coexist with the filesystem data. Since the raw partition starts at
  155: offset 0 (the disklabel data is ignored by the driver!), this does not
  156: work with every filesystem, but FFS is safe.
  157: 
  158: # Wedges
  159: 
  160: Wedges solve two problems in NetBSD. They support larger disks without
  161: the 32bit limitation of the disklabel, and they can be used with any
  162: partition information on the disk which makes it possible to exchange
  163: disks between different platforms.
  164: 
  165: Here is a disk with a wedge:
  166: <pre><code>
  167:     % sudo dkctl wd2 listwedges
  168:     /dev/rwd2d: 1 wedge:
  169:     dk6: hugedisk2, 11721043968 blocks at 1024, type: ccd
  170: </code></pre>
  171: 
  172: Wedges can be created and removed at any time with the dkctl command
  173: by specifying a name, start offset on the disk, the wedge size and
  174: a type.
  175: <pre><code>
  176:     % sudo dkctl wd2 addwedge testwedge 34 500 ffs
  177:     dk5 created successfully.
  178:     % sudo dkctl wd2 addwedge testwedge 500 100 unused
  179:     dkctl: /dev/rwd2d: addwedge: Invalid argument
  180:     % sudo dkctl wd2 addwedge testwedge 534 100 unused
  181:     dkctl: /dev/rwd2d: addwedge: File exists
  182:     % sudo dkctl wd2 addwedge testwedge2 534 100 unused
  183:     dk10 created successfully.
  184:     % sudo dkctl wd2 listwedges
  185:     /dev/rwd2d: 3 wedges:
  186:     dk5: testwedge, 500 blocks at 34, type: ffs
  187:     dk10: testwedge2, 100 blocks at 534, type: unused
  188:     dk6: hugedisk2, 11721043968 blocks at 1024, type: ccd
  189: </code></pre>
  190: 
  191: You can see that creation of a wedge is validated, it requires a unique
  192: name and the wedges must not overlap.
  193: <pre><code>
  194:     % sudo newfs NAME=testwedge
  195:     /dev/rdk5: 0.2MB (500 sectors) block size 4096, fragment size 512
  196:             using 3 cylinder groups of 0.08MB, 21 blks, 32 inodes.
  197: 	    super-block backups (for fsck_ffs -b #) at:
  198: 	    32, 200, 368,
  199: </code></pre>
  200: 
  201: Older versions of newfs do not support wedgenames, you need to
  202: specify the device, e.g. /dev/rdk5.
  203: 
  204: 
  205: # GPT
  206: 
  207: If wedges could only be created by running a command, they wouldn't be
  208: useful. But the NetBSD kernel can generate wedges automatically when
  209: a disk is attached. Most NetBSD architectures will scan for MBR (PC BIOS
  210: Master Boot Record) and GPT (GUID Partition Table). There is also support
  211: for standard BSD disklabels but which is currently disabled.
  212: 
  213: A GPT can be displayed, created and edited with the GPT command:
  214: <pre><code>
  215:     % sudo gpt show wd2
  216:             start         size  index  contents
  217:                 0            1         PMBR
  218:                 1            1         Pri GPT header
  219:                 2           32         Pri GPT table
  220:                34          990         
  221:              1024  11721043968      1  GPT part - NetBSD ccd component
  222:       11721044992          143         
  223:       11721045135           32         Sec GPT table
  224:       11721045167            1         Sec GPT header
  225:     % sudo gpt show wd1
  226:             start         size  index  contents
  227:                 0  11721045168         
  228:     % sudo gpt create wd1
  229:     % sudo gpt show wd1
  230:             start         size  index  contents
  231:                 0            1         PMBR
  232:                 1            1         Pri GPT header
  233:                 2           32         Pri GPT table
  234:                34  11721045101         
  235:       11721045135           32         Sec GPT table
  236:       11721045167            1         Sec GPT header
  237:     % sudo gpt add -a 512k -l hugedisk1 -t ccd wd1
  238:     Partition 1 added, use:
  239:             dkctl wd1 addwedge <wedgename> 1024 11721043968 <type>
  240:     to create a wedge for it
  241:     % sudo gpt show wd1
  242:             start         size  index  contents
  243:                 0            1         PMBR
  244:                 1            1         Pri GPT header
  245:                 2           32         Pri GPT table
  246:                34          990         
  247:              1024  11721043968      1  GPT part - NetBSD ccd component
  248:       11721044992          143         
  249:       11721045135           32         Sec GPT table
  250:       11721045167            1         Sec GPT header
  251: </code></pre>
  252: 
  253: Since wedges are currently only created when a device is attached, we
  254: need either a reboot or some magic using drvctl.
  255: <pre><code>
  256:     % sudo drvctl -d wd1
  257:     % sudo drvctl -d wd2
  258:     % sudo drvctl -a ata_hl -r atabus3
  259:     % sudo drvctl -a ata_hl -r atabus4
  260: </code></pre>
  261: 
  262: NetBSD-current as a of 20141104 can rescan a device for wedges instead.
  263: This will delete all unused wedges for a device and readd them according
  264: to the label.
  265: <pre><code>
  266:     % sudo dkctl wd1 makewedges
  267:     successfully scanned /dev/rwd1d.
  268:     % sudo dkctl wd2 makewedges
  269:     successfully scanned /dev/rwd2d.
  270: </code></pre>
  271: 
  272: The console or dmesg will reveal that both disks have been reattached
  273: and wedges have been created:
  274: <pre><code>
  275:     wd1 at atabus3 drive 0
  276:     wd1: <WDC WD60EFRX-68MYMN1>
  277:     wd1: drive supports 16-sector PIO transfers, LBA48 addressing
  278:     wd1: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
  279:     wd1: GPT GUID: 38ba6ff4-e48a-42e4-a513-fe217d7fa013
  280:     dk5 at wd1: hugedisk1
  281:     dk5: 11721043968 blocks at 1024, type: ccd
  282:     wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
  283:     wd1(ahcisata0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
  284:     wd2 at atabus4 drive 0
  285:     wd2: <WDC WD60EFRX-68MYMN1>
  286:     wd2: drive supports 16-sector PIO transfers, LBA48 addressing
  287:     wd2: 5589 GB, 11628021 cyl, 16 head, 63 sec, 512 bytes/sect x 11721045168 sectors
  288:     wd2: GPT GUID: 902717d6-5099-11e4-833b-001cc4d779ed
  289:     dk6 at wd2: hugedisk2
  290:     dk6: 11721043968 blocks at 1024, type: ccd
  291:     wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
  292:     wd2(ahcisata0:2:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133) (using DMA)
  293: </code></pre>
  294: 
  295: # CCD
  296: 
  297: If the partitions (and wedges) were created with type ffs you could just
  298: use them to format filesystems and mount them but I created them with
  299: type ccd to be used by the concatenating disk driver.
  300: 
  301: ccd is a very old driver and has some deficiencies. For one, the ccdconfig
  302: tool didn't understand wedge names until very recently.
  303: <pre><code>
  304:    % sudo ccdconfig -c ccd0 16 none NAME=hugedisk1 NAME=hugedisk2
  305:    ccdconfig: NAME=hugedisk1: No such file or directory
  306: </code></pre>
  307: 
  308: ## configuring
  309: For now we use the wedge device directly.
  310: <pre><code>
  311:     % sudo ccdconfig -c ccd0 16 none /dev/dk5 /dev/dk6
  312:     % disklabel ccd0
  313:     # /dev/rccd0d:
  314:     type: ccd
  315:     disk: ccd
  316:     label: fictitious
  317:     flags:
  318:     bytes/sector: 512
  319:     sectors/track: 2048
  320:     tracks/cylinder: 1
  321:     sectors/cylinder: 2048
  322:     cylinders: 11446332
  323:     total sectors: 4294967295
  324:     rpm: 3600
  325:     interleave: 1
  326:     trackskew: 0
  327:     cylinderskew: 0
  328:     headswitch: 0           # microseconds
  329:     track-to-track seek: 0  # microseconds
  330:     drivedata: 0 
  331:     
  332:     4 partitions:
  333:     #        size    offset     fstype [fsize bsize cpg/sgs]
  334:     a: 4294967295         0     4.2BSD      0     0     0  # (Cyl.      0 - 4294967295*)
  335:     d: 4294967295         0     unused      0     0        # (Cyl.      0 - 4294967295*)
  336:     disklabel: boot block size 0
  337:     disklabel: super block size 0
  338: </code></pre>
  339: 
  340: ## using
  341: Obviously the device that spans two disks is also too large for disklabel.
  342: What about wedges?
  343: <pre><code>
  344:     % sudo dkctl ccd0 listwedges
  345:     dkctl: /dev/rccd0d: listwedges: Inappropriate ioctl for device
  346: </code></pre>
  347: 
  348: The ccd driver does not support wedges at all. But we can still use
  349: the raw partition.
  350: <pre><code>
  351:     % sudo newfs -O2 -F -s 23442087936 /dev/rccd0d
  352:     /dev/rccd0d: 11446332.0MB (23442087936 sectors) block size 32768, fragment size 4096
  353:             using 15420 cylinder groups of 742.31MB, 23754 blks, 46848 inodes.
  354:     super-block backups (for fsck_ffs -b #) at:
  355:     192, 1520448, 3040704, 4560960, 6081216, 7601472, 9121728, 10641984, 12162240, 13682496, 15202752, 16723008, 18243264,
  356:     ......................................................................................................................
  357:     % sudo mount /dev/ccd0d /mnt
  358:     % df -h /mnt
  359:     Filesystem         Size       Used      Avail %Cap Mounted on
  360:     /dev/ccd0d          11T       4.0K        10T   0% /mnt
  361:     % sudo umount /mnt
  362:     % sudo ccdconfig -u ccd0
  363: </code></pre>
  364: 
  365: # Raidframe
  366: Since ccd wasn't that good, lets create a RAID0 using the raidframe driver.
  367: 
  368: First, change the partition types:
  369: <pre><code>
  370: % sudo gpt type -i 1 -t ccd -T raid wd1
  371: partition 1 type changed
  372: % sudo gpt type -i 1 -t ccd -T raid wd2
  373: partition 1 type changed
  374: % sudo gpt show wd1
  375:         start         size  index  contents
  376:             0            1         PMBR
  377:             1            1         Pri GPT header
  378:             2           32         Pri GPT table
  379:            34          990         
  380:          1024  11721043968      1  GPT part - NetBSD RAIDFrame component
  381:   11721044992          143         
  382:   11721045135           32         Sec GPT table
  383:   11721045167            1         Sec GPT header
  384: % sudo gpt show wd2
  385:         start         size  index  contents
  386:             0            1         PMBR
  387:             1            1         Pri GPT header
  388:             2           32         Pri GPT table
  389:            34          990         
  390:          1024  11721043968      1  GPT part - NetBSD RAIDFrame component
  391:   11721044992          143         
  392:   11721045135           32         Sec GPT table
  393:   11721045167            1         Sec GPT header
  394: </code></pre>
  395: 
  396: And the reattachment magic (or remake the wedges on recent NetBSD):
  397: <pre><code>
  398:     % sudo drvctl -d wd1
  399:     % sudo drvctl -d wd2
  400:     % sudo drvctl -a ata_hl -r atabus3
  401:     % sudo drvctl -a ata_hl -r atabus4
  402:     % sudo dkctl wd1 listwedges
  403:     /dev/rwd1d: 1 wedge:
  404:     dk5: hugedisk1, 11721043968 blocks at 1024, type: raidframe
  405:     % sudo dkctl wd2 listwedges
  406:     /dev/rwd2d: 1 wedge:
  407:     dk6: hugedisk2, 11721043968 blocks at 1024, type: raidframe
  408: </code></pre>
  409: 
  410: ## configuring
  411: For raidframe we need a configuration file:
  412: <pre><code>
  413:     % cat raid0.conf
  414:     START array
  415:     # numRow numCol numSpare
  416:     1 2 0
  417:     
  418:     START disks
  419:     /dev/dk5
  420:     /dev/dk6
  421:     
  422:     START layout
  423:     # sectPerSU SUsPerParityUnit SUsPerReconUnit RAID_level_0
  424:     16 1 1 0
  425:     
  426:     START queue
  427:     fifo 100
  428:     % sudo raidctl -C raid0.conf raid0
  429:     % sudo raidctl -I 7894737 raid0
  430:     % sudo raidctl -A yes raid0
  431:     raid0: Autoconfigure: Yes
  432:     raid0: Root: No
  433:     % disklabel raid0
  434:     # /dev/rraid0d:
  435:     type: RAID
  436:     disk: raid
  437:     label: fictitious
  438:     flags:
  439:     bytes/sector: 512
  440:     sectors/track: 32
  441:     tracks/cylinder: 8
  442:     sectors/cylinder: 256
  443:     cylinders: 91570655
  444:     total sectors: 4294967295
  445:     rpm: 3600
  446:     interleave: 1
  447:     trackskew: 0
  448:     cylinderskew: 0
  449:     headswitch: 0           # microseconds
  450:     track-to-track seek: 0  # microseconds
  451:     drivedata: 0 
  452:     
  453:     4 partitions:
  454:     #        size    offset     fstype [fsize bsize cpg/sgs]
  455:     a: 4294967295         0     4.2BSD      0     0     0  # (Cyl.      0 - 4294967295*)
  456:     d: 4294967295         0     unused      0     0        # (Cyl.      0 - 4294967295*)
  457:     disklabel: boot block size 0
  458:     disklabel: super block size 0
  459:     % sudo dkctl raid0 listwedges
  460:     /dev/rraid0d: no wedges configured
  461: </code></pre>
  462: 
  463: ## formatting
  464: The raidframe driver has no problems with wedges. We can create a GPT
  465: on the raid device and create the wedge and format a filesystem.
  466: This time manually but the reattachment magic would do the same.
  467: <pre><code>
  468:     % sudo gpt create raid0
  469:     % sudo gpt add -a 1024 -t ffs -l stripes raid0
  470:     Partition 1 added, use:
  471:             dkctl raid0 addwedge <wedgename> 34 23442087740 <type>
  472:     to create a wedge for it
  473:     % sudo dkctl raid0 addwedge stripes 34 23442087740 ffs
  474:     dk10 created successfully.
  475:     % sudo newfs -O2 NAME=stripes
  476:     /dev/rdk10: 11446332.0MB (23442087736 sectors) block size 32768, fragment size 4096
  477:             using 15420 cylinder groups of 742.31MB, 23754 blks, 46848 inodes.
  478:     super-block backups (for fsck_ffs -b #) at:
  479:     192, 1520448, 3040704, 4560960, 6081216, 7601472, 9121728, 10641984, 12162240, 13682496, 15202752, 16723008, 18243264,
  480:     ......................................................................................................................
  481: </code></pre>
  482: 
  483: ## using
  484: Wedges can be mounted by name.
  485: <pre><code>
  486:     % sudo mount NAME=stripes /mnt
  487:     % df -h /mnt
  488:     Filesystem         Size       Used      Avail %Cap Mounted on
  489:     /dev/dk10           11T       4.0K        10T   0% /mnt
  490:     % sudo mkdir -m 1777 /mnt/scratch
  491:     % dd if=/dev/zero bs=1024k count=1000 of=/mnt/scratch/testfile
  492:     1000+0 records in
  493:     1000+0 records out
  494:     1048576000 bytes transferred in 4.961 secs (211363837 bytes/sec)
  495:     % sudo umount /mnt
  496: </code></pre>
  497: 
  498: ## checking
  499: And also be checked.
  500: <pre><code>
  501:     % sudo fsck NAME=stripes
  502:     ** /dev/rdk10
  503:     ** File system is clean; not checking
  504: </code></pre>
  505: 
  506: Older versions of fsck do not support wedgenames, you need to
  507: specify the device, e.g. /dev/rdk10.
  508: 
  509: ## and more
  510: Verify that this would reconfigure automatically after a reboot:
  511: <pre><code>
  512:     % sudo raidctl -u raid0
  513:     % sudo drvctl -d wd1
  514:     % sudo drvctl -d wd2
  515:     % sudo drvctl -a ata_hl -r atabus3
  516:     % sudo drvctl -a ata_hl -r atabus4
  517:     % sudo raidctl -c raid0.conf raid0
  518:     % sudo mount NAME=stripes /mnt
  519:     % df -h /mnt
  520:     Filesystem         Size       Used      Avail %Cap Mounted on
  521:     /dev/dk10           11T       1.0G        10T   0% /mnt
  522:     % ls -l /mnt/scratch/testfile
  523:     -rw-r--r--  1 mlelstv  wheel  1048576000 Oct 25 21:47 /mnt/scratch/testfile
  524: </code></pre>
  525: 
  526: Unlike wedges, the raidframe devices are not automatically created when a disk attaches.
  527: That's why we needed the raidctl -c command. The raidframe driver however scans all disks
  528: available when booting and autoconfigures devices for all raidsets found that have the
  529: autoconfig flag set.
  530: 
  531: # LVM
  532: 
  533: Linux LVM is a different scheme to manage disk space, it uses its own label to group
  534: multiple disks together and to carve out blocks using the device mapper driver to
  535: form logical volumes.
  536: 
  537: The device mapper provides logical block and character devices that route I/O to
  538: physical disks or some other logical devices. Such devices can be used for filesystems
  539: like a disk partition or wedge.
  540: 
  541: LVM is mostly used on raw disks as there is rarely a necessity to partition a disk first.
  542: But it can also be used on disk partitions or wedges, this has advantages if the disks are used
  543: for booting or are moved between different systems or platforms.
  544: 
  545: ## LVM on raw disks
  546: 
  547: LVM disks are labeled first with the pvcreate command and then coalesced into a volume group.
  548: <pre><code>
  549:     % sudo lvm pvcreate /dev/rwd1d
  550:     Physical volume "/dev/rwd1d" successfully created
  551:     % sudo lvm pvcreate /dev/rwd2d
  552:     Physical volume "/dev/rwd2d" successfully created
  553:     % sudo lvm vgcreate vg0 /dev/rwd1d /dev/rwd2d
  554:     Volume group "vg0" successfully created
  555:     % sudo lvm pvs
  556:     PV         VG   Fmt  Attr PSize PFree
  557:     /dev/rwd1d vg0  lvm2 a-   2.00t 2.00t
  558:     /dev/rwd2d vg0  lvm2 a-   2.00t 2.00t
  559: </code></pre>
  560: 
  561: This shows a problem with large disks, the LVM tools only understand
  562: conventional disk partitions that are limited to 2TB each. However,
  563: neither LVM, the device mapper driver nor the disk drivers when
  564: using the raw partition are bound to the disklabel information.
  565: But you need to tell LVM the real size to override the synthesized
  566: disklabel. The real size is 11721045168 sectors by 512 bytes giving
  567: 5723166 Megabytes.
  568: <pre><code>
  569:     % sudo lvm vgremove vg0
  570:     Volume group "vg0" successfully removed
  571:     % sudo lvm pvcreate --setphysicalvolumesize=5723166 /dev/rwd1d
  572:     WARNING: /dev/rwd1d: Overriding real size. You could lose data.
  573:     Physical volume "/dev/rwd1d" successfully created
  574:     % sudo lvm pvcreate --setphysicalvolumesize=5723166 /dev/rwd2d
  575:     WARNING: /dev/rwd2d: Overriding real size. You could lose data.
  576:     Physical volume "/dev/rwd2d" successfully created
  577:     % sudo lvm vgcreate vg0 /dev/rwd1d /dev/rwd2d
  578:     Volume group "vg0" successfully created
  579:     % sudo lvm pvs
  580:     PV         VG   Fmt  Attr PSize PFree
  581:     /dev/rwd1d vg0  lvm2 a-   5.46t 5.46t
  582:     /dev/rwd2d vg0  lvm2 a-   5.46t 5.46t
  583: </code></pre>
  584: 
  585: ## LVM on wedges
  586: 
  587: Here are again the two disks with a large wedge each. The wedge type is unknown
  588: because the GPT lists the partition as type linux-lvm which has no well-known
  589: wedge type.
  590: <pre><code>
  591:     % sudo dkctl wd1 listwedges
  592:     /dev/rwd1d: 1 wedge:
  593:     dk8: hugedisk1, 11721045100 blocks at 34, type: 
  594:     % sudo dkctl wd2 listwedges
  595:     /dev/rwd2d: 1 wedge:
  596:     dk9: hugedisk2, 11721045100 blocks at 34, type: 
  597: </code></pre>
  598: 
  599: Now label the disks, form a volume group, create a logical partition and a filesystem:
  600: <pre><code>
  601:     % sudo lvm pvcreate /dev/rdk8
  602:     Physical volume "/dev/rdk8" successfully created
  603:     % sudo lvm pvcreate /dev/rdk9
  604:     Physical volume "/dev/rdk9" successfully created
  605:     % sudo lvm vgcreate vg0 /dev/rdk8 /dev/rdk9
  606:     Volume group "vg0" successfully created
  607:     % sudo lvm pvs
  608:     PV         VG   Fmt  Attr PSize PFree
  609:     /dev/rdk8  vg0  lvm2 a-   5.46t 5.46t
  610:     /dev/rdk9  vg0  lvm2 a-   5.46t 5.46t
  611:     % sudo lvm lvcreate -L 500m -n lvtest vg0
  612:     Logical volume "lvtest" created
  613:     % sudo newfs -O2 /dev/vg0/lvtest
  614:     /dev/mapper/rvg0-lvtest: 500.0MB (1024000 sectors) block size 8192, fragment size 1024
  615:     using 11 cylinder groups of 45.46MB, 5819 blks, 10976 inodes.
  616:     super-block backups (for fsck_ffs -b #) at:
  617:     144, 93248, 186352, 279456, 372560, 465664, 558768, 651872, 744976, 838080,
  618:     ...............................................................................
  619:     % sudo mount /dev/vg0/lvtest /mnt
  620:     mount_ffs: "/dev/vg0/lvtest" is a non-resolved or relative path.
  621:     mount_ffs: using "/dev/mapper/vg0-lvtest" instead.
  622:     % df -h /mnt
  623:     Filesystem                   Size       Used      Avail %Cap Mounted on
  624:     /dev/mapper/vg0-lvtest       470M       1.0K       447M   0% /mnt
  625: </code></pre>
  626: 
  627: There is one issue with LVM on wedges. LVM scans disks for its
  628: label to coalesce them into volume groups and to find the logical
  629: volumes. You can restrict the search in the LVM configuration, but
  630: if you use wedges, you must scan all wedges as the name of the
  631: device may change. Also, the result of a scan is saved to optimize
  632: subsequent scans. If your disk configuration changes, you either
  633: need to remove the cached result or configure LVM to not save it
  634: with the write_cache_state option.
  635: 

CVSweb for NetBSD wikisrc <wikimaster@NetBSD.org> software: FreeBSD-CVSweb