OpenBSD Journal

New disklabel(8) templates make for a more flexible autoinstall

Contributed by pitrh on from the we slice, we dice, those platters dept.

In a this commit, a first in a series, Henning Brauer (henning@) made disk allocations during automatic installs much more flexible via the introduction of diskablel templates. The matching installer bits came along via this commit by Robert Peichaer (rpe@).

Quoting the updated disklabel(8) man page,

A template for the automatic allocation can be passed to disklabel using -T option.

But the more exciting news is the template format:

The template consists of one line per partition, with each line giving mountpoint, min-max size range, and percentage of disk, space-separated. Max can be unlimited by specifying '*'. If only mountpoint and min size are given, the partition is created with that exact size.

And here's an example of a working disklabel template (stolen from the man page):

/              250M
swap           80-256M 10%
/tmp           120M-4G 8%
/var           80M-4G  13%
/usr           900M-2G 5%
/usr/X11R6     512M-1G 3%
/usr/local     2G-10G  10%
/usr/src       1-2G    2%
/usr/obj       1.3G-2G 4%
/home          1G-*    45%

and here is a sample of output from an autoinstall session using the new feature:

Terminal type? [vt220] vt220
System hostname? (short form, e.g. 'foo') [autoinstalltest] autoinstalltest

Available network interfaces are: vio0 vlan0.
Which network interface do you wish to configure? (or 'done') [vio0] vio0
IPv4 address for vio0? (or 'dhcp' or 'none') [dhcp] dhcp
DHCPREQUEST on vio0 to 255.255.255.255
DHCPACK from 10.1.1.250 (00:15:17:20:bb:94)
bound to 10.1.1.178 -- renewal in 21600 seconds.
IPv6 address for vio0? (or 'rtsol' or 'none') [none] none
Available network interfaces are: vio0 vlan0.
Which network interface do you wish to configure? (or 'done') [done] done
Using DNS domainname bsws.de
Using DNS nameservers at 10.53.0.1 10.53.0.2

Password for root account? 
Public ssh key for root account? [none] no-port-forwarding,no-x11-forwarding ecdsa-sha2-nistp521 AAAAE....
Start sshd(8) by default? [yes] yes
Start ntpd(8) by default? [no] yes
NTP server? (hostname or 'default') [default] ntp.bsws.de
Do you expect to run the X Window System? [yes] no
Change the default console to com0? [yes] yes
Available speeds are: 9600 19200 38400 57600 115200.
Which speed should com0 use? (or 'done') [9600] 9600
Setup a user? (enter a lower-case loginname, or 'no') [no] no

Available disks are: sd0.
Which disk is the root disk? ('?' for details) [sd0] sd0
Use DUIDs rather than device names in fstab? [yes] yes
Disk: sd0	geometry: 6527/255/63 [104857600 Sectors]
Offset: 0	Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
 2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
*3: A6      0   1   2 -   6526 254  63 [          64:   104856191 ] OpenBSD     
Use (W)hole disk, use the (O)penBSD area, or (E)dit the MBR? [OpenBSD] OpenBSD
URL to autopartitioning template for disklabel? [none] http://10.1.1.251/autodisklabel
Fetching http://10.1.1.251/autodisklabel
/dev/rsd0a: 250.0MB in 512000 sectors of 512 bytes
4 cylinder groups of 62.50MB, 4000 blocks, 8064 inodes each
/dev/rsd0g: 1024.0MB in 2097152 sectors of 512 bytes
6 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0d: 5120.0MB in 10485760 sectors of 512 bytes
26 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0e: 1024.0MB in 2097152 sectors of 512 bytes
6 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0f: 2048.0MB in 4194304 sectors of 512 bytes
11 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/sd0a (dc964774ade6fd9f.a) on /mnt type ffs (rw, asynchronous, local)
/dev/sd0g (dc964774ade6fd9f.g) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0d (dc964774ade6fd9f.d) on /mnt/usr type ffs (rw, asynchronous, local, nodev)
/dev/sd0e (dc964774ade6fd9f.e) on /mnt/var type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0f (dc964774ade6fd9f.f) on /mnt/var/log type ffs (rw, asynchronous, local, nodev, nosuid)

Let's install the sets!
Location of sets? (cd0 disk http or 'done') [http] http
HTTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] none
HTTP Server? (hostname, list#, 'done' or '?') [pkg.bsws.de] pkg.bsws.de
Server directory? [pub/OpenBSD/snapshots/amd64] pub/OpenBSD/snapshots/amd64

Select sets by entering a set name, a file name pattern or 'all'. De-select
sets by prepending a '-' to the set name, file name pattern or 'all'. Selected
sets are labelled '[X]'.
    [X] bsd           [X] base57.tgz    [X] game57.tgz    [X] xfont57.tgz
    [X] bsd.rd        [X] comp57.tgz    [X] xbase57.tgz   [X] xserv57.tgz
    [X] bsd.mp        [X] man57.tgz     [X] xshare57.tgz  [ ] site57.tgz
Set name(s)? (or 'abort' or 'done') [done] -all bsd* base* etc* man* site*
    [X] bsd           [X] base57.tgz    [ ] game57.tgz    [ ] xfont57.tgz
    [X] bsd.rd        [ ] comp57.tgz    [ ] xbase57.tgz   [ ] xserv57.tgz
    [X] bsd.mp        [X] man57.tgz     [ ] xshare57.tgz  [X] site57.tgz
Set name(s)? (or 'abort' or 'done') [done] done
Get/Verify SHA256.sig   100% |**************************|  1889       00:00    
Signature Verified
Get/Verify bsd          100% |**************************|  9708 KB    00:00    
Get/Verify bsd.rd       100% |**************************|  7429 KB    00:00    
Get/Verify bsd.mp       100% |**************************|  9748 KB    00:00    
Get/Verify base57.tgz   100% |**************************| 54679 KB    00:04    
Get/Verify man57.tgz    100% |**************************|  8769 KB    00:00    
Get/Verify site57.tgz   100% |**************************| 98142       00:00    
Checksum test for site57.tgz failed. Continue anyway? [no] yes
Unverified sets: site57.tgz. Continue without verification? [no] yes
Installing bsd          100% |**************************|  9708 KB    00:00    
Installing bsd.rd       100% |**************************|  7429 KB    00:00    
Installing bsd.mp       100% |**************************|  9748 KB    00:00    
Installing base57.tgz   100% |**************************| 54679 KB    00:06    
Extracting etc.tgz      100% |**************************|   109 KB    00:00    
Installing man57.tgz    100% |**************************|  8769 KB    00:00    
Installing site57.tgz   100% |**************************| 98142       00:00    
Location of sets? (cd0 disk http or 'done') [done] done

What timezone are you in? ('?' for list) [Europe/Berlin] Europe/Berlin
Saving configuration files...done.
Making all device nodes...done.
Multiprocessor machine; using bsd.mp instead of bsd.

CONGRATULATIONS! Your OpenBSD install has been successfully completed!
To boot the new system, enter 'reboot' at the command prompt.
When you login to your new system the first time, please read your mail
using the 'mail' command.

This will turn up in a snapshot on a mirror near you soon, testing and feedback is most welcome.

(Comments are closed)


Comments
  1. By Anonymous Coward (2601:6:5182:4595:d5d:12fa:8dc2:f311) on


    Excellent! Thank-you!

    The disk allocation was the part of the install I liked the least. This feature will allow me sail through that part of the install.

    Comments
    1. By Jeffrey Forman (jforman) on http://blog.jeffreyforman.net

      >
      > Excellent! Thank-you!
      >
      > The disk allocation was the part of the install I liked the least. This feature will allow me sail through that part of the install.

      I am also excited, partitioning was the one part of the install I had yet to tackle, and was beginning to dig through the code to see what was possible. Hopefully I'll have some time in coming days to test this and provide feedback.

      Comments
      1. By Otto Moerbeek (otto) on http://www.drijf.net

        > >
        > > Excellent! Thank-you!
        > >
        > > The disk allocation was the part of the install I liked the least. This feature will allow me sail through that part of the install.
        >
        > I am also excited, partitioning was the one part of the install I had yet to tackle, and was beginning to dig through the code to see what was possible. Hopefully I'll have some time in coming days to test this and provide feedback.

        Please note that in interactive mode, you can also modify the default auto-generated partitions. The d (delete) and R (resize) commands can be used to tune the layout to your specific needs.



        Comments
        1. By Anonymous Coward (2601:6:5182:4595:39a5:a7df:e515:8bec) on

          > Please note that in interactive mode, you can also modify the default auto-generated partitions.

          I knew that but it was so, well, you know, interactive. :)

        2. By jforman (jforman) on

          > > >
          > > > Excellent! Thank-you!
          > > >
          > > > The disk allocation was the part of the install I liked the least. This feature will allow me sail through that part of the install.
          > >
          > > I am also excited, partitioning was the one part of the install I had yet to tackle, and was beginning to dig through the code to see what was possible. Hopefully I'll have some time in coming days to test this and provide feedback.
          >
          > Please note that in interactive mode, you can also modify the default auto-generated partitions. The d (delete) and R (resize) commands can be used to tune the layout to your specific needs.
          >
          >
          >
          >


          Otto,

          Which snapshot-dated build is the first one expected to have this functionality? I'm trying a pxeboot'ed install using bsd.rd (renamed locally to /tftpd/bsd) from ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/amd64/ expecting the prompt in the course of the install for the autopartition file URL, and am not seeing it. The bsd.rd I am using from the FTP server is dated 2 May, 2015.

          $ md5 bsd
          MD5 (bsd) = a393fe483fe6d7d7dc7f4c77230a1e36

          Comments
          1. By jforman (jforman) on

            > > > >
            > > > > Excellent! Thank-you!
            > > > >
            > > > > The disk allocation was the part of the install I liked the least. This feature will allow me sail through that part of the install.
            > > >
            > > > I am also excited, partitioning was the one part of the install I had yet to tackle, and was beginning to dig through the code to see what was possible. Hopefully I'll have some time in coming days to test this and provide feedback.
            > >
            > > Please note that in interactive mode, you can also modify the default auto-generated partitions. The d (delete) and R (resize) commands can be used to tune the layout to your specific needs.
            > >
            > >
            > >
            > >
            >
            >
            > Otto,
            >
            > Which snapshot-dated build is the first one expected to have this functionality? I'm trying a pxeboot'ed install using bsd.rd (renamed locally to /tftpd/bsd) from ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/amd64/ expecting the prompt in the course of the install for the autopartition file URL, and am not seeing it. The bsd.rd I am using from the FTP server is dated 2 May, 2015.
            >
            > $ md5 bsd
            > MD5 (bsd) = a393fe483fe6d7d7dc7f4c77230a1e36
            >

            Scratch that last comment by me, I spoke to soon, as I realized the snapshot install files (pxeboot, bsd, bsd.rd) were build on 2 May, where as a commit mentioned above was made on 4 May.

            I just now built myself a current-tracking OpenBSD VM and then build a set of release files for that build.

            Using this as my disklabel template:
            / 100M-* 75%
            swap 10M-* 25%

            I was able to do a completely hands-free install!

            Upon the first boot I checked the disklabel to verify the root partition and swap were of expected sizes:

            # disklabel -pm wd0
            # /dev/rwd0c:
            type: ESDI
            disk: ESDI/IDE disk
            label: QEMU HARDDISK
            duid: 692e0bdbd89a82d6
            flags:
            bytes/sector: 512
            sectors/track: 63
            tracks/cylinder: 255
            sectors/cylinder: 16065
            cylinders: 1044
            total sectors: 16777216 # total bytes: 8192.0M
            boundstart: 64
            boundend: 16771860
            drivedata: 0

            16 partitions:
            # size offset fstype [fsize bsize cpg]
            a: 6159.5M 64 4.2BSD 2048 16384 1 # /
            b: 2029.8M 12614720 swap # none
            c: 8192.0M 0 unused

            My math puts that as the correct values. Kudos to all devs involved in this functionality. I look forward to it making it into snapshot builds and eventually 5.8 so I won't always have to track current.

            Comments
            1. By henning (80.171.66.254) on

              > Using this as my disklabel template:
              > / 100M-* 75%
              > swap 10M-* 25%

              sigh. the incredibly stupid "just one big /"-style partitioning is exactly what we don't want to see. not even for tests.

  2. By TOOGAM (TOOGAM) toogmedu@bespin.org on http://toogam.com and http://cyberpillar.com

    > feedback is most welcome.

    I see:
    > URL to autopartitioning template for disklabel? [none] http://10.1.1.251/autodisklabel

    Then, later, I see:

    > HTTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] none
    > HTTP Server? (hostname, list#, 'done' or '?') [pkg.bsws.de] pkg.bsws.de

    If I was using a proxy, I believe I'd like to input that proxy server detail before the installer tries to use the URL to the autopartitioning template.

    If I specify a directory that I don't want created as a separate partition (maybe with a maximum size of zero or negative one), and then use -A (auto allocate), will the installer be able to use the template for any specified mount points, and use automatic allocation for the other standard mount points, but leave off the unwanted one that I specified? (Presumably the specified directory would then be created under its parent, instead of mounted as a mount point.)

    Comments
    1. By Henning Brauer (80.171.69.5) on

      > I see:
      > > URL to autopartitioning template for disklabel? [none] http://10.1.1.251/autodisklabel
      >
      > Then, later, I see:
      >
      > > HTTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] none
      > > HTTP Server? (hostname, list#, 'done' or '?') [pkg.bsws.de] pkg.bsws.de
      >
      > If I was using a proxy, I believe I'd like to input that proxy server detail before the installer tries to use the URL to the autopartitioning template.

      no. The idea is for the autopartitioning template to come from the server that also provided the response file for autoinstall. For the obvious chicken'n'egg-problem, that cannot work via proxy.

      > If I specify a directory that I don't want created as a separate partition (maybe with a maximum size of zero or negative one)


      err, what?

      Comments
      1. By TOOGAM (TOOGAM) on http://toogam.com and http://cyberpillar.com

        > proxy

        I wasn't yet understanding that the article's phrase "autoinstall" was something newly added to OpenBSD 5.5, and so I thought the term was being used as a less precise term. Thank you for providing some clarification. I'm now dropping the commentary/question about an HTML proxy.

        > > If I specify a directory that I don't want created as a separate partition (maybe with a maximum size of zero or negative one)
        >
        > err, what?

        What I particularly had in mind was something like a template that specifies which directory to not make. For example, to not reserve a bunch of space for /home on a server that won't be having end users log into it, maybe the template could say "!/home", or maybe "/home ! !"

        The reference I made to "negative one" was an idea to use a number to flag an undesired value. So, specifying a size of -1 (instead of the exclamation points shown above).

        disklabel -T template -A wd0

        The reason I specify -A is to specify behavior to perform after the template is applied: I don't just want disklabel to stop; I want disklabel to proceed to make the automatically allocated mount points out (some or all) of the remaining space on the disk. But, maybe that is how -T operates anyway, so the -A may be redundant.

        I suppose that I could almost do what is desired with the current format shown, by using:
        "/home *,0 0%"
        I presume that would achieve the goal of not letting a lot of disk space get used up, but I expect it would have some undesirable effects (like reserving a letter, placing a /home in the /etc/fstab file, etc.) I'm simply not wanting one of these mount points to be created (even partially).

        The reason why something like /home is different than a custom location like /mymount, is because /home is one of the default mount points that gets auto-allocated.
        I'm not necessarily wanting to list every default mount point individually, in part because I want the OS to make intelligent decisions like whether there is enough space to create a separate mount point for something like /usr/obj

        Comments
        1. By henning (80.86.183.226) on

          > What I particularly had in mind was something like a template that specifies which directory to not make.

          Why? You just don't specify the mountpoint/partition at all, then.

          > For example, to not reserve a bunch of space for /home on a server that won't be having end users log into it, maybe the template could say "!/home", or maybe "/home ! !"

          Or it could simply not have an entry for /home at all...

          Comments
          1. By TOOGAM (TOOGAM) on http://toogam.com and http://cyberpillar.com

            If disklabel only makes the mount points specified in the template, then the template must specify every desired mount point. If that is the case, then the template ends up overriding *all* of the defaults about whether a mount point exists. It would be useful if the template could override some values, but not others, leaving some flexibility for the disklabel command to automatically determine some of the other values.

            What I've been doing, when installing, is to use the disklabel command's "automatic disk allocation" feature to automatically propose a disklabel, and then I manually edit the results to contain my desired customization(s). Other details are left to the defaults that disklabel decided. That has worked well. If the template requires more of an "all or nothing" approach, which is what I seem to be hearing, then the template code doesn't provide the same level of versatility.

            I do see how the feature saves steps for some other people. It just sounds like the template feature may not save steps for those who are mixing some of these automated decisions with a wish to manually specify just some of the details. Or, maybe some of the details just aren't being clear yet.

            Comments
            1. By henning (80.171.67.142) on

              > If disklabel only makes the mount points specified in the template, then the template must specify every desired mount point.

              yup.
              the manpage is pretty clear about that...

              > If that is the case, then the template ends up overriding *all* of the defaults about whether a mount point exists. It would be useful if the template could override some values, but not others, leaving some flexibility for the disklabel command to automatically determine some of the other values.


              geez. can you please spend some minutes with the disklabel manpage and save us from further speculation?

              disklabel has 4 internal templates (which the manpage clearly describes, minus the last one which is a kind of panic-fallback), chosen based on total disk size. the manpage even tells the disk size -> template mapping. and guess what: on anything but very laughably tiny disks it's the same one, as obvious from the manpage.
              the example template file in the manpage (did I mention the manpage? probably not, so: let me point out the manpage) is exactly that template, with 3 relatively minor differences: / is fixed instead of a range, so that there is an example for a fixed size partition, /home has no upper bound (so there's an example for that), and with the internal template swap and var max sizes are adjusted based on installed physmem during installation. which is obvious from the manpage.

              > What I've been doing, when installing, is to use the disklabel command's "automatic disk allocation" feature to automatically propose a disklabel, and then I manually edit the results to contain my desired customization(s). Other details are left to the defaults that disklabel decided. That has worked well. If the template requires more of an "all or nothing" approach, which is what I seem to be hearing, then the template code doesn't provide the same level of versatility.

              have you even read the template file example in the manpage, quoted in the article?
              you incorporate the equivalent of your modification into the sample template, done.

              > I do see how the feature saves steps for some other people. It just sounds like the template feature may not save steps for those who are mixing some of these automated decisions with a wish to manually specify just some of the details. Or, maybe some of the details just aren't being clear yet.

              the template file is the input to the automatic decisions... as documented in the manpage.

              btw, I recommend reading the manpage.

    2. By rjc (rjc) on

      > I see:
      > > URL to autopartitioning template for disklabel? [none] http://10.1.1.251/autodisklabel
      >
      > Then, later, I see:
      >
      > > HTTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] none
      > > HTTP Server? (hostname, list#, 'done' or '?') [pkg.bsws.de] pkg.bsws.de
      >
      > If I was using a proxy, I believe I'd like to input that proxy server detail before the installer tries to use the URL to the autopartitioning template.

      You are confusing the autoinstall(8) feature with HTTP as the location of the sets - these are two different things altogether.

      Having _your own_ DHCP server configured to point to _your own_ HTTP server *AND* use a proxy, simply doesn't make sense.

      > If I specify a directory that I don't want created as a separate partition (maybe with a maximum size of zero or negative one), and then use -A (auto allocate), will the installer be able to use the template for any specified mount points, and use automatic allocation for the other standard mount points, but leave off the unwanted one that I specified? (Presumably the specified directory would then be created under its parent, instead of mounted as a mount point.)

      You've lost me there but I hope I got the gist of what you were asking for.

      Short answer - if you _don't_ want a directory/mount point created as a separate partition, *don't* specify it in the template!

      Longer: If you use a custom partitioning scheme, the idea is that if you _don't_ specify a mount point, a separate partition *doesn't* get created - it has nothing to do with the creation of standard OpenBSD directory hierarchy - hier(7) - so yes, if you _don't_ specify '/var' mount point to have it created as a separate partition (BTW, *DO NOT DO THAT*), the standard directory hierarchy will still get created on the partition occupied by the parent directory.

  3. By Just Another OpenBSD User (95.42.212.65) on

    (yeah) ...it Keeps getting better, each and every day...

    One step closer to the OpenBSD adoption in the remote computing centers for mere mortals on their behalf, what could possibly go wrong there.

    Well appreciated work as always, much desired by all working with mass deployments after the installation / upgrade templates got in.

    Once again showing the rest of the developer groups how it's done elegantly. An installer can't be that hard yet they keep getting it wrong for decades. Practically we're all drawn to OpenBSD as if by a tractor beam with each release.

    Thanks, Henning! And all developers excercising brainhours on OpenBSD.

    Now how about throwing a nickel, corporation dude?

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]