Chapter 7. Creating binary packages for everything in pkgsrc (bulk builds)

Table of Contents

7.1. Think first, build later
7.2. Requirements of a bulk build
7.3. Running a pbulk-style bulk build
7.3.1. Configuration
7.4. Creating a multiple CD-ROM packages collection
7.4.1. Example of cdpack

When you have multiple machines that should run the same packages, it is wasted time if they all build their packages themselves from source. There is a ways of getting a set of binary packages: The bulk build system, or pbulk ("p" stands for "parallel). This chapter describes how to set it up so that the packages are most likely to be usable later.

7.1. Think first, build later

Since a bulk build takes several days or even weeks to finish, you should think about the setup before you start everything. Pay attention to at least the following points:

  • If you want to upload the binary packages to ftp.NetBSD.org, make sure the setup complies to the requirements for binary packages:

    • To end up on ftp.NetBSD.org, the packages must be built by a NetBSD developer on a trusted machine (that is, where you and only you have root access).

    • Packages on ftp.NetBSD.org should only be created from the stable branches (like 2009Q1), so that users browsing the available collections can see at a glance how old the packages are.

    • The packages must be built as root, since some packages require set-uid binaries at runtime, and creating those packages as unprivileged user doesn't work well at the moment.

  • Make sure that the bulk build cannot break anything in your system. Most bulk builds run as root, so they should be run at least in a chroot environment or something even more restrictive, depending on what the operating system provides. There have been numerous cases where certain packages tried to install files outside the LOCALBASE or wanted to edit some files in /etc. Furthermore, the bulk builds install and deinstall packages in /usr/pkg (or whatever LOCALBASE is) during their operation, so be sure that you don't need any package during the build.

7.2. Requirements of a bulk build

A complete bulk build requires lots of disk space. Some of the disk space can be read-only, some other must be writable. Some can be on remote filesystems (such as NFS) and some should be local. Some can be temporary filesystems, others must survive a sudden reboot.

  • 40 GB for the distfiles (read-write, remote, temporary)

  • 30 GB for the binary packages (read-write, remote, permanent)

  • 1 GB for the pkgsrc tree (read-only, remote, permanent)

  • 5 GB for LOCALBASE (read-write, local, temporary)

  • 10 GB for the log files (read-write, remote, permanent)

  • 5 GB for temporary files (read-write, local, temporary)

7.3. Running a pbulk-style bulk build

Running a pbulk-style bulk build works roughly as follows:

  • First, build the pbulk infrastructure in a fresh pkgsrc location.

  • Then, build each of the packages from a clean installation directory using the infrastructure.

7.3.1. Configuration

To simplify configuration we provide helper script mk/pbulk/pbulk.sh.

In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine). Configure network access to fetch distribution files. Create user with name "pbulk".

Fetch and extract pkgsrc. Use a command like one of these:

# (cd /usr && ftp -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
# (cd /usr && fetch -o - http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
# (cd /usr && cvs -Q -z3 -d anoncvs@anoncvs.netbsd.org:/cvsroot get -P pkgsrc)

Or any other way that fits (e.g., curl, wget).

Deploy and configure pbulk tools, e.g.:

# sh pbulk.sh -n # native (NetBSD)
# sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment
# sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build

Note

mk.conf.frag is a fragment of mk.conf that contains settings you want to apply to packages you build. For instance,

PKG_DEVELOPER=		yes		# perform more checks
X11_TYPE=		modular		# use pkgsrc X11
SKIP_LICENSE_CHECK=	yes		# accept all licences (useful when building all packages)

If configured for limited list, replace the list in /usr/pbulk/etc/pbulk.list with your list of packages one per line without empty lines or comments. E.g.:

www/firefox
mail/thunderbird
misc/libreoffice4

At this point you can also review configuration in /usr/pbulk/etc and make final amendments, if wanted.

Start it:

# /usr/pbulk/bin/bulkbuild

After it finishes, you'll have /mnt filled with distribution files, binary packages, and reports, plain text summary in /mnt/bulklog/meta/report.txt

Note

The pbulk.sh script does not cover all possible use cases. While being ready to run, it serves as a good starting point to understand and build more complex setups. The script is kept small enough for better understanding.

Note

The pbulk.sh script supports running unprivileged bulk build and helps configuring distributed bulk builds.

7.4. Creating a multiple CD-ROM packages collection

After your pkgsrc bulk-build has completed, you may wish to create a CD-ROM set of the resulting binary packages to assist in installing packages on other machines. The pkgtools/cdpack package provides a simple tool for creating the ISO 9660 images. cdpack arranges the packages on the CD-ROMs in a way that keeps all the dependencies for a given package on the same CD as that package.

7.4.1. Example of cdpack

Complete documentation for cdpack is found in the cdpack(1) man page. The following short example assumes that the binary packages are left in /usr/pkgsrc/packages/All and that sufficient disk space exists in /u2 to hold the ISO 9660 images.

# mkdir /u2/images
# pkg_add /usr/pkgsrc/packages/All/cdpack
# cdpack /usr/pkgsrc/packages/All /u2/images
      

If you wish to include a common set of files (COPYRIGHT, README, etc.) on each CD in the collection, then you need to create a directory which contains these files. e.g.

# mkdir /tmp/common
# echo "This is a README" > /tmp/common/README
# echo "Another file" > /tmp/common/COPYING
# mkdir /tmp/common/bin
# echo "#!/bin/sh" > /tmp/common/bin/myscript
# echo "echo Hello world" >> /tmp/common/bin/myscript
# chmod 755 /tmp/common/bin/myscript
      

Now create the images:

# cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images

Each image will contain README, COPYING, and bin/myscript in their root directories.