Article

dec2003.tar

Cross-Platform Native Package Creation with EPM

Jeff Layton

I'm a big fan of open source software and also of writing my own scripts and programs to automate things. Currently, I manage three major Unix-ish operating systems at work: Solaris, HP-UX, and Debian GNU/Linux. Tracking all of this added software can be a relatively large burden, especially if it's just copied or unzipped into place. Because of this, I've recently come to favor using the "native" package format for an operating system. If you simply copy or untar files on a machine, or use a non-standard packaging scheme, it's quite difficult to know when you are overwriting a file that is being tracked in the native package catalog, and to uninstall the files later.

Unfortunately, the process to generate a package for each OS often varies greatly between platforms. Generating HP-UX depots is very different from creating a Solaris package. Even on Linux, there are significant differences between the two major package formats (RPMs and debs).

The Easy Software Productions Package Manager (EPM)

Enter EPM, the cross-platform packaging tool from Easy Software Productions (better known for their fabulous CUPS software). EPM can generate binary packages for many Unix and Linux-based operating systems, including AIX, BSD, Tru64, Debian, HP-UX, IRIX, OS-X, Red Hat, and Solaris. It can also build packages in its own format that includes a GUI setup program. In this article, I will focus on creating "native" packages -- those that use the packaging scheme bundled with the OS.

EPM relies on programs that are bundled with an operating system to package software. Building an RPM package means that you need an "rpm" binary, and building a package for Solaris requires the Solaris "pkgmk" tool. Building packages in non-native formats is possible, but only if you have a version of the packaging tool available on the system on which you're building the package. For instance, it's possible to build an RPM package on a Solaris machine, but an installed version of RPM must exist on your Solaris machine.

The EPM distribution is available from:

http://www.easysw.com/epm

EPM is licensed for distribution under the GNU Public License. To begin, download the tarball and unpack it into a directory. It requires an ANSI C compiler, a make program, a Bourne-style shell, and gzip. If you want to build in support for the GUI setup program, you'll also need the FLTK library (http://www.fltk.org).

Building and Installing EPM

EPM is an autoconf-managed project, so look over the INSTALL and README files, and then issue the command:

 % ./configure

Or, do the following if you want to install it in a different location from /usr/local:

% ./configure --prefix=<destination>

Now, build the software with:

% make

Once the software is built, however, do not do the usual "make install". Instead, you can make EPM a package in the native format of the machine with the following command:

% ./epm -f native epm

On some architectures (notably, HP-UX), building packages requires root access, so you may need to be root to execute this command. The "native" keyword here tells EPM that we want to build a package in the native format for the machine on which we're working. It's possible to specify different package formats (e.g., "deb" or "rpm" instead of native), as long as the proper tools for packaging the software exist on the system. See the EPM documentation for a list of keywords for different package types.

After the command completes, the generated package should be inside a directory named after the OS and architecture of the machine (e.g., solaris-2.8-sparc or linux-2.4-intel). Install this package using the native package installation tools for your platform (e.g., dpkg on Debian, or pkgadd on Solaris).

Overview of the Packaging Process

The first thing to do when building a package is to obtain the sources and produce binaries. EPM doesn't compile the software for you; it only packages the files for distribution.

Once you generate all the files you want packaged for distribution, you'll need to write a configuration file that describes your package. These files are called "listfiles" and describe meta-information on the package itself, which files get installed, what their permissions and ownership are, and whether there are scripts that should be executed when the package is installed or removed.

Building a Package

For this example, we will package a copy of OpenSSH (available at http://openssh.com), built for installation in /usr/local, with configuration files in /etc/ssh.

Download the sources for the latest portable distribution and unpack them into a directory. Presuming you have all the necessary dependencies, run:

./configure --prefix=/usr/local --sysconfdir=/etc/ssh
make (or gmake)

Again, we skip the "make install" stage, because we'll be putting the files into a package for distribution.

Creating ListFiles

EPM uses package description files called "listfiles". These files are named with the name of the package and end with the extension ".list". For example, "openssh.list" would be the listfile describing the "openssh" package. Listfiles are documented in epm.list(5).

Package Identification and Information

Every listfile must contain a set of required information. Here's an example of the required info for "openssh.list":

%product OpenSSH
%copyright The OpenSSH Team, All Rights Reserved
%vendor http://www.openssh.com/
%license ./LICENSE
%readme ./README
%description Secure Shell
%version 3.6.1p2 3612

The %product, %copyright, %vendor, and %description are simple text fields that will populate similar fields in the generated package. %license and %readme are names of files that contain their respective information. The %version line has two fields. The first is a human-readable text version. The second is an integer version number. If the integer version number is omitted, EPM will try to generate one from the human-readable field.

Files

Next, we can start adding files and directories to be packaged. The basic format for filesystem entities is:

type mode owner group destination source options

We'll build this package with the build directory of OpenSSH as our current working directory. So, to add entries for the SSH program and its manpage, we'll add these lines to our listfile:

f 755 root sys /usr/local/bin/ssh ./ssh
f 644 root sys /usr/local/man/man1/ssh.1 ssh.1.out nostrip()

By default, EPM runs the strip(1) command on all files that it packages. The strip command removes symbols from object files, thereby reducing the size of binary programs at the expense of sometimes useful debugging information. To prevent this behavior, add the nostrip() option. Since manpages are not binaries, stripping has no effect; but on some platforms, it generates a warning.

Directories

You can also add directories to the package. Most packaging systems will create directories for you if you install a file into a directory that doesn't exist. However, if you want to create empty directories, have control over permissions and ownership, or want the directories removed on package removal, then you must specify directory entries. Here we'll create the directory that will hold the OpenSSH configuration files:

d 755 root sys /etc/ssh -

The "source" attribute for directory entries is always a hyphen, since they aren't actually copied from anything.

Symbolic Links

Symbolic links are prefixed with "l" in the listfile. Here we add the slogin link, which points to the SSH binary:

l 777 root root /usr/local/bin/slogin ssh

The source field for symlinks is interpreted as is, so you can create relative or absolute symlinks.

Configuration Files

Some packaging systems (e.g., Debian and Red Hat) handle configuration files differently from regular files. Others (such as HP-UX and Solaris) make no such distinction. If your packaging system does distinguish between configuration and regular files, then you can declare configuration files like this:

c 644 root root /etc/ssh/ssh_config ssh_config

For packaging systems that make no distinction between the two, configuration files are treated as normal files.

With the above line in your listfile, what happens if this file already exists at install time? It depends on the packaging system used. With Debian, for example, you are presented with a dialog asking whether you want to overwrite the file. On Solaris and HP-UX, this file is treated like any other and is overwritten.

If in doubt, create a test package for your platform and see how it handles overwriting an existing file before relying on this feature.

Variables

Arbitrary variables are defined in listfiles by prefixing them with a dollar sign. Since we'll be installing a lot of our files under /usr/local, we can set a variable to that location and reference it throughout the listfile:

$prefix=/usr/local

We can then reference this as either $prefix or ${prefix}. If you do not use curly braces to reference them, then variables names end at the first slash, hyphen, or whitespace.

If we use the $prefix variable above to describe the installation prefix, our file and symlink entries in the listfile might look like this:

f 755 root sys ${prefix}/bin/ssh ./ssh
f 644 root sys ${prefix}/man/man1/ssh.1 ssh.1.out nostrip()
l 777 root root ${prefix}/bin/slogin ssh

By using variables to describe installation locations, it becomes easy to change the location of package files. Now, if we want to change the package so that it installs under /usr instead of /usr/local, all we need to do is change the $prefix variable.

Variables also make it easy to build listfiles programatically, and when combined with conditionals (described later in this article) make it possible to describe very complex packaging scenarios.

A Simple Example

Listing 1 is an example listfile, describing a basic SSH client package. (Listings for this article are available from the Sys Admin Web site at: http://www.sysadminmag.com.) After building SSH and editing the ssh_config file to your liking, copy this file into the OpenSSH source directory and name it ssh-client.list. Then run:

epm -f native ssh-client

This should build the package and place it into a directory named with a combination of your OS and architecture (e.g., "linux-2.4-intel" or "solaris-2.8-sparc"). You can then install this package using the proper package installation tools.

Scripts

Most packaging systems provide hooks to allow you to run arbitrary code during the installation or removal process. EPM supports this via the following header tags:

%preremove
%postremove
%preinstall
%postinstall

EPM can either be told to run a particular command, or scripts can be embedded in the package. For example, this line in a listfile will have EPM generate an SSH host key after installation:

%postinstall /usr/local/bin/ssh-keygen -q -b 1024 -t dsa -N ''

Only one script tag of each type is allowed, but you can embed scripts that are in other files with a less-than sign:

%postinstall <${srcdir}/generate_keys

Alternatively, you can inline scripts using herefile-like syntax:

%postinstall <<EOT
printf "Generating keys: "
/usr/local/bin/ssh-keygen -q -b 1024 -t dsa -N '' -f etc/ssh/ssh_host_dsa_key
printf "done"
EOT

Package Dependencies

One great thing about using packaging systems is that it facilitates tracking dependencies and incompatibilities. EPM allows you to declare a package requirement using the %requires tag. For example, if we dynamically link OpenSSH, we should require the OpenSSL package to be installed to ensure that it will work. We'll also declare that we need version 0.9.7:

%requires openssl 0.9.7

We may obtain packages from other places that depend on the same software that is in our package, but under a different name. The %provides tag allows us to "alias" our package:

%provides ssh

We can also declare an incompatibility. Here we'll declare that this is incompatible with the kerberized version of SSH:

%incompat sshkrb5

This states that this package can replace a packaged commercial version of SSH that is named "ssh-commercial":

%replaces ssh-commercial

Conditionals

It's possible to have sections of your listfile that are applicable only on certain packaging systems, operating systems, or architectures. The %format tag declares that anything following it is applicable only for the declared packaging system types. The special name "all" refers to all format types. In this example, anything following this line applies only to RPM, deb, and Solaris packages:

%format rpm deb pkg

The %system tag is similar to the %format tag, but it works on the system name as reported by a lowercased uname -s and an optional OS version as given by uname -r. In this example, we change the package dependency on Solaris to require the OpenSSL package provided by sunfreeware.com; everything else will depend on a package called "openssl":

%system solaris
%requires SMCossl
%system !solaris
%requires openssl

There are also a number of conditional tags that can operate on variables:

%if
%ifdef
%elseif
%elseifdef
%else
%endif

These are primarily of use when you have listfiles that are generated through some automatic process.

Init Scripts

EPM treats System V init scripts a little differently. Init scripts are declared much like normal files, but are prefixed with an "i" instead:

i mode user group service-name source ["options"]

The service-name in this case is what the script that is installed in the normal init script directory should be named. This varies by platform, but on Linux and Solaris, it usually gets installed in /etc/init.d, and on HP-UX in /sbin/init.d.

The runlevel() option lets you control what rc directories should get symlinks that point to the init script. Any non-zero runlevel listed will get a start symlink, whereas runlevel 0 will get a stop symlink.

The start() and stop() options allow you to specify at what point in the startup and shutdown process the script should be run. Here's an example:

i 755 root root sshd sshd.rc "runlevel(02) start(82) stop(18)"

This would install the sshd.rc file as "sshd" in the init script directory. It would then create a symlink in the rc0.d directory called K18sshd, and one in the rc2.d directory called S82sshd. There are also some Apple OS X specific options that handle its dependency-oriented init procedure. See the documentation for more info on it.

A Complete Example Package

Listing 2 shows a listfile for a complete OpenSSH package, which uses most of the features of EPM I've covered here. As before, copy this file into the directory where you've built SSH and name it "openssh.list". Edit the configuration files to your liking and add any init scripts that are not part of the distribution. Then run:

epm -f native openssh

It should generate your package and place it into the appropriate directory.

Generating listfile Entries Automatically

Manually configuring listfiles for a large package can be very tedious, so EPM comes with another command called mkepmlist, which will automatically generate listfile entries for everything under a given directory. When generating a package with a lot of files, I often do an initial install in a temporary directory and use mkepmlist to generate the list of files that would be installed. I then do a search and replace to change the locations of the files to their actual location. This is particularly useful when installing a lot of files in a directory shared among many packages (such as /usr or /usr/local).

Conclusion and Resources

EPM has transformed how we approach software maintenance at my site. By making it simple to generate custom packages, we can install software in a consistent manner across many machines as well as track versions and dependencies. This makes it much easier to deploy "standard" machine configurations and ensure consistency across many machines.

The EPM site is the primary site for EPM-related news and information (http://www.easysw.com/epm). It also contains more comprehensive documentation on using EPM.

Major kudos and thanks to the ESP folks for making another great software tool, and for being so helpful via their newsgroups and mailing lists.

Jeff Layton has been working with computers since he got his paws on a TRS-80 in 1981 at age 10. After working in the Macintosh and PC realm for more than a decade and attaining a degree in Physics Education, he came to his senses and made the jump to Unix administration in 1996. He is currently employed as a Senior Unix Systems Administrator at Bowe Bell and Howell.