Blue Collar Computing

  • bcMPI Home
  • Features
  • License
  • Download
  • Install
  • OSC Installations
  • Tutorial
  • Documentation
  • Help

ParaMake 3

Several environments variables are assumed to be have been defined in the examples below, see the install page for details.

description

ParaMake is the program that knows how to make the packages of the ParaM project. It is loosely based on the pkgsrc package management system from the NetBSD open source operating system.

ParaMake is not a replacement for the traditional make(1) utility. It operates on packages instead of individual objects, and is able to follow a sequence of instructions to automatically extract, patch, configure, build, and install one or more packages. It reads a user specified configuration file to determine the correct options for building each package.

This is the third major revision of ParaMake. The first version was a giant shell script. Version two grew most of the capabilities present in version three, but used Python's ConfigParser library (".ini" files) for its configuration files.

installation

See the bcMPI install instructions for $BUILD_DIR initialization. The paramake3.py executable must be run from the paramake2/bin directory, or through a symlink. If copied to a different directory, it will be unable to include configurations and packages.

usage

To build and install all packages:

% $PYTHON paramake3.py $CONFIG_PY all
      

To build and install one package:

% $PYTHON paramake3.py $CONFIG_PY <package>
      

To specify target(s) for one package:

% $PYTHON paramake3.py $CONFIG_PY <package> {targets}
      

To specify target(s) for all packages:

% $PYTHON paramake3.py $CONFIG_PY all {targets}
      

To specify target(s) and package(s):

% $PYTHON paramake3.py $CONFIG_PY {packages} -- {targets}
      

packages

Each of the entries on the download page is also the name of a package, usually with the common name aliased to a specific version of the package.

One of these special package names may also be given:

all

Substitutes the packages listed in the PACKAGES option in the DEFAULT section.

restart

Continue an "all" build from the given point. Requires the name of the package as the first "target", and optionally the starting target as the second target.

show

Diagnostic display of the current configuration. Targets are "sections", "aliases", "builders", and "config"; default is to show all.

targets

fetch

Automagically download the required files from the Internet and save them to $DOWNLOAD_DIR. Any files which are already present will not be downloaded again. Note that if there was an error downloading, a partial file may remain and must be manually deleted to force a retry.

The download URL is set using the MASTER_SITES option in each package's configuration. This should be a list of prefixes (ending in "/", the distfile name will be appended) or complete URLs. It may also be None if no available URL is suitable for automated download.

See the download page for links to packages that must be manually downloaded.

checksum

Not currently implemented. Should verify the distribution file using a secure hash.

extract

Uncompress and untar the distribution tarballs (for bcMPI sources, create a symlink to the checkout directory). Broken into three sub-targets:

patch

Apply patches to the extracted sources (may create new files). Broken into three sub-targets:

configure

Detect the target compilation environment. Broken into three sub-targets:

build

Compile and link the package. Broken into three sub-targets:

install

Copy files to the installation directory. Broken into three sub-targets:

all-install

All the above targets in sequence: extract, patch, configure, build, install.

clean

Runs "make clean" in the package's source directory. Should only be used by developers.

distclean

Runs "make distclean" in the package's source directory. Should only be used by developers.

delete

Permanently delete ("rm -Rf") the package's extracted source directory. Most often used to return to a consistent starting point after a configuration has been modified, or the bcMPI sources have been updated.

configuration details

The paramake3 configuration files consist of Python code which is dynamically imported and executed by ParaMake. During execution, the configuration files access the global ParaMake object ("pm") to register sections and builders, and to include additional configuration files.

ParaMake

This is the central class in paramake3. A single instance is created in main(), it manages the rest of the objects.

PackageConfig

Each package has its own view into the configuration, managed by a PackageConfig object. Option values are resolved by searching through all sections registered under the package's name. The most recently registered section is searched first, and the global "DEFAULT" sections are appended at the end of the search list. Undefined options will raise an exception.

Each package ideally has four sections: the global "DEFAULT" section defined by paramake (in param-defaults.py), the user's global DEFAULT section, the package's version-specific default, values and the user's overrides for the package.

A package's options are also made available as attributes of the PackageConfig, allowing it to be used for string formatting.

ConfigSection

Option values are usually strings, although any simple Python data structure (list, tuple, dict) may be used. An option may also be a callable (method, lambda, or function), in which case it is called and the result is returned as the option's value. In most cases, string substitutions are much easier to use.

Substitutions allow option values to be built using other options' values, as determined by the dynamic search rules specified above. Values are given as static strings during configuration import, and substituted at lookup time when all sections are available. Substition is performed automatically by the PackageConfig when an option is requested.

Option values may contain two kinds of references to other options in the same PackageConfig. The first method is to use Python's keyword string formatting - all strings (anywhere in the data structure) are substituted with the packages's configuration:

# string formatting pseudo-code
value = pkgconf.get(option)
# assumes value is a string
value = value % pkgconf
return value
      

The second method is to replace a specially formatted string value with another option's value. This allows references to complex non-string structures:

# value substitution pseudo-code
value = pkgconf.get(option)
# assumes value is a string
m = re.match(r'@@(.+)@@$', value)
if m:
    value = pkgconf.get(m.group(1))
return value
      

Options with names ending in "_DIR" or "_PATH" also have "${NAME}" substitutions made with environment variables.

PackageBuilder(Base)

The interfaces for the standard targets are defined in PackageBuilderBase with empty implementations. PackageBuilder implements the primary targets (ie, all but pre_ and post_ methods) by calling GNU make. Most packages' builders will subclass PackageBuilder and optionally implement some pre_ and/or post_ hooks.

config files

The machine specific top level configuration file is passed to paramake3 on the command line. By convention, it first includes the "param-defaults" file:

pm.import_file('<config/param-defaults>')
      

The configuration should then create a "DEFAULT" section:

class default_section(ConfigSection):

    OPENMPI_INSTALL_DIR	= '%(INSTALL_DIR)s'
    MPI_INSTALL_DIR	= '%(OPENMPI_INSTALL_DIR)s'
    OCTAVE_INSTALL_DIR	= '%(INSTALL_DIR)s'

    PACKAGES = (
        'environ',
        'autoconf',
        'automake',
        'libtool',
        'openmpi-kodos-ib',
        'octave-2.9.5-kodos',
        'bcmpi',
        'octmpi',
        )

    LD_LIBRARY_PATH = vflags(
        '%(INSTALL_DIR)s/lib',
        '%(OPENMPI_INSTALL_DIR)s/lib',
        '%(INSTALL_DIR)s/lib/octave-%(OCTAVE_VER)s',
        )

pm.set_section('DEFAULT', default_section)
      

The name of the class is unimportant, it only needs to be unique within the file. This example shows the use of string format substitutions. It also introduces the vflags() utility function, which takes a variable number of arguments and returns them as a list, most commonly to be passed as additional flags on the command line to make.

The next example shows the standard way to add a package to a configuration. The package's default config is imported first so the local section will be searched first. The package's config file is responsible for creating its default section and including a builder file that supports the package version.

pm.import_file('<package/openmpi-1.0.2/config>')

class openmpi_kodos_ib(ConfigSection):

    DISTNAME		= 'openmpi-1.0.2'
    EXTRACT_SUFX	= '.tar.bz2'

    CONFIGURE_FLAGS = vflags(
        '--prefix=%(INSTALL_DIR)s',
        '--enable-debug',
        '--enable-pretty-print-stacktrace',
        '--disable-io-romio',
        '--with-tm=' + '/usr/local/torque-2.0.0p8/',
        '--with-mvapi=' + '/usr/local/ibgd-1.8.0/driver/infinihost',
        )

    CONFIGURE_ENV = kwenv(
        CFLAGS='-g',
        )

pm.set_section('openmpi-kodos-ib', openmpi_kodos_ib)
pm.set_alias('openmpi', 'openmpi-kodos-ib')
      

The kwenv() utility function turns keyword parameters into a dictionary, most commonly used to set environment variables when running make. Finally, an alias is created so that the user can type the common name on the paramake3 command line.