Cluster Version Management and Patching on UNIX and Linux

important:  

For LSF 7 Update 2 only, you cannot use the steps in this chapter to update your cluster from LSF 7 Update 1 to Update 3. You must follow the steps in "Migrating to LSF Version 7 Update 3 on UNIX and Linux" to manually migrate your LSF 7 cluster to Update 3.

Contents

• Scope
• Patch installation interaction diagram
• Patch rollback interaction diagram
• Version management components
• Version management concepts
• Cluster patching behavior table
• Cluster rollback behavior table
• Version management files
• Version management commands
• Installing update releases on UNIX and Linux
• Installing fixes on UNIX and Linux
• Rolling back patches on UNIX and Linux
• Patching the Oracle database
• Patching the Derby database

Scope

Operating system	Supports UNIX hosts within a single cluster
Limitations	pversions supports LSF Update 1 and later patchinstall supports LSF Update 1 and later For installation of a new cluster, see Installing Platform LSF on UNIX and Linux.

Patch installation interaction diagram

Patches may be installed using the patch installer or LSF installer. The same mechanism is used.

Patch rollback interaction diagram

Use the patch installer to roll back the most recent patch in the cluster.

Version management components

Patches and distributions

Products and versioning

Platform products and components may be separately licensed and versioned. For example, LSF and the HPC Portal are licensed together, but delivered as separate distributions and patched separately.

Product version is a number identifying the release, such as LSF version 7.0.6. The final digit changes whenever you patch the cluster with a new update release.

In addition to the product version, build date, build number, and binary type are used to identify the distributions. Build number may help identify related distributions for different binary types and is important when rolling back the cluster.

Patching the cluster is optional and clusters with the same product version may have different patches installed, so a complete description of the cluster includes information about the patches installed.

Types of distributions

Upgrades, patches, and hot fixes are used to update the software in an existing cluster.

• Product upgrades deliver a new version of the software with valuable new features. Upgrades require a new license.
• Patches deliver small changes and bug fixes that may result in a minor version change. Patches do not require a new license.
• Hot fixes deliver temporary solutions for emergency problems. Hot fixes do not require a new license.

Types of patches

This document describes installing and removing patches. Patches include fixes, fix packs, and update releases. These do not require a new license.

• Update releases-are full distributions available to all customers at regular intervals and include all fixes intended for general use. Your cluster should always use the latest update release. The same package can be used to patch a cluster or create a new cluster. Each update has a different version number (for example, LSF 7 Update 6 is version 7.0.6).
• Fixes-are partial distributions delivered as needed to resolve customer issues (identified by a specific fix number). Platform Support will advise you if you need install any fixes in your cluster. Installing or removing this type of patch does not change the version of the cluster.
• Fix packs (FP)-contain two or more related fixes in one distribution for your convenience.

Version command

The version command pversions is a tool provided to query the patch history and deliver information about cluster and product version and patch levels.

The version command includes functionality to query a cluster or check contents of a package.

The version command is not located with other LSF commands so it may not be in your path. The command location is LSF_TOP/7.0/install/pversions

Data schema update script

The update script for your data schema is a tool provided by Platform to update an existing database before you patch the cluster. The update script modifies the data schema so the database is prepared to handle data from an updated cluster.

Use the script that matches your database and cluster version. A patch may not involve any change to the database, or it may require multiple scripts to update different parts of the database.

Patch installer

The patch installer patchinstall is a tool provided to install patches on an existing cluster.

The patch installer includes functionality to query a cluster, check contents of a package and compatibility with the cluster, and patch or roll back a cluster.

Patch history

History

The patch history is a record of information about patches installed with the patch installer or the LSF installer, including products and patches installed, dates, and location of backups required for rollback purposes.

The pversions command retrieves and displays the version information. The patch installer rollback feature retrieves the backup information.

History directory

The patch history information is kept in the patch history directory. The directory location is LSF_TOP/patch by default.

The patch history directory is configurable during installation. See the PATCH_HISTORY_DIR parameter in install.config.

Patch backups

Backups

The patch installer backs up the current installation before attempting to replace files with the newer versions. The backups are saved so that rollback will be possible later on.

Patches change relatively few files, but for an update release, all the files in the cluster are backed up, so the amount of space required is large. The more patches you install, the more space is required to save multiple backups.

Backup directory

The patch backup files are kept in the patch backup directory. The directory location is LSF_TOP/patch/backup by default.

The patch backup directory is configurable during installation. See the PATCH_BACKUP_DIR parameter in install.config.

Maintenance

Over time, the backups accumulate. You may choose to manually delete old backups, starting with the oldest. Remember that rollback is performed one patch at a time, so your cluster's rollback functionality stops at the point where a backup file is unavailable.

If the backup directory runs out of space, your installations and rollbacks will fail.

You can change your backup directory by setting PATCH_BACKUP_DIR in patch.conf, but you must copy the contents of the old directory to the new directory manually (or there can be no rollback).

Update release backup control

You can disable backups when installing update releases. In this case, your update is installed without backing up the cluster first, so you cannot remove the update using the rollback functionality.

You might choose this feature to save disk space, to speed up the install process, or if you have your own methods of backing up the cluster.

Backup is always done before installing fixes, so you can always roll back if a fix does not behave as expected.w

Multiple daemon files

To make changes without affecting running daemons, the patch installer must move some files to another directory instead of overwriting.

For each file, a new directory is created in parallel with the file. The directory is called daemons_old.

Running jobs may require the old files even after you restart the updated cluster.

Version management concepts

Multiple distributions

Like installation, patching the cluster sometimes requires you to download packages for each binary type or product component.

For example, to install an update, you may need to download multiple patches, such as distributions for LSF and distributions for the HPC Portal.

Depending on the problem, a fix or fix pack may involve changes affecting just one binary type, or multiple distributions to patch multiple binary types.

Order of installation

If you have to install multiple patches, start with the most recent update, which includes all previous fixes. Install on all UNIX hosts to bring the whole cluster up to date. Then install fixes or fix packs as needed.

Installers

The LSF installer installs full distributions and can modify configuration. The LSF installer incorporates the patch installer so the process of updating the files is the same as the patch installer. However, the LSF installer should be used to install an update because the update may require configuration changes that lsfinstall can do automatically.

The patch installer installs all patches and never modifies configuration. A partial distribution (FP or fix) can only be installed by the patch installer.

Patch installer accessibility

For clusters version 7.0 or earlier, you must obtain the patch installer separately from Platform, and run the patchinstall command from your download directory.

For clusters version 7 Update 1 (7.0.1) or later, the patch installer is available under install directory under the LSF installation directory. This location may not be in your path, so run the patchinstall command from this directory (LSF_TOP/7.0/install/patchinstall).

Version command accessibility

For clusters version 7.0 or earlier, the version command is not available.

For clusters version 7 Update 1 (7.0.1) or later, the command is available under install directory under the LSF installation directory (LSF_TOP/7.0/install/pversions). It is not located with other LSF commands so it may not be in your path by default.

lsfinstall and install.config versions

The LSF installer may change with each update. You should not install a new update using the old lsfinstall program or install.config template. To properly update your cluster with new parameters and new configuration, make sure your installers match the version of the distribution you are installing.

Command environment

Both patchinstall and pversions on UNIX need environment information to identify your cluster.

Before you run the command, set your environment using profile.lsf or cshrc.lsf. You may have already done this to administer your cluster.

As a workaround, you may use the -f option in the command line and specify a file that defines your environment. For more information, see the command reference.

Silent install

The silent install option is used for automated installations.

For lsfinstall, enable silent install by the LSF_QUIET_INST parameter in install.config. Silent install hides some messages.

For patchinstall, enable silent install by the --silent option in the command line. Silent install shows all messages but does not prompt for confirmations.

Windows-UNIX clusters and Windows clusters

If your cluster has both Windows and UNIX, patch the UNIX hosts in the cluster using the patch installer. Patch the Windows hosts using Windows tools.

The Windows patch files should be installed in order from oldest to newest on every Windows host if you have more than one to install.

To install a Windows patch, double click the .msp file for the OS you want and follow the wizard. You may be asked to reboot after installing. Follow the Windows prompts if applicable.

tip:  

You can also install silently.

Cluster patching behavior table

When ...	Actions...	The result ...
Normal behavior.	The installer replaces current files with new.	Success, cluster is updated.
Installing an update and the patch history is missing (files are not found in the directory defined by the parameter PATCH_HISTORY_DIR in patch.conf)	The installer creates new history files in the directory. The installer cannot determine compatibility but installs anyway because an update is a full distribution.	Cluster is modified but if the update is not compatible (a previous version instead of newer version), the cluster may not work properly.
Installing a fix and the patch history is missing (files are not found in the directory defined by the parameter PATCH_HISTORY_DIR in patch.conf)	For a fix, the installer cannot determine compatibility.	No update, cluster remains in same state Error presented on screen and logged in patch.log and patch.err
The installer is partway through the installation when there is a problem. The cluster contains some older files and some newer files.	If the installer cannot complete, it reverses the update actions, removing the newer files and returning the older ones.	No update, cluster remains in same state. Error presented on screen and logged
Installing a fix and a file in the cluster is newer than the file in the patch (build number in cluster is larger than build number of patch).	Prompt user to overwrite or preserve file. Install other files in the patch as usual.	Each build of a file is backwards compatible, so this patch works properly with the newer file. Overwriting the newer file may break functionality of a newer patch in the cluster.
Installing a fix and a file in the cluster has been modified since the last patch (current file size does not match size recorded in patch history).	Prompt user to overwrite or exit.	Overwriting a corrupt file will result in correct behavior. Overwriting a customized file will break existing functionality. You can modify the updated file manually after installation. Patch functionality depends on updated content in the new file, so you cannot install the patch if you do not overwrite the file.

Cluster rollback behavior table

When ...	Actions...	The result ...
Normal behavior.	The installer replaces current files with previous backup.	Success, cluster reverts to previous state.
The patch history is missing (files are not found in the directory defined by the parameter PATCH_HISTORY_DIR in patch.conf)	Without the history, the installer cannot determine which backups to use. Since there is nothing to replace them with, the installer does not remove the current files.	No rollback, cluster remains in same state. Error presented on screen and logged
You did not specify the most recent patch.	The history indicates that the patch is not the newest backup. The installer must use the most recent backup to roll back.	No rollback, cluster remains in same state. Error presented on screen and logged
The backups are missing (expected files are not found in the directory defined by the parameter PATCH_BACKUP_DIR in patch.conf).	Since there is nothing to replace them with, the installer does not remove the current files.	No rollback, cluster remains in same state. Error presented on screen and logged
The installer is partway through the roll back when there is a problem. The cluster contains some older files and some newer files.	If the installer cannot complete, it reverses the rollback actions, removing the older files and returning the newer ones.	No rollback, cluster remains in same state. Error presented on screen and logged

Version management files

Logs

File	Description
patch.log	This file: Created by the patch installer (not created if you use lsfinstall) Created when you install a patch or update release Created in current working directory (or if you do not have write permission there, logs to /tmp) Logs installation steps
precheck.log	This file: Created by the patch installer Created when you install or check a patch Created in current working directory (or if you do not have write permission there, logs to /tmp) Logs precheck steps
Install.log	This file: Created by the LSF installer (not created if you use patchinstall) Created when you install a new cluster or update release Created in current working directory (or if you do not have write permission there, logs to /tmp) Logs installation steps

Version management commands

Commands to modify cluster

Command	Description
lsfinstall	This command: Creates a new cluster (using any full distribution including update releases) Patches a cluster with an update release (a full distribution) by installing binaries and updating configuration
patchinstall	This command: Patches a cluster by installing binaries from a full or partial distribution (does not update configuration, so lsfinstall is recommended for an update release)
patchinstall -r	This command Rolls back a cluster by removing binaries (does not roll back configuration, so rollback of updates may not be recommended)

Commands to monitor cluster

Command	Description
pversions	This command: Displays product version information for the entire cluster, including patch levels Displays detailed information for specific builds or files in the cluster; for example, see what files were modified after installing a patch
file_name -V	This command: Displays detailed information for a specific file in the cluster (specify the installed file, for example lim -V)

Commands to check uninstalled packages

Command	Description
pversions -c	This command: Displays detailed information about the contents of an uninstalled package
patchinstall -c	This command: Tests if an uninstalled package is compatible with the cluster

Installing update releases on UNIX and Linux

To install an update release to the cluster.

important:  

For LSF 7 Update 3, you cannot use the steps in this section to update your cluster from LSF 7 Update 1 to Update 3. You must follow the steps in "Migrating to LSF Version 7 Update 3 on UNIX and Linux" to manually migrate your LSF 7 cluster to Update 3.

1. If you need to patch the reporting database, download the corresponding database update scripts and update the database schema first.
2. Download and extract the new version of lsfinstall.

For example,

zcat lsf7Update6_lsfinstall.tar.Z | tar xvf - 

3. Prepare the install.config file using the new template and information from your original installation. The new template may have new parameters for you to set.
4. Download the patches. If hosts in your cluster have multiple binary types, you may require multiple distribution files to patch the entire cluster. Put the distribution files in the same directory as lsfinstall.
5. Run the new LSF installer.

For example,

lsfinstall -f install.config 
 

Specify the patches to install and let the installer finish.

6. Restart the cluster.

This will make changes to daemons take effect.

7. Optional. Run pversions to determine the state of the cluster.
8. Optional. Free some space by deleting the contents of backup directories under EGO and LSF installation directories.

Installing fixes on UNIX and Linux

To install fixes or fix packs to update the cluster.

1. To patch the reporting database, download the corresponding database update scripts and update the database schema first.
2. Download the patches from Platform. If hosts in your cluster have multiple binary types, you may require multiple distribution files to patch the entire cluster.

Put the distribution files on any host.

For example,

//HostB/downloads/pkg1 
//HostB/downloads/pkg2 

3. Log on to a host in the cluster.
4. Set your environment (if you cannot do this, prepare a configuration file and use the -f option in the pversions and patchinstall commands).

source LSF_TOP/conf/cshrc.lsf (for csh or tcsh)  
. LSF_TOP/conf/profile.lsf (for sh, ksh, or bash)  

5. Run the patch installer tool and specify the patches to install.

For example,

LSF_TOP/7.0/install/patchinstall //HostB/downloads/pkg1 
//HostB/downloads/pkg2 
 

Let the patch installer finish.

6. If you were prompted to do so, restart the cluster.

Patches that affect running daemons require you to restart manually.

7. Optional. Run LSF_TOP/7.0/install/pversions to determine the state of the cluster.
8. Optional. If you were prompted to restart the cluster and have done so, you can free some space by deleting the contents of backup directories under EGO and LSF installation directories.

Rolling back patches on UNIX and Linux

To remove patches that you installed using patchinstall, and return the cluster to a previous state.

1. Log on to a host in the cluster.
2. Set your environment (if you cannot, prepare a configuration file and use -f option in pversions and patchinstall commands).

source LSF_TOP/conf/cshrc.lsf (for csh or tcsh)  
. LSF_TOP/conf/profile.lsf (for sh, ksh, or bash)  

3. Run LSF_TOP/7.0/install/pversions to determine the state of the cluster and find the build number of the last patch installed (roll back one patch at a time).
4. Run patchinstall with -r and specify the build number of the last patch installed (the patch to be removed).

patchinstall -r 12345

5. If you were prompted to do so, restart the cluster.

Patches that affect running daemons require you to restart manually.

6. If necessary, modify LSF cluster configuration manually. This may be necessary to roll back an update.
7. Optional. Run LSF_TOP/7.0/install/pversions to determine the state of the cluster.

To roll back multiple builds, repeat as required until the cluster is in the state you want. The database schema is backwards compatible, so you do not need to change the reporting database.

Patching the Oracle database

Prerequisites: The Oracle database is properly configured and running:

• You have a user name, password, and URL to access the database server.
• You installed the latest JDBC driver (ojdbc14.jar or newer) for the Oracle database. This driver is available from the following URL:
• http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/index.html
• You are able to run sqlplus.

To patch the reporting database as part of patching the cluster, get the corresponding database update scripts and update the database schema first.

1. When you download the patches for your cluster, download the corresponding database update scripts from Platform.
2. In the command console, open the database schema directory.

cd LSF_TOP/perf/lsf/version/DBschema/Oracle

3. Run the scripts to create a database schema.

sqlplus user_name/password@connect_string @update_script

where

• user_name is the user name on the database server
• password is the password for this user name on the database server
• connect_string is the named SQLNet connection for this database
• update_script is the name of the patch script

Patching the Derby database

Prerequisites: The Derby database is properly configured and running:

To patch the reporting database as part of patching the cluster, get the corresponding database update scripts and update the database schema first.

1. When you download the patches for your cluster, download the corresponding database update scripts from Platform.
2. In the command console, open the database schema directory.

cd LSF_TOP/perf/lsf/version/DBschema/Derby

3. Start the ij tool.

ij.sh

4. Connect to the database.

connect 'jdbc:derby://host_name:port/db_name;user=user_name;password=password;'

where

• host_name is the Derby database host
• port is the Derby database port, 1527 by default
• db_name is the database name, app by default
• user_name is the database user name, app by default
• password is the database login password, app by default
5. Run the scripts to create a database schema.

run 'update_script' 
 

where

• update_script is the full path to the patch script