Specifying Resource Requirements

Contents

• About Resource Requirements
• Queue-level Resource Requirements
• Job-level Resource Requirements
• About Resource Requirement Strings
• Selection String
• Order String
• Usage String
• Span String
• Same String
• Compute Unit String

About Resource Requirements

Resource requirements define which hosts a job can run on. Each job has its resource requirements and hosts that match the resource requirements are the candidate hosts. When LSF schedules a job, it uses the load index values of all the candidate hosts. The load values for each host are compared to the scheduling conditions. Jobs are only dispatched to a host if all load values are within the scheduling thresholds.

By default, if a job has no resource requirements, LSF places it on a host of the same type as the submission host (i.e., type==local). However, if a job has string or Boolean resource requirements specified and the host type has not been specified, LSF places the job on any host (i.e., type==any) that satisfies the resource requirements.

To override the LSF defaults, specify resource requirements explicitly. Resource requirements can be set for queues, for application profiles, or for individual jobs.

To best place a job with optimized performance, resource requirements can be specified for each application. This way, you do not have to specify resource requirements every time you submit a job. The LSF administrator may have already configured the resource requirements for your jobs, or you can put your executable name together with its resource requirements into your personal remote task list.

The bsub command automatically uses the resource requirements of the job from the remote task lists.

A resource requirement is an expression that contains resource names and operators.

Compound Resource Requirements

In some cases different resource requirements may apply to different parts of a parallel job. The first execution host, for example, may require more memory or a faster processor for optimal job scheduling. Compound resource requirements allow you to specify different requirements for some slots within a job in the queue-level, application-level, or job-level resource requirement string.

Compound resource requirement strings can be set by the application-level or queue-level RES_REQ parameter, or used with bsub -R when a job is submitted. bmod -R also accepts compound resource requirement strings for both pending and running jobs.

Special rules take effect when compound resource requirements are merged with resource requirements defined at more than one level. If a compound resource requirement is used at any level (job, application, or queue) the compound multi-level resource requirement combinations described later in this chapter apply.

restriction:  

Compound resource requirements cannot contain cu sections, multiple -R options, or the || operator.

Resizable jobs cannot have compound resource requirements.

Resource allocation for parallel jobs using compound resources is done for each compound resource term in the order listed instead of considering all possible combinations. A host rejected for not satisfying one resource requirement term will not be reconsidered for subsequent resource requirement terms.

Compound resource requirements were introduced in LSF Version 7 Update 5, and are not compatible with earlier versions of LSF.

Resource requirements in application profiles

See Chapter 23, "Working with Application Profiles" for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.

Resizable jobs and resource requirements

In general, resize allocation requests for resizable jobs use the resource requirements of the running job. When the resource requirement string for a job is modified with bmod -R, the new string takes effects for a job resize request. The resource requirement of the allocation request is merged from resource requirements specified at the queue, job, and application levels.

restriction:  

Autoresizable jobs cannot have compute unit resource requirements. Any autoresizable jobs switched to queues with compute unit resource requirements will no longer be autoresizable.

Resizable jobs cannot have compound resource requirements.

Queue-level Resource Requirements

Each queue can define resource requirements that apply to all the jobs in the queue.

When resource requirements are specified for a queue, and no job-level or application profile resource requirement is specified, the queue-level resource requirements become the default resource requirements for the job.

Resource requirements determined by the queue no longer apply to a running job after running badmin reconfig, For example, if you change the RES_REQ parameter in a queue and reconfigure the cluster, the previous queue-level resource requirements for running jobs are lost.

Syntax

The condition for dispatching a job to a host can be specified through the queue-level RES_REQ parameter in the queue definition in lsb.queues. Queue-level RES_REQ rusage values must be in the range set by RESRSV_LIMIT (set in lsb.queues), or the queue-level RES_REQ is ignored.

Examples

RES_REQ=select[((type==LINUX2.4 && r1m < 2.0)||(type==AIX && r1m < 1.0))] 

This allows a queue, which contains LINUX2.4 and AIX hosts, to have different thresholds for different types of hosts.

RES_REQ=select[((hname==hostA && mem > 50)||(hname==hostB && mem > 100))] 

Using the hname resource in the resource requirement string allows you to set up different conditions for different hosts in the same queue.

Load thresholds

Load thresholds can be configured by your LSF administrator to schedule jobs in queues. Load thresholds specify a load index value.

loadSched

The scheduling threshold that determines the load condition for dispatching pending jobs. If a host's load is beyond any defined loadSched, a job is not started on the host. This threshold is also used as the condition for resuming suspended jobs.

loadStop

The suspending condition that determines when running jobs should be suspended.

Thresholds can be configured for each queue, for each host, or a combination of both. To schedule a job on a host, the load levels on that host must satisfy both the thresholds configured for that host and the thresholds for the queue from which the job is being dispatched.

The value of a load index may either increase or decrease with load, depending on the meaning of the specific load index. Therefore, when comparing the host load conditions with the threshold values, you need to use either greater than (>) or less than (<), depending on the load index.

See Chapter 37, "Load Thresholds" for information about suspending conditions and configuring load thresholds.

View queue-level resource requirements

1. Use bqueues -l to view resource requirements (RES_REQ) defined for the queue:

bqueues -l normal 
QUEUE: normal 
 -- No description provided.  This is the default queue. 
... 
RES_REQ:  select[type==any] 
rusage[mem=10,dynamic_rsrc=10:duration=2:decay=1] 
... 

Job-level Resource Requirements

Each job can specify resource requirements. Job-level resource requirements override any resource requirements specified in the remote task list.

In some cases, the queue specification sets an upper or lower bound on a resource. If you attempt to exceed that bound, your job will be rejected.

Syntax

To specify resource requirements for your job, use bsub -R and specify the resource requirement string as usual. You can specify multiple -R order, same, rusage, and select sections.

tip:  

Within esub, you can get resource requirements using the LSB_SUB_RES_REQ variable, which merges multiple -R from the bsub command. If you want to modify the LSB_SUB_RES_REQ variable, you cannot use multiple -R format. Instead, use the && operator to merge them manually.

Merged RES_REQ rusage values from the job and application levels must be in the range of RESRSV_LIMIT (set in lsb.queues), or the job is rejected.

Examples

bsub -R "swp > 15 && hpux order[ut]" myjob 

or

bsub -R "select[swp > 15]" -R "select[hpux] order[ut]" myjob

This runs myjob on an HP-UX host that is lightly loaded (CPU utilization) and has at least 15 MB of swap memory available.

bsub -R "select[swp > 15]" -R "select[hpux] order[r15m]"
-R "order[r15m]" -R rusage[mem=100]" -R "order[ut]" -R "same[type] -R "rusage[tmp=50:duration=60]" -R "same[model]" myjob

LSF merges the multiple -R options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts.

View job-level resource requirements

1. Use bjobs -l to view resource requirements defined for the job:

bsub -R "type==any" -q normal myjob 
Job <2533> is submitted to queue <normal>. 
bjobs -l 2533 
Job <2533>, User <user1>, Project <default>, Status <DONE>, Queue 
<normal>, 
     Command <myjob> 
Fri May 10 17:21:26: Submitted from host <hostA>, CWD <$HOME>, 
Requested Resources <type==any>; 
Fri May 10 17:21:31: Started on <hostB>, Execution Home 
</home/user1>,Execution CWD </home/user1>; 
Fri May 10 17:21:47: Done successfully. The CPU time used is 0.3 
seconds. 
... 

2. After a job is finished, use bhist -l to view resource requirements defined for the job:

bhist -l 2533 
Job <2533>, User <user1>, Project <default>, Command <myjob> 
Fri May 10 17:21:26: Submitted from host <hostA>, to Queue 
<normal>, CWD 
     <$HOME>, Requested Resources <type==any>; 
Fri May 10 17:21:31: Dispatched to <hostB>; 
Fri May 10 17:21:32: Starting (Pid 1850232); 
Fri May 10 17:21:33: Running with execution home </home/user1>, 
Execution 
     CWD </home/user1>, Execution Pid <1850232>; 
Fri May 10 17:21:45: Done successfully. The CPU time used is 0.3 
seconds; 
... 
 
note:  
 
If you submitted a job with multiple select strings using the bsub -R option, bjobs -l and bhist -l display a single, merged select string.

About Resource Requirement Strings

Most LSF commands accept a -R res_req argument to specify resource requirements. The exact behavior depends on the command. For example, specifying a resource requirement for the lsload command displays the load levels for all hosts that have the requested resources.

Specifying resource requirements for the lsrun command causes LSF to select the best host out of the set of hosts that have the requested resources.

A resource requirement string describes the resources a job needs. LSF uses resource requirements to select hosts for remote execution and job execution.

Resource requirement strings can be simple (applying to the entire job) or compound (applying to the specified number of slots).

Resource requirement string sections

• A selection section (select). The selection section specifies the criteria for selecting hosts from the system.
• An ordering section (order). The ordering section indicates how the hosts that meet the selection criteria should be sorted.
• A resource usage section (rusage). The resource usage section specifies the expected resource consumption of the task.
• A job spanning section (span). The job spanning section indicates if a parallel batch job should span across multiple hosts.
• A same resource section (same). The same section indicates that all processes of a parallel job must run on the same type of host.
• A compute unit resource section (cu). The cu section specifies how a job should be placed with respect to the underlying network architecture.

Which sections apply

Depending on the command, one or more of these sections may apply. For example:

• bsub uses all sections
• lshosts only selects hosts, but does not order them
• lsload selects and orders hosts
• lsplace uses the information in select, order, and rusage sections to select an appropriate host for a task
• lsloadadj uses the rusage section to determine how the load information should be adjusted on a host

Simple Syntax

select[selection_string] order[order_string] rusage[usage_string 
[, usage_string][|| usage_string] ...] span[span_string] 
same[same_string] cu[cu_string] 

With the bsub and bmod commands, and only with these commands, you can specify multiple -R order, same, rusage, and select sections. The bmod command does not support the use of the || operator.

The section names are select, order, rusage, span, same, and cu. Sections that do not apply for a command are ignored. Each section has a different syntax.

The square brackets must be typed as shown for each section. A blank space must separate each resource requirement section.

You can omit the select keyword and the square brackets, but the selection string must be the first string in the resource requirement string. If you do not give a section name, the first resource requirement string is treated as a selection string (select[selection_string]).

Each section has a different syntax.

By default, memory (mem) and swap (swp) limits in select[] and rusage[] sections are specified in MB. Use LSF_UNIT_FOR_LIMITS in lsf.conf to specify a larger unit for the these limits (MB, GB, TB, PB, or EB).

Compound Syntax

num1*{simple_string1} + num2*{simple_string2} + ... 

where numx is the number of slots affected and simple_stringx is a simple resource requirement string with the syntax:

select[selection_string] order[order_string] rusage[usage_string [, usage_string]...] 
span[span_string] 

Resource requirements applying to the first execution host (if used) should appear in the first compound term num1*{simple_string1}.

Place specific (harder to fill) requirements before general (easier to fill) requirements since compound resource requirement terms are considered in the order they appear. Resource allocation for parallel jobs using compound resources is done for each compound resource term independently instead of considering all possible combinations.

note:  

A host rejected for not satisfying one resource requirement term will not be reconsidered for subsequent resource requirement terms.

For jobs without the number of total slots specified using bsub -n, the final numx can be omitted. The final resource requirement is then applied to the zero or more slots not yet accounted for using the default slot setting of the parameter PROCLIMIT as follows:

• (final res_req number of slots) = MAX(0,(default number of job slots from PROCLIMIT)-(num1+num2+...))

For jobs with the total number of slots specified using bsub -n num_slots, the total number of slots must match the number of slots in the resource requirement as follows, and the final numx can be omitted:

• num_slots=(num1+num2+num3+...)

For jobs with compound resource requirements and first execution host candidates specified using bsub -m, the first allocated host must satisfy the simple resource requirement string appearing first in the compound resource requirement. Thus the first execution host must satisfy the requirements in simple_string1 for the following compound resource requirement:

• num1*{simple_string1} + num2*{simple_string2} + num3*{simple_string3}

Compound resource requirements do not support use of the || operator within the component rusage simple resource requirements, or use of the cu section.

How simple multi-level resource requirements are resolved

Simple resource requirements can be specified at the job, application, and queue levels. When none of the resource requirements are compound, requirements defined at different levels are resolved in the following ways:

• In a select string, a host must satisfy all queue-level, application-level, and job-level requirements for the job to be dispatched.
• In a same string, all queue-level, application-level, and job-level requirements are combined before the job is dispatched.
• order, span, and cu sections defined at the job level overwrite those defined at the application level or queue level. order, span, and cu sections defined at the application level overwrite those defined at the queue level. The default order string is r15s:pg.
• For usage strings, the rusage section defined for the job overrides the rusage section defined in the application. The two rusage definitions are merged, with the job-level rusage taking precedence. Similarly, rusage strings defined for the job or application are merged with queue-level strings, with the job and then application definitions taking precedence over the queue if there is any overlap.

section	simple resource requirement multi-level behavior
select	all levels satisfied
same	all levels combined
order span cu	job-level section overwrites application-level section, which overwrites queue-level section (if a given level is present)
rusage	all levels merge if conflicts occur the job-level section overwrites the application-level section, which overwrites the queue-level section.

For internal load indices and duration, jobs are rejected if the merged job-level and apllication-level resource reservation requirements exceed the requirements specified at the queue level.

note:  

If a compound resource requirement is used at one or more levels, (job, application, or queue) the compound rules apply.

How compound multi-level resource requirements are resolved

Compound resource requirements can be specified at the job, application, and queue levels. When one or more of the resource requirements is compound, requirements at different levels are resolved depending on where the compound resource requirement appears.

For internal load indices and duration, jobs are rejected if they specify resource reservation requirements that exceed the requirements specified at the application level or queue level.

note:  

If a compound resource requirement is used at one or more levels, (job, application, or queue) the compound rules apply.

Compound queue level

When a compound resource requirement is set for a queue it will be ignored unless it is the only resource requirement specified (no resource requirements are set at the job level or application level).

Compound application level

When a compound resource requirement is set at the application level, it will be ignored if any job-level resource requirements (simple or compound) are defined.

In the event no job-level resource requirements are set, the compound application-level requirements interact with queue-level resource requirement strings in the following ways:

• If no queue-level resource requirement is defined or a compound queue-level resource requirement is defined, the compound application-level requirement is used.
• If a simple queue-level requirement is defined, the application-level and queue-level requirements combine as follows:

• section	compound application and simple queue behavior
  select	both levels satisfied; queue requirement applies to all compound terms
  same	queue level ignored
  order span	application-level section overwrites queue-level section (if a given level is present); queue requirement (if used) applies to all compound terms
  rusage	both levels merge queue requirement if a job-based resource is applied to the first compound term, otherwise applies to all compound terms if conflicts occur the application-level section overwrites the queue-level section. For example: if the application-level requirement is num1*{rusage[R1]} + num2*{rusage[R2]} and the queue-level requirement is rusage[RQ] where RQ is a job-based resource, the merged requirement is num1*{rusage[merge(R1,RQ)]} + num2*{rusage[R2]}

Compound job level

When a compound resource requirement is set at the job level, any simple or compound application-level resource requirements are ignored, and any compound queue-level resource requirements are ignored.

In the event a simple queue-level requirement appears along with a compound job-level requirement, the requirements interact as follows:

section	compound job and simple queue behavior
select	both levels satisfied; queue requirement applies to all compound terms
same	queue level ignored
order span	job-level section overwrites queue-level section (if a given level is present); queue requirement (if used) applies to all compound terms
rusage	both levels merge queue requirement if a job-based resource is applied to the first compound term, otherwise applies to all compound terms if conflicts occur the job-level section overwrites the queue-level section. For example: if the job-level requirement is num1*{rusage[R1]} + num2*{rusage[R2]} and the queue-level requirement is rusage[RQ] where RQ is a job resource, the merged requirement is num1*{rusage[merge(R1,RQ)]} + num2*{rusage[R2]}

Example 1

A compound job requirement and simple queue requirement.

job level: 2*{select[type==X86_64] rusage[licA=1] span[hosts=1]} + 8*{select[type==any]}

application level: not defined

queue level: rusage[perslot=1]

The final job scheduling resource requirement merges the simple queue-level rusage section into each term of the compound job-level requirement, resulting in: 2*{select[type==X86_64] rusage[licA=1:perslot=1] span[hosts=1]} + 8*{select[type==any] rusage[perslot=1]}

Example 2

A compound job requirement and compound queue requirement.

job level: 2*{select[type==X86_64 && tmp>10000] rusage[mem=1000] span[hosts=1]} + 8*{select[type==X86_64]}

application level: not defined

queue level: 2*{select[type==X86_64] rusage[mem=1000] span[hosts=1]} +8*{select[type==X86_64]}

The final job scheduling resource requirement ignores the compound queue-level requirement, resulting in: 2*{select[type==X86_64 && tmp>10000] rusage[mem=1000] span[hosts=1]} + 8*{select[type==X86_64]}

Example 3

A compound job requirement and simple queue requirement where the queue requirement is a job-based resource.

job level: 2*{select[type==X86_64]} + 2*{select[mem>1000]}

application level: not defined

queue level: rusage[licA=1] where licA=1 is job-based.

The queue-level requirement is added to the first term of the compound job-level requirement, resulting in: 2*{select[type==X86_64] rusage[licA=1]} + 2*{select[mem>1000]}

Example 4

Compound multi-phase job requirements and simple multi-phase queue requirements.

job level: 2*{rusage[mem=(400 350):duration=(10 15):decay=(0 1)]} + 2*{rusage[mem=300:duration=10:decay=1]}

application level: not defined

queue level: rusage[mem=(500 300):duration=(20 10):decay=(0 1)]

The queue-level requirement is overridden by the first term of the compound job-level requirement, resulting in: 2*{rusage[mem=(400 350):duration=(10 15):decay=(0 1)]} + 2*{rusage[mem=300:duration=10:decay=1]}

Selection String

The selection string specifies the characteristics a host must have to match the resource requirement. It is a logical expression built from a set of resource names. The selection string is evaluated for each host; if the result is non-zero, then that host is selected. When used in conjunction with a cu string, hosts not belonging to compute unit are not considered.

Syntax

The selection string can combine resource names with logical and arithmetic operators. Non-zero arithmetic values are treated as logical TRUE, and zero (0) as logical FALSE. Boolean resources (for example, server to denote LSF server hosts) have a value of one (1) if they are defined for a host, and zero (0) if they are not defined for the host.

The resource names swap, idle, login, and cpu are accepted as aliases for swp, it, ls, and r1m respectively.

The ut index measures CPU utilization, which is the percentage of time spent running system and user code. A host with no processes running has a ut value of 0 percent; a host on which the CPU is completely loaded has a ut of 100 percent. You must specify ut as a floating-point number between 0.0 and 1.0.

For the string resources type and model, the special value any selects any value and local selects the same value as that of the local host. For example, type==local selects hosts of the same type as the host submitting the job. If a job can run on any type of host, include type==any in the resource requirements.

If no type is specified, the default depends on the command. For bsub, lsplace, lsrun, and lsgrun the default is type==local unless a string or Boolean resource is specified, in which case it is type==any. For lshosts, lsload, lsmon and lslogin the default is type==any.

tip:  

When PARALLEL_SCHED_BY_SLOT=Y in lsb.params, the resource requirement string keyword ncpus refers to the number of slots instead of the number of processors, however lshosts output will continue to show ncpus as defined by EGO_DEFINE_NCPUS in lsf.conf.

Specifying multiple -R options

bsub accepts multiple -R options for the select section in simple resource requirements.

restriction:  

Compound resource requirements do not support multiple -R options.

You can specify multiple resource requirement strings instead of using the && operator. For example:

bsub -R "select[swp > 15]" -R "select[hpux]"

LSF merges the multiple -R options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts.

When LSF_STRICT_RESREQ=Y is configured in lsf.conf, you cannot specify more than one select section in the same -R option. Use the logical and (&&) operator to specify multiple selection strings in the same select section. For example, the following command submits a job called myjob to run on a host that has more than 15 MB of swap space available, and maximum RAM larger than 100MB. The job is expected to reserve 100MB memory on the host:

% bsub -R "select [swp > 15 &&  maxmem > 100]  rusage[mem = 100] " myjob 

The number of -R option sections is unlimited.

Selecting shared string resources

You must use single quote characters (') around string-type shared resources. For example, use lsload -s to see the shared resources defined for the cluster:

lsload -s 
RESOURCE                                VALUE       LOCATION 
os_version                                4.2       pc36 
os_version                                4.0       pc34 
os_version                                4.1       devlinux4 
cpu_type                                   ia       pc36 
cpu_type                                   ia       pc34 
cpu_type                              unknown       devlinux4 

Use a select string in lsload -R to specify the shared resources you want to view, enclosing the shared resource values in single quotes. For example:

lsload -R "select[os_version=='4.2' || cpu_type=='unknown']"  
HOST_NAME       status  r15s   r1m  r15m   ut    pg  ls    it   tmp   swp   mem 
pc36                ok   0.0   0.2   0.1   1%   3.4   3     0  895M  517M  123M 
devlinux4           ok   0.0   0.1   0.0   0%   2.8   4     0 6348M  504M  205M 

note:  

When reserving resources based on host status (bsub -R "status==ok"), the host status must be the one displayed by running bhosts not lsload.

Operators

These operators can be used in selection strings. The operators are listed in order of decreasing precedence.

Syntax	Meaning
(a)	When LSF_STRICT_RESREQ=Y is configured in lsf.conf, an expression between parentheses has higher priority than other operators.
-a !a	Negative of a Logical not: 1 if a==0, 0 otherwise
a * b a / b	Multiply a and b Divide a by b
a + b a - b	Add a and b Subtract b from a
a > b a < b a >= b a <= b	1 if a is greater than b, 0 otherwise 1 if a is less than b, 0 otherwise 1 if a is greater than or equal to b, 0 otherwise 1 if a is less than or equal to b, 0 otherwise
a == b a != b	1 if a is equal to b, 0 otherwise 1 if a is not equal to b, 0 otherwise
a && b	Logical AND: 1 if both a and b are non-zero, 0 otherwise
a || b	Logical OR: 1 if either a or b is non-zero, 0 otherwise

Examples

select[(swp > 50 && type == MIPS) || (swp > 35 && type == ALPHA)] 
select[((2*r15s + 3*r1m + r15m) / 6 < 1.0) && !fs && (cpuf > 4.0)] 

Specifying shared resources with the keyword "defined"

A shared resource may be used in the resource requirement string of any LSF command. For example, when submitting an LSF job that requires a certain amount of shared scratch space, you might submit the job as follows:

bsub -R "avail_scratch > 200 && swap > 50" myjob 

The above assumes that all hosts in the cluster have access to the shared scratch space. The job is only scheduled if the value of the "avail_scratch" resource is more than 200 MB and goes to a host with at least 50 MB of available swap space.

It is possible for a system to be configured so that only some hosts within the LSF cluster have access to the scratch space. To exclude hosts that cannot access a shared resource, the defined(resource_name) function must be specified in the resource requirement string.

For example:

bsub -R "defined(avail_scratch) && avail_scratch > 100 && swap > 100" 
myjob 

would exclude any hosts that cannot access the scratch resource. The LSF administrator configures which hosts do and do not have access to a particular shared resource.

Supported resource names in the defined function

Only the following resource names are accepted as the argument in the defined(resource_name) function:

• The following builtin resource names:

LSF_Base lsf_base LSF_Manager lsf_manager LSF_JobScheduler  
lsf_js LSF_Make LSF_parallel LSF_Analyzer lsf_analyzer 

• Resource names configured inlsf.shared, except dynamic NUMERIC resource names with INTERVAL fields defined.

The following resource names are not accepted in the defined(resource_name) function:

• The following builtin resource names:

r15s r1m r15m ut pg io ls it tmp swp mem ncpus ndisks maxmem 
maxswp maxtmp cpuf type model status rexpri server and hname 

• Dynamic NUMERIC resource names configured inlsf.shared with INTERVAL fields defined. In the default configuration, these are mode, cntrl, it_t.)
• Other non-builtin resource names not configured lsf.shared.

Specifying exclusive resources

An exclusive resource may be used in the resource requirement string of any placement or scheduling command, such as bsub, lsplace, lsrun, or lsgrun. An exclusive resource is a special resource that is assignable to a host. This host will not receive a job unless that job explicitly requests the host. For example, use the following command to submit a job requiring the exclusive resource bigmem:

bsub -R "bigmem" myjob 

Jobs will not be submitted to the host with the bigmem resource unless the command uses the -R option to explicitly specify "bigmem".

To configure an exclusive resource, first define a static Boolean resource in lsf.shared. For example:

Begin Resource 
... 
bigmem Boolean () () 
End Resource 

Assign the resource to a host in the Host section of lsf.cluster.cluster_name. Prefix the resource name with an exclamation mark (!) to indicate that the resource is exclusive to the host. For example:

Begin Host 
HOSTNAME   model    type   server r1m pg tmp RESOURCES        RUNWINDOW 
... 
hostE      !        !      1      3.5 () ()  (linux !bigmem)  () 
... 
End Host 

Strict syntax for resource requirement selection strings

When LSF_STRICT_RESREQ=Y is configured in lsf.conf, resource requirement strings in select sections must conform to a more strict syntax. The strict resource requirement syntax only applies to the select section. It does not apply to the other resource requirement sections (order, rusage, same, span, or cu). When LSF_STRICT_RESREQ=Y in lsf.conf, LSF rejects resource requirement strings where an rusage section contains a non-consumable resource.

Strict syntax in EBNF form:

<expression> ::= <relation1> { <logical or> <relation1>} 
<relation1> ::=<relation2> { <logical and > <relation2>} 
<relation2> ::= <simple expression> [ <relation op> <simple 
expression> ] 
<simple expression> ::= <term> { <adding op> <term> } 
<term> ::= <factor> { <multiple op> <factor> } 
<factor> ::= [<unary op>] <primary> 
<primary> ::= <numeric> | <string> | ( <expression> )| <name or 
function call> 
<logical or> ::= || 
<logical and>::=&&  
<relation op> ::= <= | >= | == | != | < | > | =  
<adding op> ::= + | - 
<unary op> ::= - | ! 
<multiple op> ::= * | / 
<name or function call> ::= <name> [( <argument list> )] 
<argument list> ::= <empty>       | <argument> {, <argument> } 
<argument> ::= <expression>  
<name> ::= [a-zA-Z_][a-zA-Z_0-9]* 
<numeric> ::= <int> [. [0-9]*]  
<int> ::= [1-9][0-9]* | 0 
<string> ::= <single quote> {<string chars>} <single quote> | <double 
quote> {<string chars>} <double quote>  
<string chars> ::= <printable ascii characters except single/double 
quote>  
<single quote> ::= ' 
<double quote> ::= " 
<empty>= 

Strict select string syntax usage notes

The strict syntax is case sensitive.

Operators '=' and '==' are equivalent.

Boolean variables, such as fs, hpux, cs, can only be computed with the following operators:

&& || ! 

String variables, such as type, can only be computed with the following operators:

= == != < > <= >= 

For function calls, blanks between the parentheses "( )" and the resource name are not valid. For example, the following is not correct:

defined(   mg   ) 

Multiple logical NOT operators (!) are not valid. For example, the following is not correct:

!!mg 

The following resource requirement is valid:

!(!mg) 

At least one blank space must separate each section. For example, the following are correct:

type==any rusage[mem=1024] 
select[type==any] rusage[mem=1024] 
select[type==any]rusage[mem=1024] 

but the following is not correct:

type==anyrusage[mem=1024] 

Only a single select section is supported by the stricter syntax. The following is not supported in the same resource requirement string:

select[mem>0] select[maxmem>0] 

Escape characters (like '\n') are not supported in string literals.

A colon (:) is not allowed inside the select string. For example, select[mg:bigmem] is not correct.

inf and nan can be used as resource names or part of a resource name.

Single or double quotes are only supported around the whole resource requirement string, not within the square brackets containing the selection string. For example, in lsb.queues, RES_REQ='swp>100' and RES_REQ="swp>100" are correct. Neither RES_REQ=select['swp>100'] nor RES_REQ=select["swp>100"] are supported.

The following are correct bsub command-level resource requirements:

• bsub -R "'swp>100'"
• bsub -R '"swp>100"'

The following are not correct:

• bsub -R "select['swp>100']"
• bsub -R 'select["swp>100"]'

Some incorrect resource requirements are no longer silently ignored. For example, when LSF_STRICT_RESREQ=Y is configured in lsf.conf, the following are rejected by the resource requirement parser:

• microcs73 is rejected:

linux rusage[mem=16000] microcs73 

• select[AMD64] is rejected:

mem < 16384 && select[AMD64] 

• linux is rejected:

rusage[mem=2000] linux 

• Using a colon (:) to separate select conditions, such as linux:qscw .
• The restricted syntax of resource requirement select strings described in the lsfintro(1) man page is not supported.

Explicit and implicit select sections

An explicit select section starts from the section keyword and ends at the begin of next section, for example: the select section is select[selection_string]. An implicit select section starts from the first letter of the resource requirement string and ends at the end of the string if there are no other resource requirement sections. If the resource requirement has other sections, the implicit select section ends before the first letter of the first section following the selection string.

All explicit sections must begin with a section keywords (select, order, span rusage, or same). The resource requirement content is contained by square brackets ([) and (]).

An implicit select section must be the first resource requirement string in the whole resource requirement specification. Explicit select sections can appear after other sections. A resource requirement string can have only one select section (either an explicit select section or an implicit select section). A section with an incorrect keyword name is not a valid section.

An implicit select section must have the same format as the content of an explicit select section. For example, the following commands are correct:

• bsub -R "select[swp>15] rusage[mem=100]" myjob
• bsub -R "swp > 15 rusage[mem=100]" myjob
• bsub -R "rusage[mem=100] select[swp >15]" myjob

Examples

The following examples illustrate some correct resource requirement select string syntax.

• bsub -R "(r15s * 2 + r15m) < 3.0 && !(type == IBMAIX4) || fs" myjob
• If swap space is equal to 0, the following means TRUE; if swap space is not equal to 0, it means FALSE:

bsub -R "!swp" myjob  

• Select hosts of the same type as the host submitting the job:

bsub -R "type == local" myjob 

• Select hosts that are not the same type as the host submitting the job:

bsub -R "type != local" myjob 

• bsub -R "r15s < 1.0 || model ==local && swp <= 10" myjob
• Since && has a higher priority than ||, this example means:

r15s < 1.0 || (model == local  && swp <=10) 

• This example has different meaning from the previous example:

bsub -R "(r15s < 1.0 || model == local) && swp <= 10" myjob 

• This example means:

(r15s < 1.0 || model == local) && swp <= 10 

Checking resource requirement syntax

Use the BSUB_CHK_RESREQ environment variable to check the compatibility of your existing resource requirement select strings against the stricter syntax enabled by LSF_STRICT_RESREQ=Y in lsf.conf.

Set the BSUB_CHK_RESREQ environment variable to any value enable bsub to check the syntax of the resource requirement selection string without actually submitting the job for scheduling and dispatch. LSF_STRICT_RESREQ does not need to be set to check the resource requirement selection string syntax.

bsub only checks the select section of the resource requirement. Other sections in the resource requirement string are not checked.

If resource requirement checking detects syntax errors in the selection string, bsub returns and error message. For example:

bsub -R "select[type==local] select[hname=abc]" sleep 10 
Error near "select": duplicate section. Job not submitted. 
echo $? 
255 

If no errors are found, bsub returns a successful message and exit code zero (0). For example:

env | grep BSUB_CHK_RESREQ 
BSUB_CHK_RESREQ=1 
bsub -R "select[type==local]" sleep 10 
Resource requirement string is valid. 
echo $? 
0 

If BSUB_CHK_RESREQ is set, but you do not specify -R, LSF treats it as empty resource requirement. For example:

bsub sleep 120 
Resource requirement string is valid. 
echo $? 
0 

Resizable jobs

Resize allocation requests are scheduled using hosts as determined by the select expression of the merged resource requirement. For example, to run an autoresizable job on 1-100 slots, but only on hosts of type X86_64, the following job submission specifies this resource request:

bsub -ar -n "1,100" -R "select[type == X86_64]" myjob  

Every time the job grows in slots, slots are requested on hosts of the specified type.

note:  

Resizable jobs cannot have compound resource requirements.

Order String

The order string allows the selected hosts to be sorted according to the values of resources. The values of r15s, r1m, and r15m used for sorting are the normalized load indices returned by lsload -N.

The order string is used for host sorting and selection. The ordering begins with the rightmost index in the order string and proceeds from right to left. The hosts are sorted into order based on each load index, and if more hosts are available than were requested, the LIM drops the least desirable hosts according to that index. The remaining hosts are then sorted by the next index.

After the hosts are sorted by the leftmost index in the order string, the final phase of sorting orders the hosts according to their status, with hosts that are currently not available for load sharing (that is, not in the ok state) listed at the end.

Because the hosts are sorted again for each load index, only the host status and the leftmost index in the order string actually affect the order in which hosts are listed. The other indices are only used to drop undesirable hosts from the list.

When sorting is done on each index, the direction in which the hosts are sorted (increasing vs. decreasing values) is determined by the default order returned by lsinfo for that index. This direction is chosen such that after sorting, by default, the hosts are ordered from best to worst on that index.

When used with a cu string, the preferred compute unit order takes precedence. Within each compute unit hosts are ordered according to the order string requirements.

Syntax

[-]resource_name [:[-]resource_name]... 

You can specify any built-in or external load index or static resource.

When an index name is preceded by a minus sign `-', the sorting order is reversed so that hosts are ordered from worst to best on that index.

Specifying multiple -R options

bsub accepts multiple -R options for the order section.

restriction:  

Compound resource requirements do not support multiple -R options.

You can specify multiple resource requirement strings instead of using the && operator. For example:

bsub -R "order[r15m]" -R "order[ut]"

LSF merges the multiple -R options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts. The number of -R option sections is unlimited.

Default

The default sorting order is r15s:pg (except for lslogin(1): ls:r1m).

Example

swp:r1m:tmp:r15s 

Resizable jobs

The order in which hosts are considered for resize allocation requests is determined by the order expression of the job. For example, to run an autoresizable job on 1-100 slots, preferring hosts with larger memory, the following job submission specifies this resource request:

bsub -ar -n "1,100" -R "order[mem]" myjob  

When slots on multiple hosts become available simultaneously, hosts with larger available memory get preference when the job adds slots.

note:  

Resizable jobs cannot have compound resource requirements.

Usage String

This string defines the expected resource usage of the job. It is used to specify resource reservations for jobs, or for mapping jobs on to hosts and adjusting the load when running interactive jobs.

By default, no resources are reserved.

When LSF_STRICT_RESREQ=Y in lsf.conf, LSF rejects resource requirement strings where an rusage section contains a non-consumable resource.

Multi-phase resources

Multiple phases within the rusage string allow different time periods to have different memory requirements (load index mem). The duration of all except the last phase must be specified, while decay rates are all optional and are assumed to be 0 if omitted. If the optional final duration is left blank, the final resource requirement applies until the job is finished.

Multi-phase resource reservations cannot include increasing resources, but can specify constant or decreasing resource reservations over multiple periods of time.

Resource reservation limits

Resource requirement reservation limits can be set using the parameter RESRSV_LIMIT in lsb.queues. Queue-level RES_REQ rusage values (set in lsb.queues) must be in the range set by RESRSV_LIMIT, or the queue-level RES_REQ is ignored. Merged RES_REQ rusage values from the job and application levels must be in the range of RESRSV_LIMIT, or the job is rejected.

When both the RES_REQ and RESRSV_LIMIT are set in lsb.queues for a consumable resource, the queue-level RES_REQ no longer acts as a hard limit for the merged RES_REQ rusage values from the job and application levels. In this case only the limits set by RESRSV_LIMIT must be satisfied, and the queue-level RES_REQ acts as a default value.

Batch jobs

The resource usage (rusage) section can be specified at the job level, with the queue configuration parameter RES_REQ, or with the application profile parameter RES_REQ.

Basic syntax

rusage[usage_string [, usage_string][|| usage_string] ...] 

where usage_string is:

load_index=value [:load_index=value]... [:duration=minutes[m] | 
:duration=hoursh | :duration=secondss [:decay=0 | :decay=1]] 

Multi-phase memory syntax

rusage[multi_usage_string [, usage_string]...] 

where multi_usage_string is:

mem=(v1 [v2 ... vn]):[duration=(t1 [t2 ... tm])][:decay=(d1 [d2... dk])] 

for m = n|n-1. For a single phase (n=1), duration is not required.

if k > m, dm+1 to dk will be ignored; if k < m, dk+1 =.. = dm = 0.

usage_string is the same as the basic syntax, for any load_index other than mem.

Multi-phase syntax can be used with a single phase memory resource requirement as well as for multiple phases. For multi-phase slot-based resource reservation, use with RESOURCE_RESERVE_PER_SLOT=Y in lsb.params.

Multi-phase resource reservations cannot increase over time. A job submission with increasing resource reservations from one phase to the next will be rejected. For example:

bsub -R"rusage[mem=(200 300):duration=(2 3)]" myjob 

specifies an increasing memory reservation from 200 MB to 300 MB. This job will be rejected.

tip:  

When a multi-phase mem resource requirement is being used, duration can be specified separately for single-phase resources.

Load index

Internal and external load indices are considered in the resource usage string. The resource value represents the initial reserved amount of the resource.

Duration

The duration is the time period within which the specified resources should be reserved. Specify a duration equal to or greater than the ELIM updating interval.

• If the value is followed by the letter s, m, or h, the specified time is measured in seconds, minutes, or hours respectively.
• By default, duration is specified in minutes.
• For example, the following specify a duration of 1 hour for multi-phase syntax:
• duration=(60)
• duration=(1h)
• duration=(3600s)
• For example, the following specify a duration of 1 hour for single-phase syntax:
• duration=60
• duration=1h
• duration=3600s

tip:  

Duration is not supported for static shared resources. If the shared resource is defined in an lsb.resources Limit section, then duration is not applied.

Decay

The decay value indicates how the reserved amount should decrease over the duration.

• A value of 1 indicates that system should linearly decrease the amount reserved over the duration.
• A value of 0 causes the total amount to be reserved for the entire duration.

Values other than 0 or 1 are unsupported, and are taken as the default value of 0. If duration is not specified, decay value is ignored.

tip:  

Decay is not supported for static shared resources. If the shared resource is defined in an lsb.resources Limit section, then decay is not applied.

Default

If a resource or its value is not specified, the default is not to reserve that resource. If duration is not specified, the default is to reserve the total amount for the lifetime of the job. (The default decay value is 0.)

Example

rusage[mem=50:duration=100:decay=1] 

This example indicates that 50 MB memory should be reserved for the job. As the job runs, the amount reserved will decrease at approximately 0.5 MB per minute until the 100 minutes is up.

How simple queue-level and job-level rusage sections are resolved

Job-level rusage overrides the queue level specification:

• For internal load indices (r15s, r1m, r15m, ut, pg, io, ls, it, tmp, swp, and mem), the job-level value cannot be larger than the queue-level value (unless the limit parameter RESRSV_LIMIT is being used as a maximum instead of the queue-level value).
• For external load indices (e.g., licenses), the job-level rusage can be larger than the queue-level requirements.
• For duration, the job-level value of internal and external load indices cannot be larger than the queue-level value.
• For multi-phase simple rusage sections:
• For internal load indices (r15s, r1m, r15m, ut, pg, io, ls, it, tmp, swp, and mem), the first phase of the job-level value cannot be larger than the first phase of the queue-level value (unless the limit parameter RESRSV_LIMIT is being used as a maximum instead of the queue-level value).
• For duration and decay, if either job-level or queue-level is multi-phase, the job-level value will take precedence.

How simple queue-level and job-level rusage sections are merged

When both job-level and queue-level rusage sections are defined, the rusage section defined for the job overrides the rusage section defined in the queue. The two rusage definitions are merged, with the job-level rusage taking precedence. For example:

Example 1

Given a RES_REQ definition in a queue:

RES_REQ = rusage[mem=200:lic=1] ... 

and job submission:

bsub -R "rusage[mem=100]" ... 

The resulting requirement for the job is

rusage[mem=100:lic=1] 

where mem=100 specified by the job overrides mem=200 specified by the queue. However, lic=1 from queue is kept, since job does not specify it.

Example 2

For the following queue-level RES_REQ (decay and duration defined):

RES_REQ = rusage[mem=200:duration=20:decay=1] ... 

and job submission (no decay or duration):

bsub -R "rusage[mem=100]" ... 

The resulting requirement for the job is:

rusage[mem=100:duration=20:decay=1] 

Queue-level duration and decay are merged with the job-level specification, and mem=100 for the job overrides mem=200 specified by the queue. However, duration=20 and decay=1 from queue are kept, since job does not specify them.

rusage in application profiles

See Chapter 23, "Working with Application Profiles" for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.

How simple queue-level rusage sections are merged with compound rusage sections

When simple queue-level and compound application-level or job-level rusage sections are defined, the two rusage definitions are merged. If a job-level resource requirement (simple or compound) is defined, the application level is ignored and the job-level and queue-level sections merge. If no job-level resource requirement is defined, the application-level and queue-level merge.

When a compound resource requirement merges with a simple resource requirement from the queue-level, the behavior depends on whether the queue-level requirements are job-based or not.

Example 1

Job-based simple queue-level requirements apply to the first term of the merged compound requirements. For example:

Given a RES_REQ definition for a queue which refers to a job-based resource:

RES_REQ = rusage[lic=1] ... 

and job submission resource requirement:

bsub -R "2*{rusage[mem=100] ...} + 
4*{[mem=200:duration=20:decay=1] ...}" 

The resulting requirement for the job is

bsub -R "2*{rusage[mem=100:lic=1] ...} + 
4*{[mem=200:duration=20:decay=1] ...}" 

The job-based resource lic=1 from queue is added to the first term only, since it is job-based and wasn't included the job-level requirement.

Example 2

Host-based or slot-based simple queue-level requirements apply to all terms of the merged compound requirements. For example:

For the following queue-level RES_REQ which does not include job-based resources:

RES_REQ = rusage[mem=200:duration=20:decay=1] ... 

and job submission:

bsub -R "2*{rusage[mem=100] ...} + 4*{rusage[lic=1] ...}" 

The resulting requirement for the job is:

2*{rusage[mem=100:duration=20:decay=1] ...} + 
4*{rusage[lic=1:mem=200:duration=20:decay=1] ...} 

Where duration=20 and decay=1 from queue are kept, since job does not specify them in any term. In the first term mem=100 from the job is kept; in the second term mem=200 from the queue is used since it wasn't specified by the job resource requirement.

Specifying multiple -R options

bsub accepts multiple -R options for the rusage section.

restriction:  

Compound resource requirements do not support multiple -R options. Multi-phase rusage strings do not support multiple -R options.

You can specify multiple resource requirement strings instead of using the && operator. For example:

bsub -R "rusage[mem=100]" -R "rusage[tmp=50:duration=60]"

LSF merges the multiple -R options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts.

Comma-separated multiple resource requirements within one rusage string is supported. For example:

bsub -R "rusage[mem=20, license=1:duration=2]"
-R "rusage[app_lic_v201=1||app_lic_v15=1]" myjob

A given load index cannot appear more than once in the resource usage string.

Examples

• The following job requests 20 MB memory for the duration of the job, and 1 license to be reserved for 2 minutes:

bsub -R "rusage[mem=20, license=1:duration=2]" myjob 

• A queue with the same resource requirements could specify:

RES_REQ = rusage[mem=20, license=1:duration=2] 

• The following job requests 20 MB of memory and 50 MB of swap space for 1 hour, and 1 license to be reserved for 2 minutes:

bsub -R "rusage[mem=20:swp=50:duration=1h, license=1:duration=2]" myjob 

• The following job requests 50 MB of swap space, linearly decreasing the amount reserved over a duration of 2 hours, and requests 1 license to be reserved for 2 minutes:

bsub -R "rusage[swp=20:duration=2h:decay=1, license=1:duration=2]" myjob 

• The following job requests two resources with same duration but different decay:

bsub -R "rusage[mem=20:duration=30:decay=1, lic=1:duration=30] myjob 

Specifying alternative usage strings

If you use more than one version of an application, you can specify the version you prefer to use together with a legacy version you can use if the preferred version is not available. Use the OR (||) expression to separate the different usage strings that define your alternative resources.

Job-level resource requirement specifications that use the || operator are merged with other rusage requirements defined at the application and queue levels.

note:  

Alternative rusage strings cannot be submitted with compound resource requirements.

Job-level examples

The following examples assume that you are running an application version 1.5 as a resource called app_lic_v15 and the same application version 2.0.1 as a resource called app_lic_v201. The license key for version 2.0.1 is backward compatible with version 1.5, but the license key for version 1.5 will not work with 2.0.1

• If you can only run your job using version 2.0.1 of the application, submit the job without specifying an alternate resource. To submit a job that will only use app_lic_v201:

bsub -R "rusage[app_lic_v201=1]" myjob 

• If you can run your job using either version of the application, try to reserve version 2.0.1 of the application. If it is not available, you can use version 1.5. To submit a job that will try app_lic_v201 before trying app_lic_v15:

bsub -R "rusage[app_lic_v201=1||app_lic_v15=1]" myjob 

• If different versions of an application require different system resources, you can specify other resources in your rusage strings. To submit a job that will use 20 MB of memory for app_lic_v201 or 20 MB of memory and 50 MB of swap space for app_lic_v15:

bsub -R "rusage[mem=20:app_lic_v201=1||mem=20:swp=50:app_lic_v15=1]" myjob 

• You can also specify alternative multi-phase memory requirements. To submit a job that will use 20 MB of memory for app_lic_v201 or 50 MB of swap space and 50 MB of memory for 10 minutes followed by 10 MB of memory for the remainder of the job for app_lic_v15:

bsub -R "rusage[mem=20:app_lic_v201=1 
||mem=(50 10):duration=(10),swp=50:app_lic_v15=1]" myjob 

How LSF merges rusage strings that contain the || operator

The following examples show how LSF merges job-level and queue-level rusage strings that contain the || operator.

Queue level RES_REQ=rusage...	Job level bsub -R "rusage ...	Resulting rusage string
[mem=200:duration=180]	[w1=1 || w2=1 || w3=1]"	[w1=1, mem=200:duration=180 || w2=1, mem=200:duration=180 || w3=1, mem=200:duration=180]
[w1=1 || w2=1 || w3=1]	[mem=200:duration=180]"	[mem=200:duration=180, w1=1 || mem=200:duration=180, w2=1 || mem=200:duration=180, w3=1]

note:  

Alternative rusage strings cannot be submitted with compound resource requirements.

Non-batch environments

Resource reservation is only available for batch jobs. If you run jobs using only LSF Base, such as through lsrun, LIM uses resource usage to determine the placement of jobs. Resource usage requests are used to temporarily increase the load so that a host is not overloaded. When LIM makes a placement advice, external load indices are not considered in the resource usage string. In this case, the syntax of the resource usage string is

res[=value]:res[=value]: ... :res[=value] 

res is one of the resources whose value is returned by the lsload command.

rusage[r1m=0.5:mem=20:swp=40] 

The above example indicates that the task is expected to increase the 1-minute run queue length by 0.5, consume 20 MB of memory and 40 MB of swap space.

If no value is specified, the task is assumed to be intensive in using that resource. In this case no more than one task will be assigned to a host regardless of how many CPUs it has.

The default resource usage for a task is r15s=1.0:r1m=1.0:r15m=1.0. This indicates a CPU-intensive task which consumes few other resources.

Resizable jobs

Unlike the other components of a resource requirement string that only pertain to adding additional slots to a running job, rusage resource requirement strings affect the resource usage when slots are removed from the job as well.

When adding or removing slots from a running job:

• The amount of slot-based resources added to or removed from the job allocation is proportional to the change in the number of slots
• The amount of job-based resources is not affected by a change in the number of slots
• The amount of each host-based resource is proportional to the change in the number of hosts

When using multi-phase resource reservation, the job allocation is based on the phase of the resource reservation.

note:  

Resizable jobs cannot have compound resource requirements.

Duration and decay of rusage

Duration and decay of resource usage and the || operator affect resource allocation.

Duration or decay of a resource in the rusage expression is ignored when scheduling the job for the additional slots.

If a job has the following rusage string: rusage[mem=100:duration=300], the resize request of one additional slot is scheduled on a host only if there are 100 units of memory available on that host. In this case, mem is a slot-based resource (RESOURCE_RESERVE_PER_SLOT=Y in lsb.params).

Once the resize operation is done, if the job has been running less than 300 seconds then additional memory will be reserved only until the job has run for 300 seconds. If the job has been running for more than 300 seconds when the job is resized, no additional memory is reserved. The behavior is similar for decay.

The || operator lets you specify multiple alternative rusage strings, one of which is used when dispatching the job. You cannot use bmod to change rusage to a new one with a || operator after the job has been dispatched

For job resize, when the || operator is used, the resize request uses the rusage expression that was originally used to dispatch the job. If the rusage expression has been modified since the job started, the resize request is scheduled using the new single rusage expression.

Example 1

You want to run an autoresizable job such that every slot occupied by the job reserves 100 MB of swap space In this case, swp is a slot-based resource (RESOURCE_RESERVE_PER_SLOT=Y in lsb.params). The job also needs a separate license for each host on which it runs. Each additional slot allocated to the job should reserve additional swap space, and each new host should reserve an additional license. The following job submission specifies this resource request:

bsub -ar -n "1,100" -R "rusage[swp=100,license=1]" myjob  

where license is a user-defined host-based resource.

Similarly, if you want to release some of the slots from a running job, resources reserved by the job are decreased appropriately. For example, for the following job submission:

bsub -ar -n 100 -R "rusage[swp=50:license=1]" myjob  
Job <123> is submitted to default queue. 

you can run bresize release to release all the slots from the job on one host:

bresize release "hostA" 123 

The swap space used by the job is reduced by the number of slots used on hostA times 50 MB, and one host-based license resource is released from the job.

Example 2

You have a choice between two versions of an application, each version having different memory and swap space requirements on hosts and a different license (app_lic_v15 and app_lic_v201). If you submit an autoresizable job with the || operator, once the job is started using one version of an application, slots added to a job during a resize operation reserve resources depending on which version of the application was originally run. For example, for the following job submission:

bsub -n "1,100" -ar -R "rusage[mem=20:app_lic_v201=1 || mem=20:swp=50:app_lic_v15=1]" 
myjob  

If the job starts with app_lic_v15, each additional slot added in a resize operation reserves 20 MB of memory and 50 MB of swap space.

Span String

A span string specifies the locality of a parallel job. If span is omitted, LSF allocates the required processors for the job from the available set of processors.

Syntax

The span string supports the following syntax:

span[hosts=1]

Indicates that all the processors allocated to this job must be on the same host.

span[ptile=value]

Indicates the number of processors on each host that should be allocated to the job, where value is one of the following:

• Default ptile value, specified by n processors. In the following example, the job requests 4 processors on each available host, regardless of how many processors the host has:

span[ptile=4] 

• Predefined ptile value, specified by '!'. The following example uses the predefined maximum job slot limit lsb.hosts (MXJ per host type/model) as its value:

span[ptile='!'] 
 
tip:  
 
If the host or host type/model does not define MXJ, the default predefined ptile value is 1.
 
restriction:  
 
Under bash 3.0, the exclamation mark (!) is not interpreted correctly by the shell. To use predefined ptile value (ptile='!'), use the +H option to disable '!' style history substitution in bash (sh +H).

• Predefined ptile value with optional multiple ptile values, per host type or host model:
• For host type, you must specify same[type] in the resource requirement. In the following example, the job requests 8 processors on a host of type HP or SGI, and 2 processors on a host of type LINUX, and the predefined maximum job slot limit in lsb.hosts (MXJ) for other host types:

span[ptile='!',HP:8,SGI:8,LINUX:2] same[type] 

• For host model, you must specify same[model] in the resource requirement. In the following example, the job requests 4 processors on hosts of model PC1133, and 2 processors on hosts of model PC233, and the predefined maximum job slot limit in lsb.hosts (MXJ) for other host models:

span[ptile='!',PC1133:4,PC233:2] same[model] 

span[hosts=-1]

Disables span setting in the queue. LSF allocates the required processors for the job from the available set of processors.

See Controlling Processor Allocation Across Hosts for more information about specifying span strings.

Resizable jobs

For resource requirements with span[hosts=1], a resize request is limited to slots on the first-execution host of the job. This behavior eliminates the ambiguities that arise when the span expression is modified from the time that the job was originally dispatched.

For span[ptile=n], the job will be allocated exactly n slots on some number of hosts, and a number between 1 and n slots (inclusive) on one host. This is true even if a range of slots is requested. For example, for the following job submission:

bsub -n "1,20" -R "span[ptile=2]" sleep 10000 

This special span behavior does not only apply to resize requests. It applies to resizable jobs only when the original allocation is made, and in making additional resize allocations.

If every host has only a single slot available, the job is allocated one slot.

Resize requests with partially filled hosts are handled so that LSF does not choose any slots on hosts already occupied by the job. For example, it is common to use the ptile feature with span[ptile=1] to schedule exclusive jobs. Another typical use is span[ptile='!'] to make the job occupy all slots on each host it is allocated.

For a resizable job (auto-resizable or otherwise) with a range of slots requested and span[ptile=n], whenever the job is allocated slots, it will receive either of the following:

• The maximum number of slots requested, comprising n slots on each of a number of hosts, and between 0 and n-1 (inclusive) slots on one host
• n slots on each of a number of hosts, summing to some value less than the maximum

For example, if a job requests between 1 and 14 additional slots, and span[ptile=4] is part of the job resource requirement string, when additional slots are allocated to the job, the job receives either of the following:

• 14 slots, with 2 slots on one host and 4 slots on each of 3 hosts
• 4, 8 or 12 slots, such that 4 slots are allocated per host of the allocation

note:  

Resizable jobs cannot have compound resource requirements.

Example

When running a parallel exclusive job, it is often desirable to specify span[ptile=1] so that the job is allocated at most one slot on each host. For an autoresizable job, new slots are allocated on hosts not already used by the job. The following job submission specifies this resource request:

bsub -x -ar -n "1,100" -R "span[ptile=1]" myjob 

When additional slots are allocated to a running job, the slots will be on new hosts, not already occupied by the job.

Same String

tip:  

You must have the parallel batch job scheduler plugin installed in order to use the same string.

Parallel jobs run on multiple hosts. If your cluster has heterogeneous hosts, some processes from a parallel job may for example, run on Solaris and some on SGI IRIX. However, for performance reasons you may want all processes of a job to run on the same type of host instead of having some processes run on one type of host and others on another type of host.

The same string specifies that all processes of a parallel job must run on hosts with the same resource.

You can specify the same string:

• At the job level in the resource requirement string of:
• bsub
• bmod
• At the queue level in lsb.queues in the RES_REQ parameter.

When queue-level, application-level, and job-level same sections are defined, LSF combines requirements to allocate processors.

Syntax

resource_name[:resource_name]... 

You can specify any static resource.

For example, if you specify resource1:resource2, if hosts always have both resources, the string is interpreted as allocate processors only on hosts that have the same value for resource1 and the same value for resource2.

If hosts do not always have both resources, it is interpreted as allocate processors either on hosts that have the same value for resource1, or on hosts that have the same value for resource2, or on hosts that have the same value for both resource1 and resource2.

Specifying multiple -R options

bsub accepts multiple -R options for the same section.

restriction:  

Compound resource requirements do not support multiple -R options.

You can specify multiple resource requirement strings instead of using the && operator. For example:

bsub -R "same[type]" -R "same[model]"

LSF merges the multiple -R options into one string and dispatches the job if all of the resource requirements can be met. By allowing multiple resource requirement strings and automatically merging them into one string, LSF simplifies the use of multiple layers of wrapper scripts.

Resizable jobs

The same expression ensures that the resize allocation request is dispatched to hosts that have the same resources as the first-execution host. For example, if the first execution host of a job is SOL7 and the resource requirement string contains same[type], additional slots are allocated to the job on hosts of type SOL7.

Taking the same resource as the first-execution host avoids ambiguities that arise when the original job does not have a same expression defined, or has a different same expression when the resize request is scheduled.

For example, a parallel job may be required to have all slots on hosts of the same type or model for performance reasons. For an autoresizable job, any additional slots given to the job will be on hosts of the same type, model, or resource as those slots originally allocated to the job. The following command submits an autoresizable job such that all slots allocated in a resize operation are allocation on hosts with the same model as the original job:

bsub -ar -n "1,100" -R "same[model]" myjob    

Examples

bsub -n 4 -R"select[type==SGI6 || type==SOL7] same[type]" myjob 

Run all parallel processes on the same host type. Allocate 4 processors on the same host type-either SGI IRIX, or Solaris 7, but not both.

bsub -n 6 -R"select[type==any] same[type:model]" myjob 

Run all parallel processes on the same host type and model. Allocate 6 processors on any host type or model as long as all the processors are on the same host type and model.

Same string in application profiles

See Chapter 23, "Working with Application Profiles" for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.

Compute Unit String

A cu string specifies the network architecture-based requirements of parallel jobs. cu sections are accepted by bsub -R, and by bmod -R for non-running jobs.

Compute unit resource requirements are not supported in compound resource requirements.

For a complete description of compute units see Controlling Job Locality using Compute Units in Chapter 34, "Running Parallel Jobs".

Syntax

The cu string supports the following syntax:

cu[type=cu_type]

Indicates the type of compute units the job can run on. Types are defined by COMPUTE_UNIT_TYPES in lsb.params. If type is not specified, the default set by COMPUTE_UNIT_TYPES is assumed.

cu[pref=maxavail | minavail | config]

Indicates the compute unit scheduling preference, grouping hosts by compute unit before applying a first-fit algorithm to the sorted hosts. For resource reservation, the default pref=config is always used.

Compute units are ordered as follows:

• config lists compute units in the order they appear in the ComputeUnit section of lsf.hosts. If pref is not specified, pref=config is assumed.
• maxavail lists compute units with more free slots first. Should compute units have equal numbers of free slots, they appear in the order listed in the ComputeUnit section of lsf.hosts.
• minavail lists compute units with fewer free slots first. Should compute units have equal numbers of free slots, they appear in the order listed in the ComputeUnit section of lsf.hosts.

Free slots include all available slots not occupied by running jobs.

When pref is used with the keyword balance, balance takes precedence.

Hosts accept jobs separated by the time interval set by JOB_ACCEPT_INTERVAL in lsb.params; jobs submitted closer together than this interval will run on different hosts regardless of the pref setting.

cu[maxcus=number]

Indicates the maximum number of compute units a job can run over. Jobs may be placed over fewer compute units if possible.

When used with bsub -n min, max a job is allocated the first combination satisfying both min and maxcus, while without maxcus a job is allocated as close to max as possible.

cu[usablecuslots=number]

Specifies the minimum number of slots a job must use on each compute unit it occupies. number is a non-negative integer value.

When more than one compute unit is used by a job, the final compute unit allocated can provide less than number slots if less are needed.

usablecuslots and balance cannot be used together. 

cu[balance]

Indicates that a job should be split evenly between compute units, with a difference in compute unit slot allocation of at most 1. A balanced allocation spans the fewest compute units possible.

When used with bsub -n min, max the value of max is disregarded.

balance and usablecuslots cannot be used together. 

When balance and pref are both used, balance takes precedence. The keyword pref is only considered if there are multiple balanced allocations spanning the same number of compute units. In this case pref is considered when choosing the allocation.

When balance is used with span[ptile=X] (for X>1) a balanced allocation is one split evenly between compute units, with a difference in compute unit host allocation of at most 1.

cu[excl]

Indicates that jobs must use compute units exclusively. Exclusivity applies to the compute unit granularity specified by type.

Compute unit exclusivity must be enabled by EXCLUSIVE=CU[cu_type] in lsb.queues.

Resizable jobs

Auto-resizable jobs cannot be submitted with compute unit resource requirements. In the event a bswitch call or queue reconfiguration results in an auto-resizable job running in a queue with compute unit resource requirements, the job will no longer be auto-resizable.

restriction:  

Increasing resources allocated to resizable jobs with compute unit resource requirements is not supported.

Examples

bsub -n 11,60 -R "cu[maxcus=2:type=enclosure]" myjob

Spans the fewest possible compute units for a total allocation of at least 11 slots using at most 2 compute units of type enclosure. In contrast, without maxcus:

bsub -n 11,60 myjob

In this case the job is allocated as close to 60 slots as possible, with a minimum of 11 slots.

bsub -n 64 -R "cu[balance:maxcus=4:type=enclosure]" myjob

Spans the fewest possible compute units for a balanced allocation of 64 slots using 4 or less compute units of type enclosure.Possible balanced allocations (in order of preference) are:

• 64 slots on 1 enclosure
• 32 slots on 2 enclosures
• 22 slots on 1 enclosure and 21 slots on 2 enclosures
• 16 slots on 4 enclosures

bsub -n 64 -R "cu[excl:maxcus=8:usablecuslots=10]" myjob

Allocates 64 slots over 8 or less compute units in groups of 10 or more slots per compute unit (with one compute unit possibly using less than 10 slots). The default compute unit type set in COMPUTE_UNIT_TYPES is used, and are used exclusively by myjob.

bsub -n 58 -R "cu[balance:type=rack:usablecuslots=20]" myjob

Provides a balanced allocation of 58 slots with at least 20 slots in each compute unit of type rack. Possible allocations are 58 slots in 1 rack or 29 slots in 2 racks.

Jobs submitted with balance requirements choose compute units based on the pref keyword secondarily, as shown in the following examples where cu1 has 5 available slots and cu2 has 19 available slots.

bsub -n 5 -R "cu[balance:pref=minavail]"

Runs the job on compute unit cu1 where there are the fewest available slots.

bsub -n 5 -R "cu[balance:pref=maxavail]"

Runs the job on compute unit cu2 where there are the most available slots. In both cases the job is balanced over the fewest possible compute units.

Cu string in application profiles

See Chapter 23, "Working with Application Profiles" for information about how resource requirements in application profiles are resolved with queue-level and job-level resource requirements.