本文共 7067 字，大约阅读时间需要 23 分钟。
较大的集群都会设计单独的登录节点，用户只能ssh到登录节点，不能直接ssh到集群的任何主节点和计算节点。同时配置用户在计算节点之间的ssh互信，为了并行作业的运行。
登录节点也安装LSF，配置为LSF 静态Client或者MXJ值为0，也即不运行作业的客户端。集群的WEB节点与办公访问局域网一个网段。如需使用浮动client，主节点网卡需要
用户通过portal提交作业：
NIS ready；NFS/GPFS ready；
Use root to install.
Get NIS and NFS/GPFS ready.
lsf8.3_linux2.6-glibc2.3-x86_64.tar.Z
lsf8.3_lsfinstall_linux_x86_64.tar.Z
pac8.3_standard_linux-x64.tar.Z许可证文件platform_hpc_std_entitlement.dat
Put the package under /root/lsf
[root@S2 lsf]# gunzip lsf8.3_lsfinstall_linux_x86_64.tar.Z
tar -xvf lsf8.3_lsfinstall_linux_x86_64.tar
首先添加集群管理员lsfadmin。
cd lsf8.3_lsfinstall
vi install.config
[root@S2 lsf8.3_lsfinstall]# cat install.config
LSF_TOP="/opt/lsf" (安装目录)
LSF_ADMINS="lsfadmin" (先创建lsfadmin的用户名)
LSF_CLUSTER_NAME="platform" (集群名称，任意指定)
LSF_MASTER_LIST="s2 s3" (LSF管理节点)
LSF_ENTITLEMENT_FILE="/root/lsf/platform_hpc_std_entitlement.dat" (安装源许可证的地址)
LSF_TARDIR="/root/lsf/" (安装源文件包的地址)
./lsfinstall -f install.config
/opt/lsf/9.1/install
hostsetup
rhostsetup
安装目录下的/conf目录
[root@S2 conf]# source profile.lsf
Add source profile.lsf to /etc/profile
if no rsh, set ssh in lsf.conf
[root@S2 conf]# tail lsf.conf
LSF_RSH="ssh"
[root@S2 conf]# lsfstartup/lsfstop
或者
lsadmin limstatup/limshutdown
lsadmin resstartup/resshutdown
badmin hstartup/hshutdown
或者
lsf_daemons start/stop
[root@S2 conf]# lsid
IBM Platform LSF Express 8.3 for IBM Platform HPC, May 10 2012
Copyright Platform Computing Inc., an IBM Company, 1992-2012.
US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
My cluster name is platform
My master name is s2
You have new mail in /var/spool/mail/root
[root@S2 conf]# lsload
HOST_NAME status r15s r1m r15m ut pg ls it tmp swp mem
s2 ok 0.0 0.0 0.0 1% 0.0 1 0 151G 20G 61G
s4 ok 0.0 0.0 0.0 2% 0.0 1 2 183G 20G 62G
s6 ok 0.0 0.0 0.0 3% 0.0 1 2 3734M 2G 30G
s5 ok 0.0 0.0 0.0 5% 0.0 1 2 3468M 2G 30G
bsub sleep 100000
enable root to submit job:
LSF_ROOT_REX=local
重启LSF进程。
修改lsf.*配置文件后lsadmin reconfig
修改lsb.*配置文件后badmin reconfig
部分参数需要重启LSF主调度或者其他进程：badmin mbdrestart; lsadmin limrestart; lsadmin resrestart; badmin hrestart
Find the logs under log directory. LSF will run mainly 3 processes on each node, on master node will have 2 more.
Master: lim,res,sbatchd,mbatchd,mbsched
Compute:lim,res,sbatchd
Turn on debug in command line:
Run lim -2 directly on node to check why lim not startup.
目录/etc/init.d:
/etc/init.s/lsf lsf服务自启动脚本
目录/apps/platform/8.3/lsf/conf:
lsf.conf lsf配置文件
lsf.cluster.cluster83 集群配置文件
lsf.shared 共享资源定义文件
./lsbatch/cluster83/configdir/lsb.* 调度系统配置文件
lsb.users lsf用户与用户组配置文件
lsb.queues lsf队列配置文件
lsb.params lsf调度参数配置文件
lsb.applications lsf应用配置文件
lsb.hosts lsf机器与机器组配置文件
lsb.resources lsf资源配置文件
lsb.modules lsf模块配置文件
bsub:提交作业；
bjobs:查看作业信息；
bhist:查看作业历史；
lshosts:查看节点静态资源；
bhosts，lsload:查看节点状态和资源信息；
bqueues ：查看队列配置；
blimits：查看限制limit信息；
lsid:集群版本和主节点；
bmod:修改bsub option；
等等。
bsub –R “ ((type==LINUX2.4 && r1m < 2.0)||(type==AIX && r1m < 1.0)) ”
或者在队列lsb.queues或者lsb.application文件定义：
RES_REQ=select[((type==LINUX2.4 && r1m < 2.0)||(type==AIX && r1m < 1.0))]
bsub –R "select[type==any && swap>=300 && mem>500] order[swap:mem] rusage[swap=300,mem=500]" job1
bsub –R rusage[mem=500:app_lic_v2=1 || mem=400:app_lic_v1.5=1]" job1
bsub –R "select[type==any && swp>=300 && mem>500] order[mem]" job1
Modify lsb.queues, add following
Begin Queue
QUEUE_NAME = roundRobin
PRIORITY = 40
FAIRSHARE = USER_SHARES[[default,1]]
#USERS = userGroupA Define your own usergroup
End Queue
Run badmin reconfig to enable the change.
Run bqueues –l to check the queue’s configure
Add following queue to add hierarchicalshare policy:
Begin Queue
QUEUE_NAME = hierarchicalShare
PRIORITY = 40
USERS = userGroupB userGroupC
FAIRSHARE = USER_SHARES[[userGroupB,7] [userGroupC,3]]
End Queue
在lsb.queues中添加下列队列，注意节点组和用户组定义。
Begin Queue
QUEUE_NAME = verilog
DESCRIPTION = master queue definition cross-queue
PRIORITY = 50
FAIRSHARE = USER_SHARES[[user1,100] [default,1]]
FAIRSHARE_QUEUES = normal short
HOSTS = hostGroupC # resource contention
#RES_REQ = rusage[verilog = 1]
End Queue
Begin Queue
QUEUE_NAME = short
DESCRIPTION = short jobs PRIORITY = 70 # highest
HOSTS = hostGroupC
RUNLIMIT = 5 10
End Queue
Begin Queue
QUEUE_NAME = normal
DESCRIPTION = default queue
PRIORITY = 40 # lowest
HOSTS = hostGroupC
End Queue
badmin reconfig
提交作业，并查看队列的用户动态优先级变化：
bqueues –rl normal
配置最基本的slots抢占：
Begin Queue
QUEUE_NAME = short
PRIORITY = 70
HOSTS = hostGroupC # potential conflict
PREEMPTION = PREEMPTIVE[normal]
End Queue
Begin Queue
QUEUE_NAME = normal
PRIORITY = 40
HOSTS = hostGroupC # potential conflict
PREEMPTION = PREEMPTABLE[short]
End Queue
向两个队列提交作业，查看被preempt的作业的pending原因。
在lsb.users文件中添加：
Begin User
USER_NAME MAX_JOBS JL/P
user1 4 -
user2 2 1
user3 - 2
groupA 8 -
groupB@ 1 1
Default 2 -
End User
在lsb.hosts文件中：
Begin Host
HOST_NAME MXJ JL/U
host1 4 2
host2 2 1
host3 ! -
End Host
在lsb.queues中添加：
Begin Queue
QUEUE_NAME = myQueue
HJOB_LIMIT = 2
PJOB_LIMIT = 1
UJOB_LIMIT = 4
HOSTS = hostGroupA
USERS = userGroupA
End Queue
在lsb.resources文件定义全局general limits示例：
Begin Limit
USERS QUEUES HOSTS SLOTS MEM SWP
user1 - hostB - - 20%
user2 normal hostA - 20 -
End Limit
Begin Limit
NAME = limit1
USERS = user1
PER_HOST = hostA hostC
TMP = 30%
SWP = 50%
MEM = 10%
End Limit
Begin Limit
PER_USER QUEUES HOSTS SLOTS MEM SWP TMP JOBS
groupA - hgroup1 - - - - 2
user2 normal - - 200 - - -
- short - - - - - 200
End Limit
badmin reconfig
全局esub脚本在作业被提交是调用，可以被自动的或者显式的调用从而控制用户作业提交的行为。
编辑esub.project文件在$LSF_SERVERDIR下面（chmod为可执行）：
#!/bin/sh
if [ "_$LSB_SUB_PARM_FILE" != "_" ]; then
. $LSB_SUB_PARM_FILE
if [ "_$LSB_SUB_PROJECT_NAME" == "_" ]; then
echo "You must specify a project!" >&2
exit $LSB_SUB_ABORT_VALUE
fi
fi
exit 0
在lsf.conf中定义LSB_ESUB_METHOD=”project”
编辑elim文件elim.home，放置在$LSF_SERVERDIR下面。chmod为可执行。
#!/bin/sh
while true ; do
home=`df -k /home | tail -1 | awk '{printf "%4.1f", $4/(1024*1024)}'`
echo 1 home $home
sleep 30
done
编辑elim.root，放置在$LSF_SERVERDIR下面。chmod为可执行。
#!/bin/sh
while true ; do
root=`ps -ef | grep -v grep | grep -c ^root`
echo 1 rootprocs $root
sleep 30
done
#!/bin/sh
lic_X=0 ; num=0
while true ; do
# only want the master to gather lic_X
if [ "$LSF_MASTER" = "Y" ]; then
lic_X=`lmstat –a –c lic_X.dat | grep ...`>&2
fi
# only want training8, training1 to gather simpton licenses
if [ "`hostname`" = "training8" \
–o "`hostname`" = "training1" ] ; then
num=`lmstat –a –c simpton_lic.dat | grep ...`>&2
fi
# all hosts including master, will gather the following
root=`ps –efw | grep –v grep | grep –c root`>>1&2
tmp=`df –k /var/tmp | grep var | awk '{print $4 /1024}'`>&2
if [ "$LSF_MASTER" = "Y" ]; then
echo 4 lic_X $lic_X simpton $num rtprc $root tmp $tmp
else
echo 3 simpton $num rtprc $root tmp $tmp
fi
# the same INTERVAL values defined in lsf.shared
sleep 60
done
直接运行./elim.root查看elim输出是否正确。
在lsf.shared文件中添加rootprocs 定义，并在lsf.cluster resources Map中添加资源和节点的映射关系。
使能配置：lsadmin reconfig; badmin reconfig
lsload –l
转载地址：http://yrqwi.baihongyu.com/