High Performance Computational Cluster: STEP 3

Configuration of Grid Engine

Using qmon, configure all the queues for Batch and Parallel jobs only. 
If you run qmon on a remote host, you need to set xhost on your local machine and DISPLAY on the 
 remote host. For example, if I am sitting on unisys02 and running qmon on 
 node02, then on unisys02 I would type xhost +node02, and  on node02, I would run
export DISPLAY=unisys02:0.0 then run 
qmon & 
Most of the default parameters for the queues are acceptable.
Each queue is related to a particular host. For example, queue node01.q 
 should relate to node01.  Shell entry should be /bin/bash.    

Batch jobs in form of a script can be submitted to the queue system
 with qsub command.  Below is sample script
 for a job:

#!/bin/bash
#-----------------------------------------------------------------
# Template script for serial jobs to run on CODINE cluster.
# Modify it for your case and submit to CODINE with
# command "qsub batch_run.sh".

# You may want to modify the parameters for 
# "-N" (job queue name)

# You can monitor your jobs with command 
# "qstat -u your_username" or "qstat -f" to see all queues.
# To remove your job, run "qdel  job_id"
# To kill a running job, use "qdel -f job_id"
#----------------------------------------------------------------

#  Give a name to your job (for example, run-on-mphase):
#$ -N  run-in-queue
#
#  Specify the kind of shell script you use, for example, bash
#$ -S /bin/bash
#
# Standard output and error files:
#$ -o stdo.output 
#$ -e stderr.output 
#

# Start this script from the current working directory:
#$ -cwd
#

# Specify a local temporary directory on the running node:
TMPD=/tmp/$LOGNAME/$JOB_ID

echo "TMPD is $TMPD"
mkdir -p $TMPD


#  Specify the  current working directory:
CDIR=`pwd`

#   Specify the name of your executable, for example, "loop.x"
myjob=loop.x

#   Copy you compiled executable into $TMPD directory
#   along with the input files and run it from there:
cp $myjob $TMPD
cd $TMPD
$TMPD/$myjob

# if there is some output, copy it back into your current working directory
#cp output* $CDIR 
cd /tmp
rm -rf $TMPD
The above script can be used only with serial (non-parallel) jobs. To submit script batch_run.sh to the queue system, run qsub batch_run.sh. To monitor status of running jobs use command qstat To check on finished jobs, use command qacct -j. To delete a running job, use qdel -f Job_ID. You can run man pages on these commands to see their various options. Before running MPI through Grid Engine, MPI needs to be reconfigured and recompiled as follows: cd /usr/local/mpich-1.2.5/sbin/ ./mpiuninstall Untar-unzip MPICH into some directory, say, /tmp, ./configure --prefix=/usr/local/mpich-1.2.5 -rsh=rsh then run make; make install. To run MPI application through Grid Engine, you need to configure the parallel environment: start qmon; click on Parallel Environment Configuration; click Add; give it a name, for example, "mpich"; the number of slots is the number of your MPI nodes; create a queue list for MPI; in Start Proc Args, type /usr/local/sge/mpi/startmpi.sh $pe_hostfile In Stop Proc Args /usr/local/sge/mpi/stopmpi.sh Allocation Rule: $fill_up; boxes in the bottom are un-checked. To submit MPI compiled jobs to the queue system, use the script below:
#!/bin/bash -f
#
#---------------- SHORT COMMENT ----------------------------------------
# Template script for parallel MPI jobs to run on mphase Grid Engine cluster.
# Modify it for your case and submit to CODINE with
# command "qsub mpi_run.sh". 
 
# You may want to modify the parameters for 
# "-N" (job queue name), "-pe" (queue type and number of requested CPUs), 
# "myjob" (your compiled executable). 
 

# You can compile you code, for example myjob.c (*.f), with GNU mpicc or
#  mpif77 compilers as follows:
# "mpicc -o myjob myjob.c" or "mpif77 -o myjob myjob.f"
 
# You can monitor your jobs with command
# "qstat -u your_username" or "qstat -f" to see all queues.
# To remove your job, run "qdel job_id"
# To kill running job, use "qdel -f job_id"


# ------Attention: #$ is a special CODINE symbol, not a comment -----
#
#   The name, which will identify your job in the queue system 
#$ -N MPI_Job
#
#   Queue request, mpich. You can specify the number of requested CPUs, 
#   for example, from 2 to 3
#$ -pe mpich 2-3 
#
# ---------------------------
#$ -cwd
#$ -o  MPI-stdo.output 
#$ -e  MPI-stderr.output 
#$ -v  MPIR_HOME=/usr/local/mpich-1.2.5
# ---------------------------

echo "Got $NSLOTS slots."

#   Put the name of your compiled MPI file, for example, "cpi" 
myjob=cpi

#  Don't modify the line below if you don't know what it is
$MPIR_HOME/bin/mpirun -np $NSLOTS -machinefile $TMPDIR/machines $myjob
Modify the script above, mpi_run.sh, for your case; compile one of your MPI codes with mpicc or mpif77; the name of the compiled code into the script for myjob=...; submit the script to the queue system with qsub mpi_run.sh and see how it runs. Building Linux Vitual Server: STEP 3 MON scheduler and alert management tool In the previous exercises, you were using the bash script, httpdMonitor.sh, to monitor the status of httpd on the real servers and remove/add the servers in IPVSADM table upon their availability. However, httpdMonitor.sh is limited to monitoring httpd service only so it would be nice to have a more general monitoring tool, MON, suitable for monitoring various services. To install MON, you can get the source and installation instructions from http://www.kernel.org/software/mon/, however, you would save your time if you follow instructions below. First, make sure that perl is installed on the balancer computer, then cd /usr/include h2ph *.h sys/*.h asm/*.h Install the following modules for MON to function: -Mon::* -Time::Period -Time::HiRes -Convert::BER The files with the modules, Mon-0.11.tar.gz, Period-1.20.tar.gz, Time-HiRes-01.20.tar.gz, Convert-BER-1.31.tar.gz are located in 192.168.5.250:/usr/src/cd80/LVS directory. Mount the directory onto the balancer machine; copy the files into /tmp directory; gunzip/un-tar them; for each module, run installation: perl Makefile.PL make make test and if that goes well, make install Create directories /usr/lib/mon/mon.d /usr/lib/mon/alert.d /etc/mon From 192.168.5.250:/usr/src/cd80/LVS, get mon-0.99.1.tar.gz, lvs.alert, S99mon, mon.cf. Copy mon and http.monitor into /usr/lib/mon/mon.d; copy lvs.alert into /usr/lib/mon/alert.d; copy auth.cf and mon.cf into /etc/mon; copy S99mon into /etc/rc.d/init.d. Modify configuration file /etc/mon/mon.cf for your balancer and servers, then start MON with command /usr/lib/mon/mon.d/mon -f Check status of the servers with /sbin/ipvsadm -L Stop one of the web servers and check if MON notices its failure and modifies the IPVSADM table. You can also run mon through the startup script, S99mon. If you happen to need MON for monitoring the other services, for example, telnet, ftp, SNMP, etc, you would need to copy appropriate monitors, which come with the distribution, into /usr/lib/mon/mon.d/ directory; configure /etc/mon/mon.cf; then re-start mon.