Practical Exercises To be completed by 4/26/2006

Objective: build a two-node computational cluster from your desktop and the node:

MPI installation

You will configure a small two node Linux cluster out of your desktop and the node machine, say unisys18 and node18. Below, please use the host names of your desktop and the node, accordingly.
On both the desktop and the node, install rsh client, rsh server and MPICH:

apt-get install rsh-client
apt-get install rsh-server 
apt-get install mpich

In file /etc/hosts.equiv on both the machines, put the host names of your desktop and the node, for example:

unisys18
node18

It will allow rsh between the hosts without password. MPI needs this feature to start its processes on compute nodes. Make sure you can rsh between the computers without password.

Setup NFS shared directories

Users who run MPI simulations need to have their HOME directory on a shared file system (NFS) so MPI would be able to access the user files for I/O.
Sun Grid Engine (SGE) also needs to be installed on an NFS shared directory.
On the desktop (unisys18), create NFS export directories /home/nfs/MPI and /usr/local/sge

mkdir -p /home/nfs/MPI 
mkdir /usr/local/sge

Add them to /etc/exports for sharing with the node:

/home/nfs/MPI  node18(rw)
/usr/local/sge node18(rw)

Restart portmap and NFS server on the desktop machine:

/etc/init.d/portmap restart
/etc/init.d/nfs-kernel-server restart

Set up automount on the node (node18):
Create a mount point

mkdir /home/nfs

Make sure the entry for automount in file /etc/nsswitch.conf shows

automount   files

Create automount maps as follows:
/etc/auto.master:

/home/nfs  /etc/auto.MPI  --timeout=120
/usr/local /etc/auto.SGE  --timeout=120

/etc/auto.MPI:

MPI  -o rw,hard,intr  unisys18:/home/nfs/MPI

/etc/auto.SGE:

sge   -o rw,hard,intr  unisys18:/usr/local/sge

Start the automount on the node:

/etc/init.d/autofs stop
/etc/init.d/autofs start

Step into the NFS directories on the node to make sure they get mounted:

cd /home/nfs/MPI
df -h
cd /usr/local/sge
df -h

Create common users on the hosts

Users runing MPI simulations need to have accounts with the same UID on all the machines. Account for SGE, sgeadmin, should also have the same UID on all the hosts. It can be accomplished by creating the accounts on the master machine (desktop) then using either NIS or LDAP to share the accounts with the nodes. Alternatively, you can manually create the user accounts with the same UID on the nodes.
The steps below assume you already have LDAP running on your computers since lesson 8 (the desktop as LDAP server and the node as LDAP client) and group ldap (gid=1200) exists. Otherwise, you will need to create the user accounts manually on the node.
On the desktop, create two users, mpiua and sgeadmin:

useradd -m -s /bin/bash -d /home/nfs/MPI/mpiua -u 1300 -g ldap mpiua
useradd -s /bin/bash -g ldap sgeadmin
passwd mpiua

Add the created users into ldif file as you did in lesson 8:

./migrate_passwd.pl  /etc/passwd > passwd.ldif

Edit passwd.ldif and remove all entries except for users mpiua and sgeadmin then

ldapadd -x -D 'cn=Manager,dc=uni18,dc=unisys,dc=com' -W -f passwd.ldif

If LDAP doesn't work on the desktop or the node is not bound to the server for some reason, then create accounts for mpiua and sgeadmin on the node with the UID and GID same as on the desktop.
Make sure user mpiua can rsh to the node without password and get into his NFS-mounted HOME directory:

su - mpiua
rsh node18

Give user sgeadmin ownership of directory /usr/local/sge:

chown  sgeadmin /usr/local/sge

Run MPI exercises

Note, the exercises with MPI runs should be done as a user who has HOME dir on the NFS shared file system. Become user mpiua:

su - mpiua

Download MPI C codes discussed in the lecture notes, ex1.c, ex2.c, ex3.c, and ex4.c

Create a machine file, machines, with the list of the hosts running MPI:

unisys18
node18

Compile ex1.c and ex2.c and run them as follows:

mpicc  -o ex1.x ex1.c
mpicc  -o ex2.x ex2.c
mpirun -np 2 -machinefile machines ex1.x
mpirun -np 2 -machinefile machines ex2.x

Code ex3.c implies to run on 4 CPUs so you need to modify file machines as follows:

unisys18
unisys18
node18
node18

Compile ex3.c and run:

mpicc  -o ex3.x ex3.c
mpirun -np 4  -machinefile machines ex3.x

Compile and run ex4.c:

mpicc  -o ex4.x ex4.c
mpirun -np 4  -machinefile machines ex4.x

Installation of SGE

The installation should be done by user root. Download two files, sge-5.3p6-bin-glinux.tar.gz and sge-5.3p6-common.tar.gz, into directory /usr/local/sge on the desktop machine. Uncompress them and assign sgeadmin ownership:

cd /usr/local/sge
tar -zxvf sge-5.3p6-bin-glinux.tar.gz
tar -zxvf sge-5.3p6-common.tar.gz
chown -R sgeadmin /usr/local/sge

On both the computers, edit file /etc/services and add a new service for SGE communication daemon:

sge_commd       536/tcp

On the desktop computer, run installation of the Queue Master,

cd /usr/local/sge
./install_qmaster

During the installation, accept the default settings:

      $SGE_ROOT          = /usr/local/sge
      service            = sge_commd
      admin user account = sgeadmin

When prompted,
did you install this version with >pkgadd< or did you already verify and set the file permissions of your distribution (y/n) -- type "n"
Are all hosts of your cluster in a single DNS domain (y/n) [y] >> -- type "y".
Accept and keep just the "global" cluster configuration.
We can install the startup script that Grid Engine is started at machine boot (y/n) [y] >> -- type "n"
Adding admin and submit hosts Host(s): -- type the host names of your desktop and the node, for example, unisys18 node18.

When the installation is done, add the following line into /etc/profile to initialize SGE environment variables in the login shell:

. /usr/local/sge/default/common/settings.sh

Create a symbolic link of the SGE startup script:

ln -s /usr/local/sge/default/common/rcsge /etc/init.d/rcsge

Run the installation of the execution host on the desktop:

./install_execd

Accept the default settings during the installation, except when prompted,
We can install the startup script that Grid Engine is started at machine boot (y/n) [y] >> -- type "n"

On the node machine, make sure account sgeadmin exists:

id sgeadmin

Run installation of the execution host on the node:

./install_execd

Accept the default settings during the installation, except when prompted,
We can install the startup script that Grid Engine is started at machine boot (y/n) [y] >> -- type "n"

When the installation is done, add the following line into /etc/profile to initialize SGE environment variables in the login shell:

. /usr/local/sge/default/common/settings.sh

Create a symbolic link of the SGE startup script:

ln -s /usr/local/sge/default/common/rcsge /etc/init.d/rcsge

All SGE commands below should be executed only on the master machine -- your desktop. Run command qhost to see the hosts and their resources available in SGE:

HOSTNAME             ARCH       NPROC  LOAD   MEMTOT   MEMUSE   SWAPTO   SWAPUS
-------------------------------------------------------------------------------
global               -              -     -        -        -        -        -
node18.rutgers.edu   glinux         1  0.00   124.0M    22.2M   244.1M      0.0 
unisys18.rutgers.edu glinux         1  0.00   124.0M    22.2M   244.1M      0.0

If you get an error "...command not found", get env variables for SGE by running

source /etc/profile

Run command qstat to see the status of available queues.

qstat -f

It should show two available queues, node18.q and unisys18.q:

queuename            qtype used/tot. load_avg arch       states
----------------------------------------------------------------------------
node18.q             BIP   0/1       0.00     glinux     
----------------------------------------------------------------------------
unisys18.q           BIP   0/1       0.00     glinux

Download a template for MPI parallel environmet, mpich.template. Edit file mpich.template and replace unisys18.q node18.q with your queue names. Create a new parallel environment with command qconf:

qconf -Ap mpich.template

See the list of the configured parallel environments:

qconf -spl

Start the SGE GUI configuration/monitoring tool, qmon:

qmon &

If qmon fails to start and gives you an error about some missing fonts, install the fond server and some x-fonts:

apt-get install xfs
apt-get install xfonts-100dpi xfonts-75dpi xfonts-scalable

Exit the X (Ctrl - Alt - Back space), then run startx
If qmon fails to start due to missing libXltree.so, update the loadable module path:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/sge/lib/glinux

When qmon is started, a menu pops-up, click on "Queue Control", select one of the queues, click on "Modify", Change "Shell /bin/csh" for "Shell /bin/bash". Similarly, select the other queue and perform the same modification. Click "Done"; click "Exit" to exit Qmon.

In file /usr/local/sge/mpi/rsh, modify the following lines by adding "-V" option to them:

echo $SGE_ROOT/bin/$ARC/qrsh -inherit -nostdin $rhost $cmd
exec $SGE_ROOT/bin/$ARC/qrsh -inherit -nostdin $rhost $cmd
else
echo $SGE_ROOT/bin/$ARC/qrsh -inherit $rhost $cmd
exec $SGE_ROOT/bin/$ARC/qrsh -inherit $rhost $cmd

They should read:

echo $SGE_ROOT/bin/$ARC/qrsh -V -inherit -nostdin $rhost $cmd
exec $SGE_ROOT/bin/$ARC/qrsh -V -inherit -nostdin $rhost $cmd
else
echo $SGE_ROOT/bin/$ARC/qrsh -V -inherit $rhost $cmd
exec $SGE_ROOT/bin/$ARC/qrsh -V -inherit $rhost $cmd

Running MPI on SGE cluster

Download an SGE submit script for MPI jobs, mpi_script.sh. In the script, change myjob=ex.x for myjob=ex1.x
Make the script executable. Submit the script with command qsub:

qsub mpi_script.sh

Check the status of the job with command qstat. The results of the standard output of the run should be recorded in file MPI-stdo.output
In mpi_script.sh, modify the entry for myjob as follows:

myjob=ex3.x

Change the number of required processors from '2' to '4' in the script:

#$ -pe mpich 4

Submit the job to the queue system:

qsub mpi_script.sh

Monitor the submitted job by runing command qstat It would wait in the queue forever (state qw) since there are only 2 slots available in the PE configuration:

job-ID  prior name       user         state submit/start at     queue      master  ja-task-ID 
---------------------------------------------------------------------------------------------
   9     0   MPI_Job     mpiua        qw    04/24/2005 00:43:43

To reconfigure the queues and PE, start qmon as user root. Select and click on "Queue Control", select one of the queues, click on modify, increase the number of slots from 1 to 2, click "OK". Similarly, increase the number of slots in the second queue, click "Done". Click on "Parallel Environment Configuration", choose "mpich", click on "modify", increase the number of slots from 2 to 4, click "OK", click "Done", exit Qmon. When you run qstat again, it should show no jobs in the queues. which means the job is done already. It has been dispatched to the machines after the modifications have been made.

top