Practical Exercises To be completed by 5/1/2008

Objective: build a two-node computational cluster from your desktop and the node:

MPI installation

You will configure a small two node Linux cluster out of your desktop and the node machine, say unisys18 and node18. Below, please use the host names of your desktop and the node, accordingly.
Download MPICH2 package
On both the desktop and the node, install rsh client, rsh server and MPICH2:

apt-get install rsh-client
apt-get install rsh-server 
apt-get install libssl-dev 
dpkg -i mpich2-smpd_1.0.5-0_i386.deb

In file /etc/hosts.equiv on both the machines, put the host names of your desktop and the node, for example:

unisys18
node18

It will allow rsh between the hosts without password. MPI needs this feature to start its processes on compute nodes. Make sure you can rsh between the computers without password.

Setup NFS shared directories

Users who run MPI simulations need to have their HOME directory on a shared file system (NFS) so MPI would be able to access the user files for I/O.
On the desktop (unisys18), create NFS export directory /home/nfs/MPI

mkdir -p /home/nfs/MPI

Add them to /etc/exports for sharing with the node:

/home/nfs/MPI  node18(rw)

Restart portmap and NFS server on the desktop machine:

/etc/init.d/portmap restart
/etc/init.d/nfs-kernel-server restart

Set up automount on the node (node18):
Create a mount point

mkdir /home/nfs

Make sure the entry for automount in file /etc/nsswitch.conf shows

automount   files

Create automount maps as follows:
/etc/auto.master:

/home/nfs  /etc/auto.MPI  --timeout=120

/etc/auto.MPI:

MPI  -o rw,hard,intr  unisys18:/home/nfs/MPI

Start the automount on the node:

/etc/init.d/autofs stop
/etc/init.d/autofs start

Step into the NFS directories on the node to make sure they get mounted:

cd /home/nfs/MPI
df -h

Create common users on the hosts

Users runing MPI simulations need to have accounts with the same UID on all the machines. Account for SGE, sgeadmin, should also have the same UID on all the hosts. It can be accomplished by creating the accounts on the master machine (desktop) then using either NIS or LDAP to share the accounts with the nodes. Alternatively, you can manually create the user accounts with the same UID on the nodes.
The steps below assume you already have LDAP running on your computers since lesson 8 (the desktop as LDAP server and the node as LDAP client) and group ldap (gid=1200) exists. Otherwise, you will need to create the user accounts manually on the node.
On the desktop, create two users, mpiua and sgeadmin:

useradd -m -s /bin/bash -d /home/nfs/MPI/mpiua -u 1300 -g ldap mpiua
useradd -s /bin/bash -g ldap sgeadmin
passwd mpiua

Add the created users into ldif file as you did in lesson 8:

./migrate_passwd.pl  /etc/passwd > passwd.ldif

Edit passwd.ldif and remove all entries except for users mpiua and sgeadmin then

ldapadd -x -D 'cn=Manager,dc=uni18,dc=unisys,dc=com' -W -f passwd.ldif

If LDAP doesn't work on the desktop or the node is not bound to the server for some reason, then create accounts for mpiua and sgeadmin on the node with the UID and GID same as on the desktop.
Make sure user mpiua can rsh to the node without password and get into his NFS-mounted HOME directory:

su - mpiua
rsh node18

Run MPI exercises

A. In deamonless mode

Note, the exercises with MPI runs should be done as a user who has HOME dir on the NFS shared file system. Become user mpiua:

su - mpiua

Download MPI C codes discussed in the lecture notes, ex1.c, ex2.c, ex3.c, and ex4.c

Create a machine file, machines, with the list of the hosts running MPI:

unisys18
node18

Set PATH variable for MPI:

export PATH=$PATH:/usr/local/mpich2_smpd/bin

Compile ex1.c and ex2.c and run them as follows:

mpicc  -o ex1.x ex1.c
mpicc  -o ex2.x ex2.c

export MPIEXEC_RSH=rsh

mpiexec -rsh -nopm -n 2 -machinefile machines $HOME/ex1.x
mpiexec -rsh -nopm -n 2 -machinefile machines $HOME/ex2.x

Code ex3.c implies to run on 4 CPUs so you need to modify file machines as follows:

unisys18
unisys18
node18
node18

Compile ex3.c and run:

mpicc  -o ex3.x ex3.c
mpiexec -rsh -nopm -n 4  -machinefile machines $HOME/ex3.x

Compile and run ex4.c:

mpicc  -o ex4.x ex4.c
mpiexec -rsh -nopm -n 4  -machinefile machines $HOME/ex4.x

B. With process management daemon `smpd`

Start smpd daemon on the desktop and the node machine:

smpd -s
rsh node18 /usr/local/mpich2_smpd/bin/smpd -s

Run 2 and 4 instances of command uname -n through mpiexec and see the process ranks ( -l option):

mpiexec -l -n 2 uname -n
mpiexec -l -n 4 uname -n

Run ex1.x, ex2.x, ex3.x, and ex4.x through mpiexec:

mpiexec -n 2 ex1.x 
mpiexec -n 2 ex2.x 
mpiexec -n 4 ex3.x 
mpiexec -n 4 ex4.x

Shutdown smpd on the desktop and the node:

smpd -shutdown
smpd -shutdown node18

Installation of Sun Grid Engine (SGE 6.0)

The installation should be done by user root. Create directory /usr/local/sge on both the desktop and the node. Download four files, sge-6.0u10-bin-lx24-x86.tar.gz, sge-6.0u10-common.tar.gz, sge_libmotif-x86.tar.gz, and mpich2-60.tgz, into directory /usr/local/sge on the desktop machine. Uncompress them and assign sgeadmin ownership:

cd /usr/local/sge
tar -zxvf sge-6.0u10-bin-lx24-x86.tar.gz 
tar -zxvf sge-6.0u10-common.tar.gz 
tar -zxvf sge_libmotif-x86.tar.gz 
tar -zxvf mpich2-60.tgz
chown -R sgeadmin /usr/local/sge

On both the computers, edit file /etc/services and add new services for SGE qmaster and execution daemons:

sge_qmaster     536/tcp
sge_execd       537/tcp

On the desktop computer, run installation of the Queue Master,

cd /usr/local/sge
./install_qmaster

During the installation, accept the default settings:

      $SGE_ROOT          = /usr/local/sge
      service            = sge_qmaster 
  communication service  = sge_execd 
      admin user account = sgeadmin
      qmaster spool directory = /usr/local/sge/default/spool/qmaster

When prompted,
Do you want to install Grid Engine as admin user >sgeadmin< (y/n) >> [y] -- type "y"
Enter cell name [default] >> -- "Enter"
Are you going to install Windows Execution Hosts? (y/n) [n] >> -- "n"
did you install this version with >pkgadd< or did you already verify and set the file permissions of your distribution (y/n) -- type "n"
Do you want to verify and set your file permissions (y/n) [y] >> -- type "y"
Are all hosts of your cluster in a single DNS domain (y/n) [y] >> -- type "y".
Please choose a spooling method (berkeleydb|classic) [berkeleydb] >> -- classic
Grid Engine group id range. Please enter a range >> -- 20000-20100
The pathname of the spool directory of the execution hosts. Default: [/usr/local/sge/default/spool] >> -- accept (Enter)
administrator_mail Default: [none] >> -- accept (Enter)
Accept and keep just the "global" cluster configuration.
We can install the startup script that Grid Engine is started at machine boot (y/n) [y] >> -- type "y"
Do you want to use a file which contains the list of hosts (y/n) [n] >> -- n
Adding admin and submit hosts Host(s): -- type the host names of your desktop and the node, for example, unisys18 node18.
Do you want to add your shadow host(s) now? (y/n) [y] >> - n
Default configuration is [1] >> -- y

When the installation is done, add the following line into /etc/profile to initialize SGE environment variables in the login shell:

. /usr/local/sge/default/common/settings.sh

Run the installation of the execution host on the desktop:

./install_execd

Accept the default settings during the installation, except when prompted,
We can install the startup script that Grid Engine is started at machine boot (y/n) [y] >> -- type "y"

Archive the SGE installation directory and copy the tar ball to the node in directory /usr/local/sge:

cd /usr/local/sge
tar -cvf SGE.tar .
scp SGE.tar root@node18:/usr/local/sge

On the node machine, make sure account sgeadmin exists:

id sgeadmin

Uncompress SGE.tar and run installation of the execution host on the node:

cd /usr/local/sge
tar -xvf SGE.tar
./install_execd

Accept the default settings during the installation, except when prompted,
We can install the startup script that Grid Engine is started at machine boot (y/n) [y] >> -- type "y"

When the installation is done, add the following line into /etc/profile to initialize SGE environment variables in the login shell:

. /usr/local/sge/default/common/settings.sh

All SGE commands below should be executed only on the master machine -- your desktop. Run command qhost to see the hosts and their resources available in SGE:

HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
-------------------------------------------------------------------------------
global                  -               -     -       -       -       -       -
node18                  lx24-x86        1  0.00  250.9M   17.8M  634.9M     0.0
unisys18                lx24-x86        1  0.01  250.7M   36.5M  486.3M   72.0K

If you get an error "...command not found", get env variables for SGE by running

source /etc/profile

Run command qstat to see the status of available queues.

qstat -f

It should show two available queues, all.q@node18 and all.q@unisys18:

queuename                      qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@node18.rutgers.edu       BIP   0/1       0.00     lx24-x86      
----------------------------------------------------------------------------
all.q@unisys18.rutgers.edu     BIP   0/1       0.00     lx24-x86

Download a template for MPI parallel environmet, mpich2_smpd_rsh. Create a new parallel environment with command qconf:

qconf -Ap mpich2_smpd_rsh

See the list of the configured parallel environments:

qconf -spl

Start the SGE GUI configuration/monitoring tool, qmon:

qmon &

If qmon fails to start and gives you an error about some missing fonts, install the fond server and some x-fonts:

apt-get install xfs
apt-get install xfonts-100dpi xfonts-75dpi xfonts-scalable

Exit the X (Ctrl - Alt - Back space), then run startx
When qmon is started, a menu pops-up, click on "Queue Control", select all.q, click on "Modify", In Shell menu, Change "Shell /bin/csh" for "Shell /bin/bash". Click on Parallel Environment, move 'mpich2_smpd_rsh' from the left window (Available PEs) to the righ window (Referenced PEs). Click OK. Click "Done"; click "Exit" to exit Qmon.

Running MPI on SGE cluster

Download an SGE submit script for MPI jobs, mpi_script.sh. In the script, change myjob=ex.x for myjob=ex1.x
Make the script executable. Submit the script with command qsub:

qsub mpi_script.sh

Check the status of the job with command qstat. The results of the standard output of the run should be recorded in file MPI-stdo.output
In mpi_script.sh, modify the entry for myjob as follows:

myjob=ex3.x

Change the number of required processors from '2' to '4' in the script:

#$ -pe mpich2_smpd_rsh 4

Submit the job to the queue system:

qsub mpi_script.sh

Monitor the submitted job by runing command qstat It would wait in the queue forever (state qw) since there are only 2 slots available in the PE configuration:

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
     11 0.55500 MPI_Job    mpiua        qw    04/19/2007 10:03:47                                    4

To reconfigure the queues and PE, start qmon as user root. Select and click on "Queue Control", all.q queues, click on modify, increase the number of slots from 1 to 2, click "OK", click "Done". Exit Qmon. When you run qstat again, it should show no jobs in the queues. which means the job is done already. It has been dispatched to the machines after the modifications have been made.

top