                                 MPICH2 
                                 -----
                 Grid Engine Parallel Support for MPICH2
                 ---------------------------------------

Content
-------

1) Content of this directory hierarchy
2) mpich2.template
3) startmpich2.sh
4) start_mpich2
5) stopmpich2.sh
6) 'mpich.sh' job example
7) Copyright


1) Content of this directory hierarchy
--------------------------------------

This directory contains the following files and directories:

   README           this file 
   README.atm       desribes how to run MPI jobs over ATM
   startmpich2.sh      startup script for MPI
   stopmpich2.sh       shutdown script for MPI
   mpich2template   a MPICH PE template configuration for Grid Engine
                    (tight integration)
   mpich2.sh           a sample MPI job
   hostname         a wrapper for the hostname command (see README.atm)
   src              subdirectory with the helping program
   rsh              a wrapper around the default rsh command to access
                    the SGE built-in qrsh instead

Please refer to the "Installation and Administration Guide" Chapter "Support
of Parallel Environments" for a general introduction to the Parallel
Environment Interface of Grid Engine.


2) mpich2.template
-----------------

   Use this template as a starting point when establishing a parallel
   environment for MPICH2 with tight integration. You need to replace
   <the_number_of_slots> and <your_sge_root> with the appropriate information.

   Here is a list of problems for which tight integration provides solutions

   - resource limits are enforced also for tasks at slave hosts
   - resource consumption at slave hosts can be accounted
   - no need to write a customized terminate method to ensure
     that whole job is finished on qdel
 
   Here is a list of problems which are not solved by the tight integration

   - can't trigger job finish if application finishes partially


3) startmpich2.sh
-----------------

   The starter script 'startmpich2.sh' needs some command line arguments, to 
   be configured by use of either qmon or qconf. The first one is the path
   to the "$pe_hostfile" that gets transformed by startmpich2.sh into a
   MPICH2 machine file. On successful completion startmpich2.sh creates a
   machine file in $TMPDIR/machines to be passed to "mpirun" at job
   start.

   $TMPDIR is a temporary directory created and removed by the Grid Engine
   execution daemon.

   The second argument is the path to your MPICH2 installation. In this location
   the script expect a subdirectory with the daemons program (i.e. ./bin/smpd).
   This daemon will be started on all nodes.
   
   You also need to supply the -catch_rsh option to achieve a tight integration.
   Please have a look at the mpich2.template for more information.

   For the communication of each job a special port is calculated by the
   formular:

   port=$((JOB_ID % 5000 + 20000))

   You may want to change to any restrictions you have in your cluster. Depending
   on your job turn around time, it may also be necessary to change the suggested
   5000 to a higher value. Be sure, that after changing this it must also be
   changed in your job script and the stopmpich2.sh procedure.


4) start_mpich2
---------------

   You need to compile a small helping program, which will fork-off the daemon
   tasks on the slave nodes. Doing this in a program will force the shell not
   to be involved, like it would with just starting the tasks with &. The started
   qrsh-commands will start the daemons on the nodes, and leave only the one idling
   qrsh on the headnode of the job (per slave node).

   After setting the working directory to be $SGE_ROOT/mpich2_smpd/src execute
   the commands:

   $ ./aimk
   $ ./install.sh

   which will create the executable and put it in a directory in
   $SGE_ROOT/mpich2_smpd/bin.


5) stopmpich2.sh
----------------

   In the usual execution, this will shut down the daemons on the nodes in the
   reverse order as they were started. The reason is, that a a qrsh is no longer
   allowed at this point of execution, and an rsh is not really a tight integration.
   So the daemon of the headnode of the job (where also stopmpich2.sh is running)
   will shut down the smpds on another node with the built-in mpich2 communication.

   As this is only possible, as long as the daemon on the headnode is running, it
   has to be the last one to be shut down.

   If this fails, the last resort would be the shutdown of all qrsh created tasks
   on the node by SGE after the stopmpich2.sh finished.

   stopmpich2.sh will need one argument whicvh is the path to your MPICH2 installation
   and an optional -catch_rsh in case of a tight integration.


6) 'mpich2.sh' job example
-----------------------

   This is an example of a possible job script to start the mpihello program,
   which you find in another archive. 

   Instead of setting any $PATH by hand in this script, you may want to put
   the necessray commands in a file, which is sourced during a non-interacive
   login like .bashrc.


7) Copyright
------------

   (c) 2002 Sun Microsystems, Inc. Use is subject to license terms.
