A. Sensors/Actuators and Interaction Objects

Interaction objects extend application computational objects with interaction and steering capabilities, by providing them with co-located sensors and actuators. Computational objects are the data-structures/objects used by the application. Sensors enable the object to be queried while actuators allow it to be steered. Efficient abstractions are essential for converting computational objects to interaction objects especially when the computational objects are distributed and dynamic. In the DISCOVER system, this is achieved by deriving the computational objects from a virtual interaction base class of the DISCOVER Distributed Interaction Object library. The derived objects define its interaction interfaces as a set of Views that they can provide and a set of Commands that they can accept. Views represent sensors and define the type for information that the object can provide. For example, a Grid object might export views for its structure and distribution. Similarly, a GridFunction (application field defined on a grid) object might export views such as iso-surface plots, norms, and maximum/minimum values. Commands represent actuators and define the type of controls that can be applied to the object. Commands for the Grid object may include refine, coarsen, and redistribute. Similarly, those for the GridFunction may set or reset its data values. Interaction agents then export this interface to the interaction server using a simple Interaction IDL (Interface Definition Language) that can be easily interfaced to standard distributed object interfaces like CORBA and Java RMI. Interaction objects can be either local to a single computational node, distributed across multiple nodes, or shared between some or all of the nodes. Distributed objects have an additional distribution attribute that describes their layouts. DISCOVER interaction objects can be created or deleted during application execution and can migrate between computational nodes. Furthermore, a distributed interaction object can modify its distribution at any time.

1. Local, Global & Distributed Interaction Objects

Interaction objects can be classified based on the address space(s) they can span during the course of computation as local, global, and distributed objects. Local interaction objects are created locally by a computational node. These objects may migrate to another processor during the lifetime of the application, but exist in a single processor's address space at any point of time. Multiple instances of a local object could exist on different processors at the same time. Global interaction objects are similar to local objects, except that there can be exactly one instance of the object (across all processors) at any time. A distributed interaction object spans multiple processors' address spaces. An example is a distributed array partitioned across available computational nodes. These objects contain an additional distribution attribute that maintains its current distribution type (blocked, inverse space filling curve-based, or custom) and layout. This attribute can change during the lifetime of the object if the object is redistributed. Like local and global interaction objects, distributed objects can be dynamically created, deleted, or redistributed. In order to enable interaction with distributed objects, each distributed type is associated with gather and scatter operations. Gather aggregates distributed components of the objects while scatter performs the reverse operation. Canned gather/scatter operations are provided for popular distributions (block-based, Space Filling Curve-based, etc.). However, custom distributions can be defined by explicitly providing these operations.

2. Definition and Deployment of Interaction Objects

In the presented framework, interaction objects are defined by deriving an existing computational object from a library of interaction virtual classes provided. Transforming an existing computational object into an interaction object is performed in two steps:

1. The computational object is derived from an appropriate virtual interaction class, depending on whether they are local, global or distributed.

2. Views and commands relevant to the computational object are defined and registered. This involves defining and implementing the methods that will perform the desired functionality (generate a view or execute a command), if they do not already exist. Registering a view/command consists of providing a name for the view/command and a callback that is invoked to process an associated request. For example, computing the desired one-dimensional slice corresponding to a 1-D Plot view; or setting the value of a variable in response to a SetValue command. This step is accomplished by overriding specific virtual functions of the interaction base class the computational object is derived from.

Non-object-oriented (C/Fortran) data-structures can converted into interaction objects by first defining C++ wrappers to the objects. The resulting computational objects are then converted into interaction objects as described above. Although this requires some application modification, the wrappers are only required for those data-structures that have to be made interactive, and the effort is far less than rewriting the entire application to be interactive. We have successfully applied this technique to enable interactivity within the Fortran-based IPARS parallel oil-reservoir simulator developed at the Center for Subsurface Modeling, University of Texas at Austin.

B. A Control Network for Interaction and Steering

The control network has a hierarchical "cellular" structure with three components as shown in Figure 2. Computational nodes are partitioned into interaction cells, each cell consisting of a set of Discover Agents and a Base Station.

The number of nodes per interaction cell is programmable. Discover Agents are present on each computational node and manage run-time references to the interaction objects on the node. The Base Station maintains information about interaction objects for the entire interaction cell. The highest level of the hierarchy is the Interaction Gateway that provides a Java-enabled proxy to the entire application. The cellular control network is automatically configured at run-time using an underlying messaging environment (e.g. MPI) and the available number of processors.

1. Discover Agents, Base Stations and Interaction Gateway

Every computation node houses a Discover Agent (DA) that maintains a local object registry of all interaction objects currently active and registered by that node and maintains references to them. The interaction interfaces for the objects in the local registry are exported by each DA (using the interaction IDL) to its Base Stations. Base Stations (BS) form the next level of control network hierarchy. They maintain interaction object registries containing interaction interfaces only, for an entire interaction cell and export these to the Interaction Gateway. The Interaction Gateway (IG) represents an interaction proxy for the entire application. It exports the interaction interfaces provided by the all interaction objects of the application and is responsible for interfacing with external interaction servers or brokers, delegating interaction requests to the appropriate base stations and discover agents, and for combining and collating responses. Object migrations and re-distributions are handled by the respective DAs (and BSs if the migration/re-distribution is across interaction cells) by updating corresponding registries. Interactions between the Server and the IG are achieved using two approaches. In the first approach, the IG connects to the Server and performs object serialization to export all the interaction objects exported by the application to the server. A set of Java classes at the server parses the interaction IDL stream to de-serialize the interaction objects. Interaction requests coming from the interacting clients are routed to the IG through the Server. The IG in turn parses the interaction request and forwards it to the appropriate DA(s) that in turn respond to the request. In the second approach, the Interaction Gateway uses the Java Native Interface to create Java mirrors of registered interaction objects. These mirrors are registered with a RMI (Remote Method Invocation) registry service also executing at the IG. This enables the Server to gain access to and control the interaction objects using the Java RMI API. We are currently evaluating the performance overheads of using Java RMI and JNI.

C. Control Network Initialization and Interaction Sequences

The initialization sequence of interactions between the Discover Agents, corresponding Base Stations, Interaction Gateway, and the external Interaction Server (IS) are illustrated in Figure 3. The application, using the DIOS API, creates and registers its interaction objects with its local DAs. The BSs setup interaction cells, and establish communication with their respective DAs to initialize their cell object registries. At the IG, the central object registry is created. The DAs now export object registries to their respective BSs that are forwarded to the IG. The IG now communicates with the IS to register the application and export the central object registry to the IS. At the IS, the interaction IDL messages are parsed and interaction objects are recreated. Once the initial object registration process is complete, the application begins its computations. It is assumed that the application proceeds through interleaved computation and interaction phases.

The interaction phase is shown in Figure 4. The IG looks for any outstanding interaction requests from the IS. If there are any incoming requests, it parses the header of the request to identify the compute node from which the object was exported. In the case of a distributed object, this would be a set of nodes. The interaction request is now forwarded to the destination node(s) identified. All other nodes are sent a go-ahead message indicating that there is no interaction request for any of the objects they registered during this interaction phase. The IG then waits until the corresponding response arrives from the DAs. If the responding object is distributed, the IG performs a gather operation on the individual responses. The response then shipped to the IS.

1. Interacting with Local and Distributed Objects

 
The processing of interaction requests is slightly different for local and distributed objects. In the case of a local object residing on a single computational node, processing is straightforward. On receiving the request from the IS, IG parses the message header to identify the computational node that registered the object. The steering request is then forwarded to the appropriate node. The corresponding DA on the node uses its reference to the associated interaction object to process the request. The response generated is then sent back to the IG, which in turn, exports it to the IS. This process is illustrated in Figure 5. Processing interaction requests in the case of a distributed object is very similar as shown in Figure 6. The IG once again parses the message header to identify the nodes across which the object is distributed. The Gateway then forwards the steering request to these nodes. The corresponding DAs receive the steering request, look up the associated interaction objects and locally process the message. Each DA sends its portion of the response back to the IG. The IG then performs a gather operation to collate the responses and forwards them to the IS.

D. Experimental Evaluation

Sample Application Driver

This section summarizes the results of an initial experimental evaluation of the DIOS library using the Sun E10000 cluster. The evaluation consists of 4 experiments.

1. End-to-end Steering Latency - The DISCOVER system exhibits latencies varying between 10 - 45 ms for transfer of data sizes ranging from a few bytes to 10KB. This is comparable to steering systems like the MOSS and Autopilot systems, as reported in.

2. Minimum Steering Overhead - In the minimum steering mode, the application continuously updates the external interactivity system (web server and collaborating clients) with changes in the important steering parameters of the simulation. The overhead incurred in exporting scalars were measured with respect to the average time spent in a computation iteration and this was found to be within a small fraction of the time spent in computation (ranging from 1% for exporting a single scalar parameter to about 5% for exporting 10 scalars).

3. Object Registration Overhead - One of the key sources of overheads was object registration process, including interaction IDL generation and exporting at the Discover Agents (to the Base Station), and IDL processing and exporting at the Base Station and Gateway. These steps are necessary for registering the interaction objects at startup. The different overheads measured were (1) 500 msec at each Discover Agent, (2) 10 ms at each Base Station for each compute node in its interaction cell and (3) 10 ms at the Gateway for each Base Station in the control network. We are current working on optimizing the registration process. Note that this is a one-time cost required at startup only.

4. Query Processing and Steering Overhead - This cost largely depends to the nature of interaction/steering requested, and the processing required at the application to satisfy the request and generate a response. In the experiments conducted, data sizes generated (for View requests) ranged from a few bytes to about 10 KB and this took between 10-45 ms. Command processing took about 30 ms to refine a grid hierarchy, 1.2 sec to checkpoint execution state to a file and 45 ms to rollback to a previous checkpoint state and resume execution. In this experiment, distributed collaborating clients generated all view and command requests.