You are here
A Graphical User Interface of Petri Nets-based Activity Visualization
I am currently working on the project of "Activity Detection in Scientific Visualization" under the supervision of Dr. Ozer and Prof. Silver, and am making a graphical user interface (GUI) software based on our recently proposed activity detection framework “Token Tracking Petri Net” (TTPN). As semantic approach, the framework (and thus the GUI) allows scientists to semantically define an activity and generate a model of the defined activity from its semantic description. Particularly, a graph-based semantic technique - Petri Net is chosen to describe various stages of an activity as individual nodes. The fundamental purpose of the GUI is to provide hypothesis-testing through: 1. an interface that facilitates scientific modeling by promoting direct editing of the model by the scientist; 2. a visual connection between the model and the output; and 3. modes of visualization of activities that are scalable to data sets containing millions of features.
The GUI "TTPN", written with standard C++ language, is created by Qt, a cross-platform application framework that is widely used for developing application software with a GUI, and VTK, an open-source, freely available software system for 3D computer graphics, image processing and visualization. It is compiled and built in Ubuntu Linux system. Figure 1 shows the GUI with toolkit, a designed Petri Net model "Packet Formation Activity", and data panels.
Figure 1: The GUI visualizes activity detection within the 20th time step of a wall bounded turbulence simulation
I am listing some features of this GUI software here and continuously to add more functions to it, please feel free to check back soon to find more interesting features. I will upload executable file for Windows System upon finishing coding work.
Feature 1. Modeling activity graphically and interactively
The GUI use Petri Net as a tool of modeling and searching for interested activities from the phenomena evolution of the data set. Creating Petri Net is the first step for scientists to design their activity model. A Petri Net is a directed graph consisting of places, transitions, arcs and tokens where arcs represent the directed edges on the graph and places and transitions represent the nodes. In TTPN, places of Petri Net stand for object states while transitions represent condition. Scientists can create their Petri Nets by choosing places (Initial Place, Final Place, Ordinary Place) and transitions from the up-left panel titled "Primary Items", placing them on any positions of the "white board", and then connecting the added places and transitions with the arcs chosen from the same panel. Place can be only connected with transition and vise versa. The table titled "Summary of Current Petri Net" lists the number of each kind of nodes and arcs in the board. Any items (nodes, arcs, and labels) can be selected (clicked by left button), moved, rotated, and deleted. Users can move whole Petri Net by selecting all items with a mouse dragging square. Especially, place and transition has its own editable label. Whenever a node or arc is selected, its information will be listed in the table titled "Item Information". (See Figure 2)
The GUI use a text based config file to save the designed activity model. If a config file lies in local machine, users can load it to the GUI and show the model directly.
Figure 2: Petri Net for an activity model
After "drawing" Petri Net on the GUI, scientists can set model parameters from their domain knowledge to complete the model design and save it as a config file. Some parameters have been set with the creation of Petri Net, such as incomming and outgoing arcs. While some parameters like place and transition conditions are set from corresponding nodes of the Petri Net, others lile starting and ending time steps are input from the panel "Configure File Information"(Figure 3). Of course, scientsits can always access to and modify their activity models by loading the saved config file to the GUI and changing parameters. An example of config file can be found here.
Figure 3: Design the activity model
Feature 2. Token-feature visualization
The activity is detected and saved in a text based activity list. Activity list records the evolution process of each objects in the activity detection process. It is generated by a standalone program writtern by Dr. Ozer and will be incorporated to the GUI in future. An example of activity list can be found here. To visualize and analyze the detected activity, scientists should first load data sets. Figue 4 shows the Petri Net created earliar and the first time step of a time varying 3D data set in an isosurface rendering. Scientists can browse all data frames they select one by one by clicking Next/Last button or all at a time by clicking Play Forward/Play Backward button.
Figure 4: Visualize original data set and designed activity model
The GUI can read the activity list and visualize the detected activities in conjunction with the designed model. In each time step of the data, features that are currently performing the modeled activity will be shown as solid color while others become transparent. For each active feature, a corresponding token with the same color will be shown under the place standing for the status the feature is in. By using the GUI in this mode, a scientist can visualize an activity by viewing both the feature tracking results in the data and the token tracking process in the Petri Net simultaneously. Figure 5 shows a complete merge-split activity (complex events) which includes several events: arising, merging, continuation, and splitting. In the 31th time step, two features first arise in the data, and two tokens with the same colors appear correspondingly under P1 which stands for arising state. In the 32th time step, the two arising features are combined into one feature, and similarly the two tokens in P1 move to P2 representing merging state and combine into one token. In the 33th time step, the merged feature split into two features with three new features arise in the data at the same time. Correspondingly, the single token in P2 splits into two and move to P3 which represent splitting state, and three new tokens appear under P1.
Figure 5: Visualization of merge-split activity with designed model
Besides visualizing all features performing the modeled activity at current time step, the GUI can also visualize only the active features in a certain state by clicking the corresponding place in the model (Figure 6). In this state- or place-based feature visualization, a table listing all token ID of the clicked place (or all feature ID of the data in the corresponding status) will be generated, which provides a tool for scientists to locate and view each feature by choosing token ID from the token list table of the place (Figure 6 and 7). For large data with numerous features at different state, showing the position, shape, and ID of the active features in a certain state of the modeled activity or each individual feature is helpful to analyze the activity.
Figure 6: (a) Activity model and data with active features in the 33th time step of the merge-split activity. Click on P1 to see (b) the arising features and a list of tokens. Click on a token ID from the list and see (c) the corresponding feature in the data.
Figure 7: (a) Data with active features of Place 4 (P4 in Figure 1) in the 20th time step of the packet formation activity demonstrated in Figure 1, click on a token ID from the table of token list and see (b) the corresponding feature in the data.
Feature 3. Histogram of features
The GUI puts a bar chart under each place as a histogram which counts the number of token at each place in every time step. When clicking a bar chart, a histogram window with a series of green bars will pop up. Scientists can view both the data with features and the designed activity model with tokens at any time step by interactively clicking on the specific bar in the histogram window. If a scientist is using the histogram to test the hypothesis that the number of growing packets increases over time (Fig. 1), he/she can verify that the packets involved have grown by clicking on the bars of the histogram to bring up a rendering of the data with the involved packets highlighted (Figure 8).
Figure 8: Visualizing chosen frame by double-clicking corresponding bar in the histogram window
Feature 4. Whole activity visualization
The GUI can visualize the evolution process of all active features in the modelled activity by using multi-view. Figure 9 visualizes a merge-split activity as multiple stages in the multi-view window. A multi-view window is divided into two parts: in the upper part, features are extracted from the data frames that span the activity and shown frame by frame; while in the lower part, all detected features in this activity are combined into a new data and visualized.
Figure 9: Multi-view of all detected features and their evolutions in a merge-split activity