These html pages are based on the PhD thesis "Cluster-Based Parallelization of Simulations on Dynamically Adaptive Grids and Dynamic Resource Management" by Martin Schreiber.
There is also more information and a PDF version available.

8.2 Invasive client layer

The Invasive paradigm was originally developed for specialized Invasive hardware, whereas we apply this paradigm on HPC shared-memory systems. This leads to different programming interfaces which are discussed next.

This results in a simplified command space without infects and also no claim for Invasive Computing with our iteration-based shared-memory application. We next discuss the required constraint system for our simulations on dynamically changing grids (Sec. 8.2.1), the communication to the resource manager (Sec. 8.2.2) and the Invasive Computing API for shared-memory systems (Sec. 8.2.3).

8.2.1 Constraints

To schedule the compute resources for which the applications compete for, the applications have to provide information on their current state. Such a constraint specification can lead to very complex structures if requiring a general applicability on heterogeneous systems. E.g. an application can either require 3 cores and one accelerator or in case that the accelerator was not invadable, require more than three cores. This can be expressed with a tree-structure, resulting in a constraint-hierarchy (see [BRS+13]). Then, different constraint constellations can be combined with AND or OR relations. On the one hand, this constraint hierarchy yields a high flexibility, whereas on the other hand, forwarding this information to the resource manager can lead to expensive communication overheads to/from the resource manager. This is due to serialization of the constraint tree, sending the serialized version to the RM, receiving it and unpacking it by de-serialization. Additional complexity is generated inside the RM which has to evaluate the constraint hierarchy by using the constraint trees of different applications to optimize the resource assignment concurrently scheduled applications.

With our HPC shared-memory systems considered in this work which are only based on homogeneous cores, we decided to avoid such a constraint hierarchy by using a fixed number of non-hierarchical constraint properties in a list. Our constraint system is then based on the following properties:

Once specified by the application, these constraints then have to be communicated to the resource manager with details on the resource scheduling for scalability graphs and distribution hints further described in Section 8.4.

8.2.2 Communication to resource manager

A centralized resource manager is used which is responsible to optimize the resources based on constraints received from applications. Our client-server design (see Fig 8.1) suggests a message-based communication with the clients transferring constraints of a typically small message size to the server running as a dedicated process. Then the server (resource manager) requires receiving the message, processing it and sending one or more messages to the client (application). To overcome message processing latencies, also the ability of asynchronous communication is required.

All these requirements are fulfilled by the IPC System V message queues (MQ), supporting small message sizes which can be efficiently exchanged with the server. MQs also support testing whether a message can be dequeued for non-blocking communication.

During its setup, the centralized resource manager creates a message queue with a unique identifier. This makes the message queue singleton-like for all invasive applications which use the same identifier to communicate to the RM. Both the client and the server applications can then en- and dequeue messages using this message-queue identifier. Each message further has to be marked with a token specifying the receiver. We then mark the RM with identifier 0 and each invasive application with the application’s system-wide unique process id.

8.2.3 Invasive Computing API

The following invasive interfaces are then offered to the application developer: