Once a model is developed for ROSS, the performance of simulations becomes hugely important. One factor that can play a role is how the LPs are mapped to the physical hardware underneath. LPs that communicate frequently may benefit from being placed within the same MPI process, while LPs that communicate infrequently may be placed on different MPI processes.

Mapping Basics

Before we can jump into the details of how mapping is done, there are few terms that need to be defined:

  • Logical Process or LP: These are the essential objects within discrete-event simulation. They represent the objects within a model and contain most of the simulation state. Each LP has a global ID or GID which uniquely defines it within the whole simulation. Additionally, LP’s have a local ID which identify it locally in the PE’s LP array.
  • Kernel Process or KP: These structures represent groups of LPs. They allow for some ROSS optimizations under the covers, particularly for optimistic simulations and rollbacks. Considering LP-to-KP mapping may be useful for more advanced performance tuning.
  • Physical Process or PE: These are analogous to MPI processes. In the current release of ROSS, there is only 1 PE per MPI rank.

There are also several important global variables and configuration parameters that users should know about:

  • The command line option --nlp=n defines the number of LPs per PE. This option should only be used with models that allow for variable numbers of LPs.
  • Within the application, users can directly define the number of LPs per PE as the first argument to tw_define_lps.
  • The number of PEs is defined by the number of MPI tasks launched.
  • The command line option --nkp=n defines the number of KPs per PE (the default is 16 or the number of LPs per PE, whichever is lower).
  • The tw_nnodes() function defines the number of PEs (MPI ranks) within the current simulation.
  • The g_tw_mynode variable defines which MPI rank the code is being run on. This is equivalent to identifying which PE is executing.

Provided Mappings

ROSS provides 2 built-in mappings: LINEAR and ROUND_ROBIN. To use either of these mappings, users should set the g_tw_mapping variable within the main function, before calling tw_define_lps. The implementation of these mapping functions can be found in ROSS/core/tw-setup.c.

Linear Mapping

The linear mapping is the most basic. It assigns each LP, in order, to a PE.

For example, if a simulation has 5 LPs per PE and 2 PEs, the mapping would look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

| LP  | PE or    |
| GID | MPI Rank |
|-----+----------|
|  0  |          |
|  1  |          |
|  2  |     0    |
|  3  |          |
|  4  |          |
|-----+----------|
|  5  |          |
|  6  |          |
|  7  |     1    |
|  8  |          |
|  9  |          |
|-----+----------|

Round Robin Mapping

The round robin mapping describes the round-robin fashion in which LPs are mapped to PEs. LP 0 maps to the PE, LP 1 maps to PE 1, and so on.

Using the example of 5 LPs per PE and 2 PEs, the mapping would look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

| LP  | PE or    |
| GID | MPI Rank |
|-----+----------|
|  0  |          |
|  2  |          |
|  4  |     0    |
|  6  |          |
|  8  |          |
|-----+----------|
|  1  |          |
|  3  |          |
|  5  |     1    |
|  7  |          |
|  9  |          |
|-----+----------|

Custom Mappings

ROSS also supports the CUSTOM mappings for models that want a more fine-grained control on the LP to PE mapping. This may be particularly useful for models with a fixed number of LPs that do not always map directly. When creating a custom mapping, users must define both an LP-to-PE mapping and LP-to-KP mapping.

There are three mapping functions users must define. Their function signatures are:

1
2
3

tw_peid custom_mapping_lp_to_pe(tw_lpid gid);
tw_lp * custom_mapping_lpgid_to_local(tw_lpid gid);
void custom_mapping_setup(void);

We explore the implementation of these functions through a working example.

Working Example

To describe the various mapping functions, we create an example setup. Suppose our model has exactly 7 LPs and we want to ensure that they are always evenly distributed over the PEs (that is the difference between the PE with the most LPs and the least LPs is 1). This mapping is very similar to the round-robin mapping.

We must ensure that our mapping functions work for various numbers of PEs, from 1 to 7. Here are some example layouts, note that both LP GID and local ID are specified:

1
2
3
4
5
6
7
8
9

| LP GID |   | 1  PE | Local ID |   | 2 PEs | Local ID |   | 3 PEs | Local ID |
|--------+---+-------+----------+---+-------+----------+---+-------+----------|
|      0 |   |     0 |        0 |   |     0 |     0    |   |     0 |        0 |
|      1 |   |     0 |        1 |   |     1 |     0    |   |     1 |        0 |
|      2 |   |     0 |        2 |   |     0 |     1    |   |     2 |        0 |
|      3 |   |     0 |        3 |   |     1 |     1    |   |     0 |        1 |
|      4 |   |     0 |        4 |   |     0 |     2    |   |     1 |        1 |
|      5 |   |     0 |        5 |   |     1 |     2    |   |     2 |        1 |
|      6 |   |     0 |        6 |   |     0 |     3    |   |     0 |        2 |

LP to PE Mapping Function

Given an LP’s GID, this function returns the PE on which that LP resides. This is used during event routing to identify which PE (MPI rank) an event should be sent to.

For our example, we can use a modulus function:

1
2
3

tw_peid custom_mapping_lp_to_pe(tw_lpid gid) {
    return gid % tw_nnodes();
}

Global LP ID to Local Index Function

This function takes an LP’s Global ID and maps it to a local index on the PE. It is used during event processing when selecting which local LP should process an event.

For our example, we first calculate the local index using integer division. We then return a pointer to LP from the ROSS LP array.

1
2
3
4

tw_lp * custom_mapping_lpgid_to_local(tw_lpid gid) {
    tw_lpid local_index = gid / tw_nnodes();
    return g_tw_lp[local_index];
}

Mapping Setup Function

The mapping setup function can be quite complex. It sets up LPs and KPs, using the following functions:

1
2
3

tw_kp_onpe(tw_kpid id, tw_pe * pe);
tw_lp_onpe(tw_lpid index, tw_pe * pe, tw_lpid gid);
tw_lp_onkp(tw_lp * lp, tw_kp * kp);

The tw_kp_onpe function defines a KP within the PE. The tw_lp_onpe function defines a LP within the PE, specifying both the local index and GID. The tw_lp_onkp function defines which KP an LP should map to (note that this functions deals with LP and KP pointers, rather than indexes or IDs).

For our example, we will assume that there is only ever 1 KP per PE.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

void custom_mapping_setup(void) {
    int lp_index;
    int total_lps = 7;

    // set up the single KP
    tw_kp_onpe(0, g_tw_pe);

    // figure out how many LPs are on this PE
    int min_num_lps_per_pe = floor(total_lps/tw_nnodes());
    int pes_with_extra_lp = total_lps - (min_num_lps_per_pe * tw_nnodes());
    int lps_on_pe = min_num_lps_per_pe;
    if (g_tw_mynode < pes_with_extra_lp) {
        lps_on_pe += 1;
    }

    // set up the LPs
    for (lp_index = 0; lp_idex < lps_on_pe; lp_index++) {
        // map LP to KP
        tw_lp_onkp(g_tw_lp[lp_index], g_tw_kp[0]);

        // calculate LP's GID
        tw_lpid lp_gid = g_tw_mynode + (lp_index * tw_nnodes());

        // map LP to PE
        tw_lp_onpe(lp_index, g_tw_pe, lp_gid);
    }
}

Adding a Custom Mapping to ROSS

To set a custom mapping, users must include the LP-to-PE mapping function in the LP-Type array and set some ROSS variables.

The set up of the example custom mapping is demonstrated in the following code snippet:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

tw_lptype lp_types[] = {
    {   (init_f) lp_init_func,
        (pre_run_f) NULL,
        (event_f) lp_fwd_event_func,
        (revent_f) lp_reverse_event_func,
        (commit_f) NULL,
        (final_f) lp_final_func,
        (map_f) custom_mapping_lp_to_pe,  // Mapping function goes here!
        sizeof(lp_state)  },
    { 0 },
};

int main (int argc, char** argv) {
    //...

    g_tw_mapping = CUSTOM;
    g_tw_custom_initial_mapping = &custom_mapping_setup;
    g_tw_custom_lp_global_to_local_map = &custom_mapping_lpgid_to_local;

    g_tw_lp_types = lp_types;

    //...
}

More Resources

Custom mappings can be essential for more complex models, particularly for models with a fixed number of LPs. These projects implement their own custom mappings: