Thread affinity advisor

Linaro MAP collects CPU affinity binding information of threads that are sampled and presents it in the Thread Affinity Advisor dialog. To open this dialog, select the View ‣ Thread Affinity Advisor menu option or click the Thread Affinity Advisor button in the toolbar.

Note

The Thread Affinity Advisor dialog is not available for .map files that were generated without the Thread Affinity Advisor feature included in the license.

../../_images/map_thread_affinity_advisor.png — Fig. 1 Thread affinity advisor dialog

The Thread Affinity Advisor dialog provides affinity binding information in two main parts: settings that affect affinity bindings, and snapshots of the resultant affinity bindings recorded at different points in the application’s runtime. Sections can be resized as needed.

Affinity binding snapshots

The affinity bindings snapshots are taken at the following points:

Library load: Before execution enters the main method, typically before OpenMP thread pools are initialized.
MPI_Init or MPI_Init_thread: In the MPI_Init or MPI_Init_thread call initializing MPI communication. For non-MPI applications this will take place at the start of the application’s main method.
MPI_Finalize: In the MPI_Finalize call. For non-MPI programs this will take place when the application is about to shut down. OpenMP threads typically still exist at this point.

Toggle between these using the selection slider.

../../_images/map_thread_affinity_advisor_snapshot_slider.png — Fig. 2 Thread affinity snapshot slider

Affinity binding variables

The MPI launch and process command lines are presented at the top of the Thread Affinity Advisor dialog. Select an individual process to display its command line.

Global environment variables taken when the launch line was executed, and process-specific environment variables set by the job scheduler. Global environment variables are separated between OpenMP variables, submissions script variables (e.g. variables set as a consequence of #SBATCH lines in a job submission script), and all other launch-time variables of relevance.

../../_images/map_thread_affinity_advisor_env_vars.png — Fig. 3 Thread affinity relevant environment variables

Node topology viewer

The center of the Thread Affinity Advisor dialog contains the node topology viewer, which is an abstract depiction of the main hardware components of the selected exemplar node.

Note

CPUs reserved by the system or omitted via Linux cgroups cannot be displayed. This can be the case when a system is configured with “core-specialization” or a “low-noise” mode.

Note

The hardware displayed in the node topology viewer may not be accurate in a virtualized environment.

The node topology viewer can be panned by clicking and dragging with the mouse, or zoomed in/out with the mouse-wheel. Buttons to zoom in/out or reset the zoom level are available in the top right when the mouse cursor is in the viewer.

CPUs are displayed by their logical index in the view.

Note

These indexes are an abstraction, and may not match the physical or OS indexes as used by tools such as numactl or taskset.

Match the CPU item’s background color and pattern in the processes table to find the MPI process that was bound to that CPU. A solid red background indicates that threads from different processes are bound to that CPU (i.e. there are overlapping affinity masks), while red text is used to indicate that multiple threads from a single process are bound.

Select a CPU item to highlight the processes and threads that are bound to it in their respective tables.

Selecting any item displays additional details underneath the node topology viewer if available.

../../_images/map_thread_affinity_advisor_node_viewer.png — Fig. 4 Thread affinity node topology viewer

Exemplar nodes

The Thread Affinity Advisor dialog condenses the amount of information displayed when conveying the affinity binding information of the sampled program. Therefore, affinity bindings that are similar across different nodes are incorporated into a single exemplar node, where the lowest ranked MPI process was run. Nodes that are similar to the exemplar have:

similar hardware
same number of MPI processes
same number and type of threads in those processes
same affinity bindings of those threads

These nodes are listed under Exemplar nodes. Hover over exemplar nodes to show tooltips which highlight the hardware differences between them.

Processes and threads

Processes that were run on the selected exemplar node are displayed on the right, along with their corresponding MPI rank. Select a process to display its threads.

Threads are categorized by their type and activity. Low-activity “monitor” or “helper” threads from known MPIs, OpenMP or accelerator runtimes are listed with gray text and unchecked by default. These threads do not contribute to the background colors and patterns of CPU items in the node topology viewer. This behavior can be overridden by checking the box next to the listed thread.

Select a thread to highlight the CPUs in the node topology viewer that it was bound to.

../../_images/map_thread_affinity_advisor_processes_threads.png — Fig. 5 Highlighted CPU in tread affinity node topology viewer

Commentary

Commentary is presented at the bottom of the Thread Affinity Advisor dialog, which provides brief summaries of the main affinity binding issues detected in the sampled application. These are prefixed by their severity level.