6.11. MPI statistic information¶
In job script, by specifying the following table MCA parameter, it can display the statistic information concerning to MPI communication. About MPI statistic information detail, please see “MPI User’s Guide”-“6.15 MPI Statistical Information”.
-mca mpi_print_stats
-mca mpi_print_stats_ranks
[mpi_print_stats]
Area |
Value |
Specification contents |
---|---|---|
None |
0 |
Specifies not to output MPI statistics. If a value other than an integer from 0 to 4 is specified for this parameter, the value 0 is assumed to be specified. The default value of this parameter is 0. |
Whole mode |
1 |
Specifies that MPI statistics are to be output to the standard error output. However, in this case, the MPI statistics of all the parallel processes are summarized, and the parallel processes with rank number 0 belonging to MPI_COMM_WORLD are output. |
Whole mode |
2 |
Specifies that MPI statistics are to be output to the standard error output. However, in this case, MPI statistics for each parallel process are output by each parallel process itself. Which parallel process to output is determined by specifying MCA parameter |
Section mode |
3 |
Same as parameter value 1. However, it is necessary to specify the FJMPI_COLLECTION_PRINT routine to output to the standard error output. Furthermore, the output contents are divided into a header part, a body part including section lines, and a footer part and output. |
Section mode |
4 |
Same as parameter value 2. However, it is necessary to specify the FJMPI_COLLECTION_PRINT routine to output to the standard error output. Furthermore, the output contents are divided into a header part, a body part including section lines, and a footer part and output. |
[mpi_print_stats_ranks]
Value |
Specification contents |
---|---|
Integer value than 0 |
Identify parallel process rank number to output MPI statistic information. This parameter is only available when specify 2 or 4 to MCA parameter |
-1 |
Specify to output all parallel process MPI statistic information. This parameter is only available when specify 2 or 4 to MCA parameter |
6.11.1. Whole mode¶
Gather statistic information of whole of MPI application and output.
It is available by specifying 1
or 2
to MCA option mpi_print_stats
.
[Option specification example]
$ mpiexec -n 16 -mca mpi_print_stats 1 ./a.out
6.11.2. Section mode¶
It can be used by specifying 3
or 4
to MCA option mpi_print_stats
and using the section specification routine described later.
In MPI application, gather the statistic information that the user specified the section area and output.
[Option specification example]
$ mpiexec -n 16 -mca mpi_print_stats 3 ./a.out
6.11.2.1. Setion specification routine¶
The following table shows a list of specific routines for the section-specific MPI statistical information interface. When using the section mode, it is necessary to specify the measurement section (FJMPI_COLLECTION_START / FJMPI_COLLECTION_STOP) and output (FJMPI_COLLECTION_PRINT) in the source code.
To use this interface, it is required to specify 3 or 4 to MCA option mpi_print_stats
.
Routine name
Argument
Function overview
Sync process
FJMPI_COLLECTION_START
None
Start gathering of section specification MPI statistic information
-
FJMPI_COLLECTION_STOP
None
Stop gathering of section specification MPI statistic information
-
FJMPI_COLLECTION_PRINT
String type
Output gathering data of section specification MPI statistic information
*1
FJMPI_COLLECTION_CLEAR
None
Initialize gathering data of section specification MPI statistic information
-
Note
*1 : Only when mpi_print_stats is 3, sync process is included in FJMPI_COLLECTION_PRINT.
For details of the section specification routine, refer to “The MPI Statistical Information Section Specifying Routine” in the manual “MPI User’s Guide”.
[Section specification MPI routine use image]
1MPI_Init(); 2for(i = 0; i < N; i++){ 3 FJMPI_Collection_start(); 4 //(Process 1) 5 FJMPI_Collection_stop(); 6} 7FJMPI_Collection_print("main1"); // Tirak value of process 1 in for-sentence is output 8 9FJMPI_Collection_clear(); // Initialixe statistic information 10 11FJMPI_Collection_start(); 12//(Process 2) 13FJMPI_Collection_stop(); 14 15FJMPI_Collection_print("main2"); // Statistic information in line 12 process 2 16 17MPI_Finalize();
[If setting specification of section mode to the nest]
1 FJMPI_Collection_start(); 2 //(Process 1) 3 FJMPI_Collection_start(); // Note *1 4 //(Process 2) 5 FJMPI_Collection_stop(); // Note *2 6 //(Process 3) 7 FJMPI_Collection_stop(); // Note *3 8 FJMPI_Collection_print("s1");Note
*1: Only start time is reset that is used for calculation of time displayed in the output function. Counter value is continued.*2: Stop data gathering.*3: Nothing proceeded.
6.11.2.2. Section mode application examples¶
Attention
When executing a program using the procedure using PMPI shown here, the profiler and runtime information output function cannot be used. When using the profiler and runtime information output function, do not use the init_finalize_hook program shown here.
- Prepara source program for section specificationPrepare “
init_finalize_hookf.f
“ if executing MPI_Init/MPI_Finalize from Fortran or “init_finalize_hookc.c
“ if executing from C/C++.Compile depending on using language[init_finalize_hookf.f]
SUBROUTINE MPI_INIT( ierr ) INCLUDE "mpif.h" INTEGER ierr CALL PMPI_INIT( ierr ) CALL FJMPI_Collection_start() RETURN END SUBROUTINE MPI_FINALIZE( ierr ) INCLUDE "mpif.h" INTEGER ierr CHARACTER*(2) STR STR="" CALL FJMPI_Collection_stop() CALL FJMPI_Collection_print(STR) CALL PMPI_FINALIZE( ierr ) RETURN END[init_finalize_hookc.c]
#include <mpi.h> #include <mpi-ext.h> #include <stdio.h> int MPI_Init(int *argc, char ***argv) { int rc; rc = PMPI_Init(argc, argv); FJMPI_Collection_start(); return rc; } int MPI_Finalize() { int rc; FJMPI_Collection_stop(); FJMPI_Collection_print(""); rc = PMPI_Finalize(); return rc; }[Compile execution]
$ mpifrtpx -G -KPIC init_finalize_hookf.f -o libhookf.so $ mpifrtpx -c -KPIC init_finalize_hookf.f $ mpifccpx -G -KPIC init_finalize_hookc.c -o libhookc.so $ mpifccpx -c -KPIC init_finalize_hookc.c
- Execution by dynamic linkingIf it is difficult to re-link the execution module, execute it using dynamic linking. In this method, the MPI latency performance is lower than in the case of the static link described later.
To execute by dynamic link, when execution, only specify
libhookf.so
(libhookc.so
) toLD_PRELOAD
byLD_PRELOAD
. As shown in the following dynamic link execution example (job script), execute by adding the specification ofLD_PRELOAD
tolibhookf.so
(If C/C++,libhookc.so
) .#!/bin/bash -x # #PJM -L "node=8" #PJM -L "rscgrp=small" #PJM -L "elapse=01:00:00" #PJM -g groupname #PJM -x PJM_LLIO_GFSCACHE=/vol000N #PJM -s # mpiexec -x LD_PRELOAD=./libhookf.so -mca mpi_print_stats 3 ./a.out
- Execution by static linkThis is the way to proceed execution module re-linking.Recreate the execution module by linking
init_finalize_hookf.o
(orinit_finalize_hookc.o
). With created execution module, gather MPI statistic information.How to re-link is indicated as below. Link by addinginit_finalize_hookf.o
(orinit_finalize_hookc.o
) when linking.Execute the execution module that is created on re-linking.
$ mpifrtpx -o a.out init_finalize_hookf.o # Program object $ mpifccpx -o a.out init_finalize_hookc.o # Program object[Execution example by static link (job script)]
#!/bin/bash -x # #PJM -L "node=8" #PJM -L "rscgrp=small" #PJM -L "elapse=01:00:00" #PJM -g groupname #PJM -x PJM_LLIO_GFSCACHE=/vol000N #PJM -s # mpiexec -mca mpi_print_stats 3 ./a.out
6.11.2.3. Output contents of section mode¶
On section mode, if FJMPI_COLLECTION_PRINT is not specified, the statistic information is not output. As specified FJMPI_COLLECTION_PRINT, the statistic information is output.
For details on each output item, see “Contents of MPI statistical information output for section specifying output mode” in the “MPI User’s Guide”.
Note
The statictic information of Hasty_Rendezvous is displayed when “Use Hasty Rendezvous communication (--mca pml_ob1_use_hasty_rendezvous 1
) “ is specified.
6.11.2.4. Notes on using section mode¶
If FJMPI_COLLECTION_PRINT is not specified, the statistic information is not output.
If specified
3
tompi_print_stats
, sync process is proceeded in FJMPI_COLLECTION_PRINT.Do not specify measurement section specification (FJMPI_COLLECTION_START/ FJMPI_COLLECTION_STOP) to the nest. If so, the result is not guaranteed.
Until specifying FJMPI_COLLECTION_CLEAR, the statistical information of the measurement section is accumulated. When setting multiple measurement sections, keep in mind which section is accumulated.
The argument (character string) of FJMPI_COLLECTION_PRINT is output to the section heading when output. It has no effect other than being output to the headline. The character string must be a printable character within 30 characters.