Performance impact
CUDA kernel analysis
Enabling the CUPTI sampling will impact the target program in the following ways:
A short amount of time will be spent post-processing at the end of each kernel. This will depend on the length of the kernel and the CUPTI sampling frequency.
Kernels will be serialized. Each CUDA kernel invocation will not return until the kernel has finished and CUPTI post-processing has been performed. Without CUDA kernel analysis mode kernel invocation calls return immediately to allow CUDA processing to be performed in the background.
Increased memory usage whilst in a CUDA kernel. This may manifest as fluctuations between two memory usage values, depending on whether a sample was taken during a CUDA kernel or not.
Taken together the above may have a significant impact on the target
program, potentially resulting in orders of magnitude slowdown. To
combat this profile and analyze CUDA code kernels (with
--cuda-kernel-analysis) and non-CUDA code (no
--cuda-kernel-analysis) in separate profiling sessions.
The NVIDIA GPU metrics will be adversely affected by this overhead,
particularly the GPU utilization metric. See Accelerator.
CUDA memory transfer analysis
Enabling the CUDA memory transfer analysis feature will impact the target program in the following ways:
Time overhead will be incurred at every CUDA memory transfer call. The impact of this will depend on the frequency of such calls. This overhead, if significant, will be shown by Linaro MAP as Profiler callsite tracing overhead.
Minor memory overhead dependent on the number of unique stack traces that lead to CUDA memory transfer calls. This is unlikely to be noticeable unless the number of unique callsites is very large.
This overhead will primarily impact the host (CPU). GPU kernel performance should be unaffected unless the host overhead delays one or more memory transfers that a GPU kernel needs in order to progress.
Overhead mitigation
When profiling CUDA code it may be useful to only profile a short subsection of the program so time is not wasted waiting for CUDA kernels you do not intend to examine. See Profiling only part of a program in Profile a program for instructions.