You can attach to a running GPU program, and then debug the GPU threads.
For details how to attach to a running job, see Attach to running programs, and select the appropriate GPU runtime on the Attach window.