Skip to content

Conversation

@reoLantern
Copy link

Problem

On newer CUDA drivers, cuLaunchKernel can receive a CUkernel handle instead of a CUfunction.
In that case cuFuncGetAttribute() returns CUDA_ERROR_INVALID_HANDLE, so tracer headers
show garbage values (shmem/nregs/binary version).

Solution

Add a minimal fallback: when cuFuncGetAttribute() returns INVALID_HANDLE, query the
attribute via cuKernelGetAttribute() using the same handle and current device.

Changes

  • Add helper get_attr_with_kernel_fallback(...) in tracer_tool.cu
  • Use it for NUM_REGS, SHARED_SIZE_BYTES, and BINARY_VERSION

Testing

  • Rebuilt tracer_tool and generated traces; header fields now report valid values.

Copilot AI review requested due to automatic review settings December 29, 2025 18:12
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a compatibility issue with newer CUDA drivers where cuLaunchKernel may receive CUkernel handles instead of CUfunction handles, causing cuFuncGetAttribute() to fail with CUDA_ERROR_INVALID_HANDLE and resulting in garbage values in trace headers.

Key Changes:

  • Introduces a helper function get_attr_with_kernel_fallback() that attempts to query attributes via cuFuncGetAttribute() first, then falls back to cuKernelGetAttribute() if an INVALID_HANDLE error occurs
  • Replaces direct cuFuncGetAttribute() calls with the new helper for NUM_REGS, SHARED_SIZE_BYTES, and BINARY_VERSION attributes
  • Removes explicit error checking via CUDA_SAFECALL macros in favor of silent error handling with fallback

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +41 to +46
if (res == CUDA_ERROR_INVALID_HANDLE) {
CUdevice dev = 0;
if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
}
}
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The return value of cuKernelGetAttribute is not checked. If this call fails, value might remain uninitialized or contain incorrect data, which would then be returned and used in trace headers. The original code used CUDA_SAFECALL which would have caught such errors. Consider checking the return value and handling failures appropriately, or at least logging when the fallback also fails.

Suggested change
if (res == CUDA_ERROR_INVALID_HANDLE) {
CUdevice dev = 0;
if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
}
}
if (res == CUDA_SUCCESS) {
return value;
}
if (res == CUDA_ERROR_INVALID_HANDLE) {
CUdevice dev = 0;
CUresult dev_res = cuCtxGetDevice(&dev);
if (dev_res == CUDA_SUCCESS) {
CUresult kern_res = cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
if (kern_res != CUDA_SUCCESS) {
fprintf(stderr,
"tracer_tool: cuKernelGetAttribute failed (attr=%d) with error %d\n",
(int)attr, (int)kern_res);
}
} else {
fprintf(stderr,
"tracer_tool: cuCtxGetDevice failed with error %d while getting attribute %d\n",
(int)dev_res, (int)attr);
}
} else {
fprintf(stderr,
"tracer_tool: cuFuncGetAttribute failed (attr=%d) with error %d\n",
(int)attr, (int)res);
}
/* On any failure, return the default value (0) */

Copilot uses AI. Check for mistakes.
if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
}
}
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function should document its behavior when both cuFuncGetAttribute and cuKernelGetAttribute fail. Currently, it silently returns 0 (the initialized value) without any indication that the attribute query failed. This differs from the original implementation which used CUDA_SAFECALL to enforce error handling. Consider adding a comment explaining that 0 is returned on failure, or adding optional error logging.

Suggested change
}
}
/* If both attribute queries fail, this function returns 0 (the default
* initialized value) without reporting an error. Callers should treat 0
* as a possible "attribute unavailable" result. */

Copilot uses AI. Check for mistakes.
Comment on lines +43 to +44
if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
Copy link

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If cuCtxGetDevice fails, the function skips the cuKernelGetAttribute call and returns 0. However, it's unclear whether this is the intended behavior. Consider checking if cuCtxGetDevice can legitimately fail in scenarios where cuFuncGetAttribute returns INVALID_HANDLE, and whether silently returning 0 is appropriate in such cases.

Suggested change
if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
CUresult ctx_res = cuCtxGetDevice(&dev);
if (ctx_res == CUDA_SUCCESS) {
cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
} else {
fprintf(stderr,
"get_attr_with_kernel_fallback: cuCtxGetDevice failed with "
"error code %d\n",
(int)ctx_res);

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant