-
Notifications
You must be signed in to change notification settings - Fork 181
Fix kernel attribute query when cuLaunchKernel uses CUkernel handles #521
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR addresses a compatibility issue with newer CUDA drivers where cuLaunchKernel may receive CUkernel handles instead of CUfunction handles, causing cuFuncGetAttribute() to fail with CUDA_ERROR_INVALID_HANDLE and resulting in garbage values in trace headers.
Key Changes:
- Introduces a helper function
get_attr_with_kernel_fallback()that attempts to query attributes viacuFuncGetAttribute()first, then falls back tocuKernelGetAttribute()if an INVALID_HANDLE error occurs - Replaces direct
cuFuncGetAttribute()calls with the new helper for NUM_REGS, SHARED_SIZE_BYTES, and BINARY_VERSION attributes - Removes explicit error checking via
CUDA_SAFECALLmacros in favor of silent error handling with fallback
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (res == CUDA_ERROR_INVALID_HANDLE) { | ||
| CUdevice dev = 0; | ||
| if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) { | ||
| cuKernelGetAttribute(&value, attr, (CUkernel)func, dev); | ||
| } | ||
| } |
Copilot
AI
Dec 29, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The return value of cuKernelGetAttribute is not checked. If this call fails, value might remain uninitialized or contain incorrect data, which would then be returned and used in trace headers. The original code used CUDA_SAFECALL which would have caught such errors. Consider checking the return value and handling failures appropriately, or at least logging when the fallback also fails.
| if (res == CUDA_ERROR_INVALID_HANDLE) { | |
| CUdevice dev = 0; | |
| if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) { | |
| cuKernelGetAttribute(&value, attr, (CUkernel)func, dev); | |
| } | |
| } | |
| if (res == CUDA_SUCCESS) { | |
| return value; | |
| } | |
| if (res == CUDA_ERROR_INVALID_HANDLE) { | |
| CUdevice dev = 0; | |
| CUresult dev_res = cuCtxGetDevice(&dev); | |
| if (dev_res == CUDA_SUCCESS) { | |
| CUresult kern_res = cuKernelGetAttribute(&value, attr, (CUkernel)func, dev); | |
| if (kern_res != CUDA_SUCCESS) { | |
| fprintf(stderr, | |
| "tracer_tool: cuKernelGetAttribute failed (attr=%d) with error %d\n", | |
| (int)attr, (int)kern_res); | |
| } | |
| } else { | |
| fprintf(stderr, | |
| "tracer_tool: cuCtxGetDevice failed with error %d while getting attribute %d\n", | |
| (int)dev_res, (int)attr); | |
| } | |
| } else { | |
| fprintf(stderr, | |
| "tracer_tool: cuFuncGetAttribute failed (attr=%d) with error %d\n", | |
| (int)attr, (int)res); | |
| } | |
| /* On any failure, return the default value (0) */ |
| if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) { | ||
| cuKernelGetAttribute(&value, attr, (CUkernel)func, dev); | ||
| } | ||
| } |
Copilot
AI
Dec 29, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function should document its behavior when both cuFuncGetAttribute and cuKernelGetAttribute fail. Currently, it silently returns 0 (the initialized value) without any indication that the attribute query failed. This differs from the original implementation which used CUDA_SAFECALL to enforce error handling. Consider adding a comment explaining that 0 is returned on failure, or adding optional error logging.
| } | |
| } | |
| /* If both attribute queries fail, this function returns 0 (the default | |
| * initialized value) without reporting an error. Callers should treat 0 | |
| * as a possible "attribute unavailable" result. */ |
| if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) { | ||
| cuKernelGetAttribute(&value, attr, (CUkernel)func, dev); |
Copilot
AI
Dec 29, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If cuCtxGetDevice fails, the function skips the cuKernelGetAttribute call and returns 0. However, it's unclear whether this is the intended behavior. Consider checking if cuCtxGetDevice can legitimately fail in scenarios where cuFuncGetAttribute returns INVALID_HANDLE, and whether silently returning 0 is appropriate in such cases.
| if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) { | |
| cuKernelGetAttribute(&value, attr, (CUkernel)func, dev); | |
| CUresult ctx_res = cuCtxGetDevice(&dev); | |
| if (ctx_res == CUDA_SUCCESS) { | |
| cuKernelGetAttribute(&value, attr, (CUkernel)func, dev); | |
| } else { | |
| fprintf(stderr, | |
| "get_attr_with_kernel_fallback: cuCtxGetDevice failed with " | |
| "error code %d\n", | |
| (int)ctx_res); |
Problem
On newer CUDA drivers,
cuLaunchKernelcan receive aCUkernelhandle instead of aCUfunction.In that case
cuFuncGetAttribute()returns CUDA_ERROR_INVALID_HANDLE, so tracer headersshow garbage values (shmem/nregs/binary version).
Solution
Add a minimal fallback: when
cuFuncGetAttribute()returns INVALID_HANDLE, query theattribute via
cuKernelGetAttribute()using the same handle and current device.Changes
get_attr_with_kernel_fallback(...)intracer_tool.cuTesting
tracer_tooland generated traces; header fields now report valid values.