[rcore] GetCurrentMonitor(), fix for integer overflow in distance calculation

[Vasilis-Narain]

Problem

GetCurrentMonitor() in rcore_desktop_glfw.c computes int dist = dx*dx + dy*dy where dx, dy are pixel offsets from window centre to monitor corner. On my setup (multi-monitor WSL2 through Windows 11) the squared sum exceeds INT_MAX, causing signed integer overflow UB. Today GCC wraps silently, but UBSan-instrumented builds (eg. Zig or clang with -fsanitize=undefined) fail.

Environment

• OS: Ubuntu 24.04.1 on WSL2 (Windows 11 host), X11 via WSLg
• Compiler: gcc 13.3.0
• Display 1920x1200
• Detected on: Zig 0.16 bundling raylib 6.0

Repro

Originally caught by Zig 0.16's UBSan trapping on the bundled raylib build:

thread panic: signed integer overflow: 1173062500 + 1050343281 cannot be represented in type 'int'
rcore_desktop_glfw.c:887

Reproduced standalone by adding a printf next to the overflow site and running examples/core/core_basic_window:

REPRO i=0 dx=-34250 dy=-32484 prod=2228272756 INT_MAX=2147483647 overflow=1
REPRO i=1 dx=-32330 dy=-32484 prod=2100439156 INT_MAX=2147483647 overflow=0

prod (computed as (long long)dx*dx + (long long)dy*dy) exceeds INT_MAX, confirming the truncation.

Fix

Promote dist and closestDist to long long. No behavior change for monitors within INT_MAX distance and correctly handles cases beyond it without UB.

[raysan5]

@Vasilis-Narain I would prefer to avoid the usage of long long, raylib minimizes its usage. Could that code be refactored in some other way to avoid long long usage?

[Vasilis-Narain]

@raysan5 i tried to think of various convoluted ways to check for overflow to explicitly wrap since wrapping in this case is acceptable behaviour...
As it turns out unsigned int cleanly handles overflow by wrapping, but without it being UB. Latest commit tested on gcc with ubsan flags and Zig 0.16 compiler.

Edit: Thinking some more this could theoretically still choose the wrong monitor if ux*ux + uy*uy > UINT_MAX but at least it won't be UB. Not sure if a real fix is worth it as this already covers a much wider range of dx and dy values.

[raysan5]

@Vasilis-Narain thanks for the further review, looks good enough to me!