refactor(dx8): Replace unsafe Matrix4x4/D3DMATRIX casts with proper transpose conversion

[bobtista]

• Closes Properly implement transpose between Matrix4 and D3DMatrix #552

Summary

Replaces all C-style casts between Matrix4x4/Matrix3D and D3DXMATRIX/D3DMATRIX with proper conversion functions that handle the row-major/column-major transpose.

Before:

DX8CALL(SetTransform(transform, (D3DMATRIX*)&m));

After:

D3DXMATRIX d3dMat = Build_D3DXMATRIX(m);
DX8CALL(SetTransform(transform, &d3dMat));

[xezon]

Goes into the right direction but it still looks like there are quirks.

Did you test the code paths? Is the math still correct?

[xezon]

Was this inverse meant to be a transpose or does it really want to do the inverse here on a D3DXMATRIX that is in DX coordinates?

[bobtista]

I'm unsure - the old code cast a column-major Matrix4x4 to D3DXMATRIX* (row-major) without conversion, then inverted.
For orthogonal view matrices, inverse equals transpose.
In the Thyme commit, they still use D3DXMatrixInverse after removing the casts, so maybe inverse is intended? I'm going to test both and see what's right.

[bobtista]

I just played a skirmish with this build, and it seems normal to me. I imagine it would be very obvious if matrix math was inverted. I suppose I could run two clients and set a breakpoint and see if any values differ? What would you recommend?

[bobtista]

Ok update: I spent today testing and debugging this. I think it's ready now.

On main:

• Matrix4x4 and D3DMATRIX have the same memory layout (both row-major in memory).
• The unsafe cast is a direct memory copy with no transpose, which worked because both types share the same memory layout.

Main branch: Matrix4x4 M → direct copy → D3DMATRIX (same bytes) → direct copy → Matrix4x4 M
Fix branch: Matrix4x4 M → transpose → D3DMATRIX M^T → transpose → Matrix4x4 M

In today's testing I noticed that the ship cannon fire animations were glitching in the shell map.
The issue was that, while projection is directly sent to directx8 (transposed) and then we get it from directx8 (transpoesd), render_state.world and render_state.view are cached as Matrix4x4, only transposed when we send them to directx8 - and we can get them from the cache as Matrix4x4 too. The bug was that I was transposing world and view, but they're already in the right format. Once I removed the extra transpose, the cannon fire animations were back to normal.

I added debug lines and verified that the math now comes out the same on main and the branch (in the previous commit, the math was sometimes off because of the extra transpose).

[bobtista]

Also, this branch uses D3DXMatrixInverse() in W3DWater.cpp (lines 245, 1605, and 3001). This seems to be consistent with both main and the Thyme change. I tested changing them to D3DXMatrixTranspose() during debugging and water rendering still looked correct, but I haven't committed those changes. Google says For orthogonal matrices (rotation only), inverse = transpose, so if the view matrix is orthogonal at those points, transpose would be correct and more efficient. It's correct as it is, and matches the original, so I left it this way.

[bobtista]

Ok I found that I needed to update a bunch of other callers with my approach from before, and it was making the diff bigger. So, instead of storing Matrix4x4 in RenderStateStruct and converting at every use site, we now store D3DXMATRIX directly - this is more like how it was in the Thyme version too. Tested again, it looks good.

[xezon]

Yes storing D3DMATRIX in RenderStateStruct makes sense.

I reimplemented the change with #2052 from scratch because I was not entirely happy with the implementation details of this change.