Skip to content

Playground: fix referenceImage fields, re-enable 7 tests#1719

Merged
bkaradzic-microsoft merged 1 commit into
BabylonJS:masterfrom
bkaradzic-microsoft:import-bjs-reference-images-1593
May 28, 2026
Merged

Playground: fix referenceImage fields, re-enable 7 tests#1719
bkaradzic-microsoft merged 1 commit into
BabylonJS:masterfrom
bkaradzic-microsoft:import-bjs-reference-images-1593

Conversation

@bkaradzic-microsoft
Copy link
Copy Markdown
Contributor

@bkaradzic-microsoft bkaradzic-microsoft commented May 28, 2026

Summary

Update 35 entries' referenceImage field in Apps/Playground/Scripts/config.json to the canonical filenames on disk (hyphen-slugified names, full .png extension, exact case). Most were field-only mismatches against PNGs that were already present in Apps/Playground/ReferenceImages/; the others mapped to PNGs that exist on disk under a different spelling. The renderer does an exact-string XHR GET for the file, so any field mismatch short-circuits the test at the MISSING_REFERENCE_IMAGE: pre-flight check in validation_native.js.

What's in this PR

1. Apps/Playground/Scripts/config.json

  • 35 entries: referenceImage updated to the canonical PNG name.
  • 7 passing entries: excludeFromAutomaticTesting + stale reason dropped (same pattern as Re-enable 3 stale-excluded Playground tests (345, 346, 533). #1693).
  • 28 failing entries: reason rewritten with a self-contained technical description (instead of misleading text like "No matching reference image found" or "Requires WebGL2-only feature").

2. Three PNGs refreshed

Asset-Container-Instantiate-to-Scene.png, Asset-Container-Instantiate-to-Scene-2.png, wgsl-in-shadermaterial.png were previously committed RGB-encoded; replaced with their upstream Babylon.js RGBA-encoded counterparts. BN renders RGBA, so 3-byte/px RGB references throw Reference data length ... must match render data length in compare().

The other 32 PNGs already in the repo matched byte-for-byte; only the referenceImage field needed to change.

Verification

Each re-enabled test was run by default (no --include-excluded) on Win32 V8 D3D11 Debug:

idx title px diff
115 OpenPBR Base Weight 1254
116 OpenPBR Base Diffuse Roughness Analytical Lights 1346
360 Asset Container Instantiate to Scene 810
361 Asset Container Instantiate to Scene 2 270
713 Depth Renderer and RTT - debug markers 974
717 Unbalanced push pop - debug markers 178
718 Pending pop - debug markers 121

All sub-threshold under default threshold=25, errorRatio=2.5; max 1346 of 240000 pixels. No custom margin needed.

Each of the 28 failing entries was confirmed to skip by default (skipped=1) and reach the documented JS/native error under --include-excluded.

Failing-test buckets

reason count indices
Cubemap load not implemented on Babylon Native 2 114, 122
CDF renderer / IBL prefiltering fails with 'FLOAT' undefined 6 117-120, 482, 483
Pixel diff: ~99% of pixels differ vs reference 1 121
Loading remote https:// assets is not supported 1 123
Chakra parse failure on modern JS syntax 14 445, 493-502, 709, 712, 714
FrameGraph render pipeline hangs past 120s 2 450, 715
NativeEngine missing createStorageBuffer for WGSL ShaderMaterial 1 708
ReflectionProbe: 'TEXTURE_CUBE_MAP' undefined on NativeEngine 1 716

These categories are individually tracked outside this PR; each reason string maps 1:1 to a known issue.

Copilot AI review requested due to automatic review settings May 28, 2026 16:25
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes Playground visual-test catalog entries so they point to existing canonical reference PNGs, while re-enabling the subset that now validates and keeping known failures excluded with clearer tracking reasons.

Changes:

  • Updates 35 referenceImage values in Playground config.json.
  • Re-enables 7 tests by removing stale automatic-test exclusions.
  • Refreshes 3 reference PNGs to RGBA-encoded versions expected by the native validation path.

Reviewed changes

Copilot reviewed 1 out of 4 changed files in this pull request and generated no comments.

File Description
Apps/Playground/Scripts/config.json Corrects reference image names and updates exclusion reasons for affected Playground tests.
Apps/Playground/ReferenceImages/Asset-Container-Instantiate-to-Scene.png Refreshes reference PNG encoding for validation compatibility.
Apps/Playground/ReferenceImages/Asset-Container-Instantiate-to-Scene-2.png Refreshes reference PNG encoding for validation compatibility.
Apps/Playground/ReferenceImages/wgsl-in-shadermaterial.png Refreshes reference PNG encoding for validation compatibility.

Update 35 entries' `referenceImage` field in
`Apps/Playground/Scripts/config.json` to the canonical filenames on disk
(hyphen-slugified names, full `.png` extension, exact case). Most were
field-only mismatches against PNGs that were already present in
`Apps/Playground/ReferenceImages/`; the others mapped to PNGs that exist
on disk under a different spelling. The renderer does an exact-string XHR
GET for the file, so any field mismatch short-circuits the test at the
`MISSING_REFERENCE_IMAGE:` pre-flight check in `validation_native.js`.

Re-enable 7 tests that pass with the corrected references (drop
`excludeFromAutomaticTesting` + stale `reason`):

  idx 115 -- OpenPBR Base Weight                              pixelDiff=1254
  idx 116 -- OpenPBR Base Diffuse Roughness Analytical Lights pixelDiff=1346
  idx 360 -- Asset Container Instantiate to Scene             pixelDiff= 810
  idx 361 -- Asset Container Instantiate to Scene 2           pixelDiff= 270
  idx 713 -- Depth Renderer and RTT - debug markers           pixelDiff= 974
  idx 717 -- Unbalanced push pop - debug markers              pixelDiff= 178
  idx 718 -- Pending pop - debug markers                      pixelDiff= 121

(all sub-threshold under default threshold=25, errorRatio=2.5; same pattern
as PR BabylonJS#1693)

Rewrite `reason` for the remaining 28 tests with self-contained
technical descriptions matching observed failures (no more misleading
"No matching reference image found" or "Requires WebGL2-only feature"):

  Cubemap load not implemented on Babylon Native (2)
  CDF renderer / IBL prefiltering fails with 'FLOAT' undefined (6)
  Pixel diff: ~99% of pixels differ vs reference (1)
  Loading remote https:// assets is not supported (1)
  Chakra parse failure on modern JS syntax (14)
  FrameGraph render pipeline hangs past 120s (2)
  NativeEngine missing createStorageBuffer for WGSL ShaderMaterial (1)
  ReflectionProbe: 'TEXTURE_CUBE_MAP' undefined on NativeEngine (1)

Refresh 3 stale PNGs (`Asset-Container-Instantiate-to-Scene.png`,
`Asset-Container-Instantiate-to-Scene-2.png`, `wgsl-in-shadermaterial.png`)
with the upstream Babylon.js RGBA-encoded versions. BN renders RGBA, so
3-byte/px RGB references throw `Reference data length ... must match
render data length` in `compare()`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bkaradzic-microsoft bkaradzic-microsoft force-pushed the import-bjs-reference-images-1593 branch from a30b873 to e40aadb Compare May 28, 2026 16:35
@bkaradzic-microsoft bkaradzic-microsoft changed the title Playground: fix referenceImage fields for 35 tests (#1593) Playground: fix referenceImage fields, re-enable 7 tests May 28, 2026
@bkaradzic-microsoft bkaradzic-microsoft merged commit 9b95140 into BabylonJS:master May 28, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants