docs(examples): add uncertainty calibration plot example#18
Open
dimasd-angga wants to merge 2 commits into
Open
Conversation
Give new contributors a quick diagnostic for the synthetic world model's quality uncertainty. The script trains with small defaults so it runs fast, then saves a scatter of predicted std versus absolute error to make miscalibration visually obvious.
Owner
|
Thanks for the contribution — this is exactly the kind of beginner-friendly diagnostic example I was hoping for. Before I merge, could you please reformat the Python file with standard line breaks and indentation? The raw file appears to have many statements collapsed onto the same lines, so it may not run correctly as Python. Could you also confirm that this command works from the industrial-world-model directory? python examples/plot_uncertainty_calibration.py Once the formatting is fixed and the script runs, I’d be happy to review again. |
Contributors had no entry point for the new `plot_uncertainty_calibration.py` script, so it was easy to miss when exploring the benchmark. Listing it alongside the existing demo and explaining how to read the resulting scatter plot makes the calibration check discoverable and interpretable without reading the source.
Author
|
Pushed an update addressing the review feedback. Commit: docs(industrial-world-model): document uncertainty calibration example |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a small example script that produces an uncertainty calibration plot for the synthetic industrial world model, addressing the request in #17 for a beginner-friendly diagnostic. The script trains the demo
ManufacturingRSSMwith reduced defaults, runs the held-out test set through it, and saves a scatter of predicted quality uncertainty vs. absolute prediction error.Changes
benchmarks/toy-process-window/industrial-world-model/examples/plot_uncertainty_calibration.py: generates a dataset viagenerate_world_model_dataset, splits it withsplit_dataset, buildsManufacturingRSSM, trains for 10 epochs withtrain_world_model, then iterates the testManufacturingSequenceDatasetto collectstate_mean,state_std, and targetnext_stateon theQUALITY_INDEXdimension.uncertainty_calibration.pngnext to itself using matplotlib.Testing
Manual run only:
cd benchmarks/toy-process-window/industrial-world-model && python examples/plot_uncertainty_calibration.py. The script trains the demo model, prints the summary, and writesuncertainty_calibration.pngin theexamples/directory. No automated tests were added — the issue asks for an example/plot, not a code path under test.Notes
run_demo.pyto keep first-run time low; bump them for a smoother plot.main()since it is only needed at plotting time; it is already a transitive dependency viaimwm'svisualization.py.imwm/were modified.Closes #17