Add mktestdocs to catch docs/code drift by cianc · Pull Request #1096 · mlco2/codecarbon

cianc · 2026-03-03T21:47:42Z

Description

Adds automated testing of Python code examples in the documentation to prevent examples from drifting out of sync with the library.

Add mktestdocs and pytest to the doc dependency group in pyproject.toml so they are available alongside the other doc-build tools without pulling in the full dev group.
Add tests/check-docs-drift.py: a pytest-based script that uses mktestdocs.grab_code_blocks() to collect every ```python fenced block under docs/, skips any block whose first line is `# skip`, and executes the rest via `exec_python()`. A new taskipy task `docs-check-drift` runs it with `pytest scripts/check-docs-drift.py -v`.
Fix all ```python code blocks across docs/ so they are correctly picked up by mktestdocs. Some are marked "#skip" because it's infeasible to run them in CI.
Set save_to_file=False, log_level="error" via environment variables in check-docs-drift.py to avoid creating CSV files or noisy output during CI runs.
Update .github/workflows/build-docs.yml to run docs-check-drift as a step before the docs build, triggered on changes to docs/**, mkdocs.yml, or scripts/check-docs-drift.py.
Document the drift check and the # skip convention in CONTRIBUTING.md under the "Build Documentation" section.

Related Issue

Please link to the issue this PR resolves: #1083

Motivation and Context

Helps prevent drift of code blocks in documentation

How Has This Been Tested?

Ran all tests including new mktestdocs tests.

Screenshots (if appropriate):

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Go over all the following points, and put an x in all the boxes that apply.

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING.md document.
I have added tests to cover my changes.
All new and existing tests passed.

codecov · 2026-03-03T23:17:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.74%. Comparing base (46ac6dd) to head (987ecdf).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1096   +/-   ##
=======================================
  Coverage   80.74%   80.74%           
=======================================
  Files          41       41           
  Lines        3921     3921           
=======================================
  Hits         3166     3166           
  Misses        755      755

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

davidberenstein1957 · 2026-03-04T14:49:30Z

Hi @cianc,

Thanks a lot for this PR. Do you have time to also move some of the documentation examples to docstring examples in the Python code? We would then be able to load them from here, while keeping a single source of truth directly from the Python code, while also exposing examples in the docstring :)
If not, we can focus on keeping the current approach.

Also, I feel that we might not need a separate script directory, but rather just keep Python file as part of the tests directory.

Anywhow. Thahnks a lot for tackling this :)

docs/getting-started/api.md

davidberenstein1957

I left some small remarks and a comment in the main thread. Happy to hear your thoughts and thanks a lot for the help :)

docs/getting-started/api.md

davidberenstein1957 · 2026-03-04T14:52:15Z

docs/getting-started/examples.md


-``` python
+```python
+# skip


we can actually run this for a small subset of the data, right?

Same comment as below, I'm not sure we want to run something this heavy in CI.

davidberenstein1957 · 2026-03-04T14:52:35Z

docs/getting-started/examples.md


-``` python
+```python
+# skip


same here, we could run this with a small subset of the data, right?

I'm not sure, this is quite a lot of code to run in CI - I updated the comment, what do you think?

docs/getting-started/usage.md

davidberenstein1957 · 2026-03-09T12:36:20Z

@cianc just following up here to see if the extended contribution fits in your time schedule, if not feel free to let us know, and we'll handle the rest :)

cianc · 2026-03-09T22:08:18Z

Yes sorry, work has been quite busy. These suggestions seems reasonable and I hope to get to them soon.

…

On Mon, Mar 9, 2026 at 12:36 PM David Berenstein ***@***.***> wrote: *davidberenstein1957* left a comment (mlco2/codecarbon#1096) <#1096 (comment)> @cianc <https://github.com/cianc> just following up here to see if the extended contribution fits in your time schedule, if not feel free to let us know, and we'll handle the rest :) — Reply to this email directly, view it on GitHub <#1096 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADD7PTBJQDHYNZWPCHSYZD4P23FVAVCNFSM6AAAAACWF6E7M2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMRTGQ2TAOJYGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

davidberenstein1957 · 2026-03-19T09:05:35Z

@cianc no problem, happy to iterate on the solution and wait for a while from our side :)

Adds automated testing of Python code examples in the documentation to prevent examples from drifting out of sync with the library. Changes: - Add `mktestdocs` and `pytest` to the `doc` dependency group in `pyproject.toml` so they are available alongside the other doc-build tools without pulling in the full `dev` group. - Add `scripts/check-docs-drift.py`: a pytest-based script that uses `mktestdocs.grab_code_blocks()` to collect every ```python fenced block under `docs/`, skips any block whose first line is `# skip`, and executes the rest via `exec_python()`. A new taskipy task `docs-check-drift` runs it with `pytest scripts/check-docs-drift.py -v`. - Fix all ```python code blocks across `docs/` so they are correctly picked up by mktestdocs: - Remove the stray space in ``` python fences (changed to ```python) so that mktestdocs can identify them (it matches on the exact string "python" immediately after the backticks). - Add `save_to_file=False, log_level="error"` to `EmissionsTracker` and `OfflineEmissionsTracker` instantiations to avoid creating CSV files or noisy output during CI runs. - Add `# skip` as the first line of blocks that cannot run in CI because they depend on external services or optional heavy dependencies (TensorFlow, Prometheus, Logfire, Google Cloud, Comet ML, live CodeCarbon API). - Correct a `pip install` command that was incorrectly fenced as ```python in `comet.md`; changed to ```console. - Update `.github/workflows/build-docs.yml` to run `docs-check-drift` as a step before the docs build, triggered on changes to `docs/**`, `mkdocs.yml`, or `scripts/check-docs-drift.py`. - Document the drift check and the `# skip` convention in `CONTRIBUTING.md` under the "Build Documentation" section.

…heck-docs-drift.py and the CI workflow triggers on changes to that path. 2. Replaced inline save_to_file=False/log_level="error" arguments with environment variables — CODECARBON_SAVE_TO_FILE=false and CODECARBON_LOG_LEVEL=error are now set at the top of the test script, so the doc examples themselves stay clean and uncluttered. 3. Reduced the number of skipped blocks — the task manager example in usage.md was skippable only because it was missing an import; adding from codecarbon import EmissionsTracker made it runnable. The remaining # skip blocks (TensorFlow examples, Comet integration, Prometheus/Logfire output) are unavoidable due to missing dependencies. Skip comments now explain why each block is excluded.

… set via env vars in test script

…g, and skip pseudocode blocks

cianc · 2026-03-20T16:04:40Z

I think I've addressed all comments except for the one on running tensorflow modeling in CI - let me know what you think.

docs/getting-started/examples.md

docs/introduction/power_estimation.md

docs/logging/output.md

docs/logging/to_logger.md

Clean up the #skip comments for some python blocks Co-authored-by: David Berenstein <david.m.berenstein@gmail.com>

cianc marked this pull request as ready for review March 3, 2026 21:51

cianc requested a review from a team as a code owner March 3, 2026 21:51

davidberenstein1957 reviewed Mar 4, 2026

View reviewed changes

docs/getting-started/api.md Show resolved Hide resolved

davidberenstein1957 reviewed Mar 4, 2026

View reviewed changes

davidberenstein1957 self-assigned this Mar 4, 2026

cianc added 2 commits March 20, 2026 15:29

cianc force-pushed the test/add-mktestdocs branch from 9057de3 to 3c99b8e Compare March 20, 2026 15:30

cianc added 2 commits March 20, 2026 15:35

docs: remove save_to_file and log_level args from api.md example, now…

df88e78

… set via env vars in test script

docs: fix broken python snippets across docs — correct imports, casin…

e28687d

…g, and skip pseudocode blocks

davidberenstein1957 approved these changes Mar 20, 2026

View reviewed changes

Apply suggestions from code review

987ecdf

Clean up the #skip comments for some python blocks Co-authored-by: David Berenstein <david.m.berenstein@gmail.com>

Uh oh!

Conversation

cianc commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

codecov bot commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

davidberenstein1957 commented Mar 4, 2026

Uh oh!

Uh oh!

davidberenstein1957 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

davidberenstein1957 Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

cianc Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

davidberenstein1957 Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

cianc Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidberenstein1957 commented Mar 9, 2026

Uh oh!

cianc commented Mar 9, 2026 via email

Uh oh!

davidberenstein1957 commented Mar 19, 2026

Uh oh!

cianc commented Mar 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cianc commented Mar 3, 2026 •

edited

Loading

codecov bot commented Mar 3, 2026 •

edited

Loading