v0.6.0

What's Changed

This release brings some exciting features to pytask! 🥳

Lockfiles

From now on, pytask uses a lockfile named pytask.lock to store the state of tasks, dependencies and products. It is placed at the root of your project directory and should be added to version control. It enables two features

• You can move your project to another machine and continue developing without rebuilding from scratch.
• Use pytask lock to update the lockfile after moving or renaming tasks or integrating externally created products

For backward compatibility, pytask continues to update the state in the old .pytask/pytask.sqlite3 so you can safely downgrade to an older pytask version if necessary (🤞 hopefully not).

Logs

pytask is now able to capture logging and write your logs to files. Use it to capture lots of information about your running tasks while keeping your terminal clean.

Others

• networkx is now an optional dependency enabling faster startup.
• The documentation was migrated to Zensical and has been improved in many ways, for example, a new overview for the CLI.

Full List of Changes

• Remove pytest markers by @tobiasraabe in #741
• chore: update install docs and typing setup by @tobiasraabe in #742
• Move runtimes to json journal store by @tobiasraabe in #766
• Normalize programmatic build_dag config defaults by @tobiasraabe in #767
• Guard against nested live displays by @tobiasraabe in #769
• Parse database_url strings from config by @tobiasraabe in #768
• Clear annotation_locals after collection by @tobiasraabe in #776
• Remove attrs dependency. by @tobiasraabe in #744
• ENH: Make attributes mandatory on PNode and PProvisionalNode by @tobiasraabe in #787
• Test #312 by unxfailing pdb capture tests in CI by @tobiasraabe in #788
• Add Hamilton to workflow tool comparison by @tobiasraabe in #797
• Implement a portable lockfile. by @tobiasraabe in #743
• Migrate docs to zensical by @tobiasraabe in #789
• Fix collection for remote UPath nodes by @tobiasraabe in #820
• Fix stable node signatures for remote UPath nodes by @tobiasraabe in #822
• Refactor DAG internals and lazy-load networkx by @tobiasraabe in #830
• chore: update pre-commit hooks by @tobiasraabe in #833
• Introduce scheduler protocol and simple scheduler by @tobiasraabe in #831
• Harden GitHub Actions workflows with zizmor by @tobiasraabe in #836
• Skip non-interactive live updates by @tobiasraabe in #837
• Add task log capture and export by @tobiasraabe in #821
• docs: consolidate CLI docs into command reference by @tobiasraabe in #858
• Use Zensical card grid on docs homepage by @tobiasraabe in #859
• Reset marker config during unconfigure by @tobiasraabe in #868
• Add pytask lock command by @tobiasraabe in #860
• Improve lockfile documentation structure by @tobiasraabe in #874
• docs: prepare for 0.6.0, improve landing page and little errors by @tobiasraabe in #875

Full Changelog: v0.5.8...v0.6.0

v0.5.8

What's Changed

The main changes are

• Full support for Python 3.14
• Moving to ty for type checking

Full list

• Fix pickle hash test for Python 3.14 by @tobiasraabe in #725
• Fix test_debugging for Python 3.14. by @tobiasraabe in #726
• Bump the github-actions group across 1 directory with 5 updates by @dependabot[bot] in #730
• Enable automerge for pre-commit and dependabot. by @tobiasraabe in #731
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #727
• Fix importing packages with missing __init__.py by @tobiasraabe in #732
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci[bot] in #733
• Enable merge groups. by @tobiasraabe in #737
• Update the comparison to other tools. by @tobiasraabe in #736
• Move to ty. by @tobiasraabe in #734
• Release capture resources between CLI runs by @tobiasraabe in #739
• Handle lazy annotations for task generators in Python 3.14 by @tobiasraabe in #724
• Add support for Python 3.14 by @tobiasraabe in #710
• Prepare 0.5.8 release by @tobiasraabe in #740

Full Changelog: v0.5.7...v0.5.8

v0.5.7

What's Changed

• Fix TypeError with Click Sentinel in Python 3.14 by @DrorSh in #723
• Clarify documentation on repeated tasks in notebooks. by @tobiasraabe in #721

New Contributors

• @DrorSh made their first contribution in #723

Full Changelog: v0.5.6...v0.5.7

v0.5.6

What's Changed

A Halloween release 👻

• Python 3.9 support is dropped
• Let pytask explain why tasks are executed
• An example of repeated tasks in Jupyter notebooks
• ... and some bug fixes

Here is the complete list

• Use justfile commands in CI. by @tobiasraabe in #695
• Fix --capture tee-sys. by @tobiasraabe in #703
• Explain why tasks will be executed. by @tobiasraabe in #704
• Disable syntax highlighting for platform version information by @tobiasraabe in #706
• Drop support for Python 3.9. by @tobiasraabe in #707
• Update mypy. by @tobiasraabe in #708
• Add uv lock pre-commit check. by @tobiasraabe in #709
• Remove uv as a test dependency. by @erooke in #713
• Properly parse pdbcls from the configuration. by @tobiasraabe in #718
• Fix passing repeated tasks to the functional interface. by @tobiasraabe in #719

Full Changelog: v0.5.5...v0.5.6

v0.5.5

What's Changed

• Don't exit early to detect missing files. by @tobiasraabe in #694
• Add note about using pytask with workspaces. by @tobiasraabe in #692

Full Changelog: v0.5.4...v0.5.5

v0.5.4

What's Changed

• Use uv more. by @tobiasraabe in #680
• Remove differentiation between test kinds. by @tobiasraabe in #683
• Test lowest dependency resolution. by @tobiasraabe in #684
• Ensure compatibility with click v8.2. by @tobiasraabe in #676
• Improve installation instructions and fix terminal animations by @tobiasraabe in #686
• Fix file URLs to paths and tasks and enable colors and icons on Windows. by @tobiasraabe in #685
• Update changelog and publish in the repo root. by @tobiasraabe in #687

Full Changelog: v0.5.3...v0.5.4

v0.5.3

What's Changed

This release was overdue. This is not because it includes a big feature, but because little changes kept piling up and should have been released earlier.

• Add data catalog name to nodes to make the representation unique. by @tobiasraabe in #650
• Ignore the .pixi folder during the collection. by @tobiasraabe in #662
• Update pre-commit and pin click. by @pre-commit-ci in #677
• Add de-/serializer func to PickleNode attributes. by @tobiasraabe in #673
• Enable some tests in CI or offline, remove refurb. by @tobiasraabe in #678

Full Changelog: v0.5.2...v0.5.3

v0.5.2

What's Changed

This release comes with only some minor improvements. Notably,

• pytask adds a .gitignore to .pytask, preventing users from accidentally committing the folder following best practices from venv and pytest.
• Support for Python 3.8 ended, and support for Python 3.13 was added. There might be an issue with uv's standalone Python versions and being stuck in the debugger launched by pytask. If it happens, filing an issue would be appreciated.

Changes

• Add dark mode svgs to index page. by @tobiasraabe in #636
• Stop live execution after exception. by @tobiasraabe in #640
• Add .gitignore to .pytask to avoid committing it. by @tobiasraabe in #646
• Support Python 3.13 and drop 3.8. by @tobiasraabe in #633
• Fix type annotation for hash function. by @tobiasraabe in #655
• Document pipefunc. by @tobiasraabe in #656

Full Changelog: v0.5.1...v0.5.2

v0.5.1

What's Changed

This release contains mostly bug fixes but also the guide for complex task repetitions has been updated and includes some learnings from recent projects.

• [automated] Update plugin list by @github-actions in #615
• Fix interaction between provisional nodes and @mark.persist. by @tobiasraabe in #617
• Ensure that root_dir of DirectoryNode is created. by @tobiasraabe in #618
• tests: remove dependence on root folder name by @erooke in #620
• tests: make coiled an optional import by @erooke in #619
• Fix the pull request template. by @tobiasraabe in #621
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #610
• Bump peter-evans/create-pull-request from 6.0.5 to 6.1.0 in the github-actions group by @dependabot in #623
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #622
• Add warning when non-matching files are passed to pytask. by @tobiasraabe in #627
• Fix duplicated collection of task modules. by @tobiasraabe in #628
• Fix display issues with the programmatic interface. by @tobiasraabe in #631
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #629
• Bump sigstore/gh-action-sigstore-python from 2.1.1 to 3.0.0 in the github-actions group by @dependabot in #630
• Fix issue with rerunning tasks via pytask.build. by @tobiasraabe in #626
• Redesign the scaling tasks guide. by @tobiasraabe in #616
• Follow-up on #616. by @tobiasraabe in #632

New Contributors

• @erooke made their first contribution in #620

Full Changelog: v0.5.0...v0.5.1

v0.5.0

Highlights

✨ pytask v0.5.0 is released and contains two big features. ✨

🤖 Provisional Nodes and Task Generators

pytask now has mechanisms to define tasks that produce an unknown number of products, which are provisional nodes. For example, when a task splits data by rows into chunks of roughly 100MB.

If you want further to process each chunk of data with a separate task, use task generators to create as many tasks as there are chunks.

Everything is explained in the how-to guide Provisional Nodes and Task Generators.

🚀 Support for HPC and your favorite cloud provider (AWS, GCP, Azure) using dask / coiled

pytask can build your project on virtual machines from the cloud provider of your choice. Read more about it in the release announcement of pytask-parallel v0.5.0.

Removals

• The markers @pytask.mark.depends_on and @pytask.mark.produces are removed. Please define dependencies and products using the new approaches.
• Instead of @pytask.mark.task use from pytask import task and @task.

What's Changed

• Fix type hints for Task.execute() and TaskWithoutPath.execute(). by @tobiasraabe in #548
• [automated] Update plugin list by @github-actions in #550
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #549
• Remove depends_on and produces markers. by @tobiasraabe in #551
• Remove @pytask.mark.task and switch to @task. by @tobiasraabe in #552
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #554
• Forbid strings as values for paths in config file. by @tobiasraabe in #553
• Use new-style hook wrappers and require pluggy 1.3. by @tobiasraabe in #555
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #556
• Fix issue with @task(after=...) in notebooks and terminals. by @tobiasraabe in #557
• Fix path in "Node is dir" error message by @egerlach in #561
• Bump the github-actions group with 2 updates by @dependabot in #560
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #559
• Make universal-pathlib 0.2.2 an official dependency. by @tobiasraabe in #566
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #562
• Bump the github-actions group with 1 update by @dependabot in #564
• Fix typos in defining_dependencies_products.md by @ChristianZimpelmann in #563
• Improve handling of task_files. by @tobiasraabe in #568
• Remove hooks related to the DAG. by @tobiasraabe in #569
• Remove redundant calls of PNode.state(). by @tobiasraabe in #571
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #573
• Remove pytask_execute_create_scheduler hook. by @tobiasraabe in #575
• Implement task generators and provisional nodes. by @tobiasraabe in #487
• Fix interaction with --pdb, --trace and tasks that return. by @tobiasraabe in #579
• Simplify the code related to tracebacks. by @tobiasraabe in #581
• [automated] Update plugin list by @github-actions in #584
• Improve typing of capture.py. by @tobiasraabe in #587
• Improve linting. by @tobiasraabe in #586
• Reset class variables of ExecutionReport and Traceback. by @tobiasraabe in #588
• Fix error introduced by #588. by @tobiasraabe in #590
• Invalidate cache to check whether remote files exist. by @tobiasraabe in #591
• Resolve root paths and module names for imported files. by @tobiasraabe in #589
• Use uv to speed up CI. by @tobiasraabe in #567
• Bump the github-actions group with 1 update by @dependabot in #578
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #577
• Recreate PythonNodes every time. by @tobiasraabe in #593
• Publish NodeLoadError. by @tobiasraabe in #594
• Manage project with rye. by @tobiasraabe in #596
• Bump peter-evans/create-pull-request from 6.0.2 to 6.0.3 in the github-actions group by @dependabot in #597
• Replace requests with httpx. by @tobiasraabe in #598
• Stop unwrapping coiled functions. by @tobiasraabe in #595
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #592
• Add fixture for changing cwd. by @tobiasraabe in #599
• Refactor tests using subprocesses. by @tobiasraabe in #600
• Bump peter-evans/create-pull-request from 6.0.3 to 6.0.4 in the github-actions group by @dependabot in #602
• [automated] Update plugin list by @github-actions in #601
• Fix example about capturing warnings in docs. by @tobiasraabe in #603
• Fix PythonNode examples in docs. by @tobiasraabe in #604
• Improve infra and add step for type checking. by @tobiasraabe in #605
• Bump peter-evans/create-pull-request from 6.0.4 to 6.0.5 in the github-actions group by @dependabot in #608
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #607
• Allow tasks to be pending. by @tobiasraabe in #609
• Remove status from pytask_execute_task_log_start. by @tobiasraabe in #611
• Validate the names of data catalogs. by @tobiasraabe in #612
• Improve documentation for data catalogs. by @tobiasraabe in #606
• [automated] Update plugin list by @github-actions in #613

New Contributors

• @egerlach made their first contribution in #561
• @ChristianZimpelmann made their first contribution in #563

Full Changelog: v0.4.5...v0.5.0