Update Development Documentation (waltsims#595)

* Update pytest.yml

* Update development_environment.rst

* Reformulate development workflow

* Clarify parallel install

* Update development_environment.rst

* Update development_environment.rst

* Add simple hatch workflow

Author: waltsims

.github/workflows/pytest.yml

@@ -58,16 +58,15 @@ jobs:
           files: |
             ./collectedValues
           outPath: collectedValues.tar.gz
-      - name: upload reference values artifact
+      - name: Upload reference values artifact
         id: artifact-upload-step
-        if: ${{ steps.matlab-refs-cache.outputs.cache-hit != 'true' }}
         uses: actions/upload-artifact@v4
         with:
           name: matlab_reference_test_values
           path: collectedValues.tar.gz
           # overwrite: true
+          
       - name: Output artifact URL
-        if: ${{ steps.matlab-refs-cache.outputs.cache-hit != 'true' }}
         run:  echo 'Artifact URL is ${{ steps.artifact-upload-step.outputs.artifact-url }}'
   test:
     needs: collect_references

docs/development/development_environment.rst

@@ -1,108 +1,190 @@
 Development Environment
 =======================
 
-Currently, this package serves as an interface to the cpp binaries of k-Wave.
-For this reason, binaries are required to run simulations with `k-Wave-python`.
-The binaries are downloaded by k-Wave-python when the package is run for the first time.
+Overview
+--------
+k-Wave-python is a Python interface to the k-Wave C++ binaries, which are required to run simulations. The binaries are automatically downloaded when the package runs for the first time.
 
-To correctly set up your development environment for this repository, clone the repository from github, and install the project dependencies.
+Environment Setup with uv
+-------------------------
 
+1. Install uv
+~~~~~~~~~~~~~
+.. code-block:: bash
+
+    curl -LsSf https://astral.sh/uv/install.sh | sh
+
+2. Clone the Repository
+~~~~~~~~~~~~~~~~~~~~~~~
 .. code-block:: bash
 
     git clone https://github.com/waltsims/k-wave-python
     cd k-wave-python
-    pip install -e '.[dev,test]' 
 
-This installs all the dependencies for development, and testing.
+3. Create and Activate Virtual Environment
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. code-block:: bash
 
-Ensure pre-commit is configured by running the following command:
+    uv venv
+    source .venv/bin/activate  # On Windows: .venv\Scripts\activate
+
+4. Install Dependencies
+~~~~~~~~~~~~~~~~~~~~~~~
+Install development and testing dependencies using uv:
 
+.. code-block:: bash
+
+    uv pip install -e '.[dev,test]'
+
+5. Configure Pre-commit Hooks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. code-block:: bash
 
     pre-commit install
 
-Running Tests
-=======================
-Assuming matlab is installed locally, and `k-wave <https://github.com/ucl-bug/k-wave>` is installed in a parallel directory, testing can be performed using the make file located in the project root.
+Testing
+-------
 
-.. code-block:: bash
+Prerequisites
+~~~~~~~~~~~~~
+- MATLAB installed locally (see `Alternative to MATLAB`_ if you don't have MATLAB)
+- `k-Wave <https://github.com/ucl-bug/k-wave>`_ installed in a parallel directory to k-wave-python
 
-    make test
+**Directory structure for parallel installation:**
+
+.. code-block:: text
 
-This process will first generate reference files in matlab and run the complete python test suite against them.
+    <parent_dir>/
+    ├── k-wave/
+    │   ├── k-Wave/
+    │   ├── LICENSE.txt
+    │   └── README.md
+    └── k-wave-python/
+        ├── kwave/
+        └── ...
 
-To run the tests manually after reference generation, use the following command:    
+
+Running Tests
+~~~~~~~~~~~~~
+
+Full Test Suite
+^^^^^^^^^^^^^^^
+Generate reference files with MATLAB and run the complete Python test suite:
 
 .. code-block:: bash
+    cd k-wave-python/
+    make test
+
+Manual Test Execution
+^^^^^^^^^^^^^^^^^^^^^
+After reference generation:
 
-    pytest 
+.. code-block:: bash
 
-To run the tests with coverage, use the following command:
+    pytest
 
+Test Coverage
+^^^^^^^^^^^^^
 .. code-block:: bash
 
     coverage run
 
-To run all examples, to ensure they still run after changes use the following command:
+Running Examples
+~~~~~~~~~~~~~~~~
 
+Default (GPU-enabled)
+^^^^^^^^^^^^^^^^^^^^^
 .. code-block:: bash
 
     make run-examples
+    # or
+    MPLBACKEND=Agg python run_examples.py
 
-or
-
+Force CPU Execution
+^^^^^^^^^^^^^^^^^^
 .. code-block:: bash
 
-    MPLBACKEND=Agg python run_examples.py
+    MPLBACKEND=Agg KWAVE_FORCE_CPU=1 python run_examples.py
 
+Test Architecture
+-----------------
 
+The test suite compares Python and MATLAB outputs through two methodologies:
 
-If you want to force the examples to run on the cpu:
+1. Unit Testing
+~~~~~~~~~~~~~~~
+- Tests k-Wave-python functions against their MATLAB counterparts
+- Reference outputs stored in ``.mat`` files
+- Generated by MATLAB scripts in ``tests/matlab_test_data_collectors/matlab_collectors/``
+- Master script: ``tests/matlab_test_data_collectors/run_all_collectors.m``
+- Output location: ``tests/matlab_test_data_collectors/python_testers/collectedValues/``
 
-.. code-block:: bash
+.. _`Alternative to MATLAB`:
 
-    MPLBACKEND=Agg KWAVE_FORCE_CPU=1 python run_examples.py
+.. note::
+   **Alternative to MATLAB:** If you don't have a local MATLAB installation, you can download pre-generated reference artifacts from `GitHub CI <https://nightly.link/waltsims/k-wave-python/workflows/pytest/master/matlab_reference_test_values.zip>`_.
 
 
-Test References
-=======================
+.. _`GitHub CI artifacts link`: https://nightly.link/waltsims/k-wave-python/workflows/pytest/master/matlab_reference_test_values.zip
+
+2. Integration Testing
+~~~~~~~~~~~~~~~~~~~~~~
+- Validates ``.h5`` files produced by k-Wave-python against original k-Wave outputs
+- Uses hash values from MATLAB examples stored in JSON files
+- Hash files location: ``tests/reference_outputs/``
+- These files are committed to the repository and only require updates for new k-Wave releases
+
+Generating MATLAB Reference Files
+---------------------------------
+
+Process for Creating Reference Files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Open target MATLAB example (e.g., ``example_pr_2D_TR_directional_sensors.m`` from the `k-Wave repository <https://github.com/ucl-bug/k-wave/blob/main/k-Wave/examples/example_pr_2D_TR_directional_sensors.m>`_)
+
+2. Locate ``kSpaceFirstOrder`` function call:
+
+   .. code-block:: matlab
+   
+       input_args = {'PMLInside', false, 'PMLSize', PML_size, 'PlotPML', false, 'Smooth', false};
+       sensor_data = kspaceFirstOrder2D(kgrid, medium, source, sensor, input_args{:});
+
+3. Add save options to ``input_args``:
+
+   .. code-block:: matlab
+   
+       input_args = {'PMLInside', false, 'PMLSize', PML_size, 'PlotPML', false, 'Smooth', false, 'SaveToDisk', true, 'SaveToDiskExit', true};
+
+4. Run modified example to generate ``.h5`` files in your ``tmp`` folder:
+
+   - Single function call: creates ``example_input.h5``
+   - Multiple calls: creates ``example_input_1.h5``, ``example_input_2.h5``, etc.
+
+5. Convert ``.h5`` files to JSON hashes using ``H5Summary``:
 
-Tests compare the outputs of the python and the matlab interfaces.
-These tests are located in the ``tests`` directory. The comparison between ``matlab`` and ``python`` outputs are done in two ways:
+   - Single file: see `lines 92-95 <https://github.com/waltsims/k-wave-python/blob/1f9df5d987d0b3edb1a8a43fad0885d3d6079029/tests/h5_summary.py#L92-L95>`_
+   - Multiple files: see `lines 97-106 <https://github.com/waltsims/k-wave-python/blob/1f9df5d987d0b3edb1a8a43fad0885d3d6079029/tests/h5_summary.py#L97-L106>`_
+
+Publishing k-wave-python
+-----------------
+
+`Hatch <https://hatch.pypa.io/latest/>`_ is used to publish k-wave-python to `PyPI <https://pypi.org/>`_. 
 
-- **Unit testing**: k-Wave-python functions that have a direct counterpart in original k-Wave are tested by comparing the outputs of the two functions.
-  The output of the original k-Wave functions are stored in ``.mat`` files.
-  These files can be generated by running the corresponding MATLAB scripts located in the ``tests/matlab_test_data_collectors/matlab_collectors/`` directory by running ``tests/matlab_test_data_collectors/run_all_collectors.m``.
-  After running the scripts, the reference files can be found in ``tests/matlab_test_data_collectors/python_testes/collectedValues/``.
- 
 .. note::
-    If you do not have MATLAB installed to generate the reference files, you can download recently generated reference file outputs from the GitHub CI and place them in the ``python_testers/collectedValues/`` directory.
-    The latest reference files can be found in the artifacts of the latest CI run of ``pytest.yml`` (e.g. `here <https://github.com/waltsims/k-wave-python/actions/runs/7770639710/artifacts/1217868112>`_).
-
-- **Integration testing**: k-Wave-python tests output .h5 files that are passed to the k-Wave binaries and ensures that they match the output of the original k-Wave.
-  This testing compares the output for many of the example scripts from the original k-Wave package.
-  Hash values of the reference output ``.h5`` file from MATLAB examples are generated and stored in ``.json`` files in ``tests/reference_outputs/``.
-  These ``.json`` files are stored in the code repository and do not need to be regenerated.
-  Since these files are generated from the original k-Wave package, they only need to be updated when a new release of k-Wave is made.
-
-**Matlab reference file generation** can be described in the following steps.
-
-#. Open desired example in matlab, e.g. `example_pr_2D_TR_directional_sensors.m <https://github.com/ucl-bug/k-wave/blob/main/k-Wave/examples/example_pr_2D_TR_directional_sensors.m>`_
-#. Find the lines where the call to one of the `kSpaceFirstOrder-family` function is made. For example,
-
-    .. code-block:: matlab
-    
-        input_args = {'PMLInside', false, 'PMLSize', PML_size, 'PlotPML', false, 'Smooth', false};
-        sensor_data = kspaceFirstOrder2D(kgrid, medium, source, sensor, input_args{:});
-
-#. Update the ``input_args`` field by adding two new options - ``{'SaveToDisk', true, 'SaveToDiskExit': true}``. These options will ensure that we a ``.h5`` file will be created and saved in your ``tmp`` folder, while avoiding to run the actual simulation.
-#. Run the modified example. You will find created files in your ``tmp`` folder. Usually exact file name depends on how many calls are made to the `kSpaceFirstOrder-family` function in the example:
-    * If there is only a single call, created file name will be ``example_input.h5``
-    * If there are two or more calls, created files will have names like ``example_input_1.h5``, ``example_input_2.h5``, ``example_input_3.h5`` and so on
-#. Now it is time to turn the ``.h5`` files to the hashed ``.json`` files. This can be done with the ``H5Summary``.
-    * If you have a single ``.h5`` file, adapt the lines below and run the script:
-        https://github.com/waltsims/k-wave-python/blob/1f9df5d987d0b3edb1a8a43fad0885d3d6079029/tests/h5_summary.py#L92-L95
-    * For multiple files, adapt the lines below:
-        https://github.com/waltsims/k-wave-python/blob/1f9df5d987d0b3edb1a8a43fad0885d3d6079029/tests/h5_summary.py#L97-L106
+    This is only performed by developers with write access to the k-wave-python package on PiPI.
+
+The package can be built using:
+
+.. code-block:: bash
+
+    hatch build
+
+And pushed to the production index with:
+
+.. code-block:: bash
+
+    hatch publish -u __token__
+
+