Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 8 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# PyDDA (Pythonic Direct Data Assimilation)
![alt text](https://github.com/openradar/PyDDA/blob/pydda_devel/pydda%20logo.png "Logo Title Text 1")
![alt text](https://github.com/openradar/PyDDA/blob/master/pydda%20logo.png "Logo Title Text 1")

[<img src="https://anaconda.org/conda-forge/pydda/badges/version.svg">](https://anaconda.org/conda-forge/pydda)
[<img src="https://anaconda.org/conda-forge/pydda/badges/downloads.svg">](https://anaconda.org/conda-forge/pydda/files)
[<img src="https://img.shields.io/travis/openradar/PyDDA.svg">](https://travis-ci.org/openradar/PyDDA)
[<img src="https://github.com/openradar/PyDDA/actions/workflows/python-package-conda.yml/badge.svg">](https://github.com/openradar/PyDDA/actions/workflows/python-package-conda.yml)
[<img src="https://img.shields.io/badge/docs-here-green">](https://openradarscience.org/PyDDA)

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3942686.svg)](https://doi.org/10.5281/zenodo.3942686)
Expand All @@ -19,7 +19,7 @@ This package is a rewrite of the Potvin et al. (2012) and Shapiro et al (2009) w
This new package also uses a faster minimization technique, L-BFGS-B, which provides a factor of 2 to 5 speedup versus
using the predecessor code, [NASA-Multidop](https://github.com/nasa/MultiDop), as well as a more elegant syntax
as well as support for an arbitrary number of radars. The code is also threadsafe and has been tested using
HPC tools such as Dask on large (100+ core) clusers.
HPC tools such as Dask on large (100+ core) clusters.


The user has an option to adjust strength of data, mass continuity constraints as well as implement a low pass filter.
Expand All @@ -28,22 +28,21 @@ field from a grid.

Angles.py is from Multidop and was written by Timothy Lang of NASA.

We recommend using Python 3.8+ or better and using anaconda or pip to install
We recommend using Python 3.9+ or better and using anaconda or pip to install
the required dependencies of PyDDA.

In addition, in order to use the capability to load HRRR data as a constraint, the [cfgrib](https://github.com/ecmwf/cfgrib) package is needed. Since this does not work on Windows, this is an optional depdenency for those who wish to use HRRR data. To install cfgrib, simply do:
In addition, in order to use the capability to load HRRR data as a constraint, the [cfgrib](https://github.com/ecmwf/cfgrib) package is needed. Since this does not work on Windows, this is an optional dependency for those who wish to use HRRR data. To install cfgrib, simply do:

pip install cfgrib

Finally, PyDDA now supports Jax and TensorFlow as optional dependencies. PyDDA can be configured to use these two packages to perform the cost function minimization. We highly encourage users to take advantage of these two new engines, as they offer advantages such as better calculation of gradients and faster convergence. In addition, GPUs and TPUs are supported by these packages that can help drastically accelerate the calculations. For Tensorflow type:

conda install -c conda-forge tensorflow=2.6 tensorflow-probability
conda install -c conda-forge tensorflow tensorflow-probability

For Jax, type:

conda install -c conda-forge jax

=======
## Links to important documentation

1. [Examples](http://openradarscience.org/PyDDA/source/auto_examples/plot_examples.html)
Expand All @@ -70,23 +69,19 @@ order to install from source, in a bash shell or the Anaconda prompt if you are
```
git clone https://github.com/openradar/PyDDA
cd PyDDA
python setup.py install
pip install -e .
```

=======
## Acknowledgments
Core components of the software are adopted from the [Multidop](https://github.com/nasa/MultiDop) package by converting the C code to Python.

The development of this software is supported by the Climate Model Development and Validation (CMDV) activity which is funded by the Office of Biological and Environmental Research in the US Department of Energy Office of Science.

=======
## Contributing

We have a set of goals that we wish to accomplish using PyDDA, including the assimilation of data from various models in the retrieval,
improved visualizations, use of radar data in antenna coordinates, and improved documentation. For more details on what contributions
would be useful to acheiving these goals, see the [PyDDA Roadmap](https://github.com/openradar/PyDDA/blob/master/ROADMAP.md).

=======
would be useful to achieving these goals, see the [PyDDA Roadmap](https://github.com/openradar/PyDDA/blob/master/ROADMAP.md).

## Further support

Expand All @@ -95,8 +90,6 @@ relegated to the [openradar Discourse group](https://openradar.discourse.group)
enables the entire open radar science community to answer questions related to PyDDA so that both the maintainer
and users can answer questions people may have.

=======

## References
You must cite these papers if you use PyDDA:

Expand Down
6 changes: 3 additions & 3 deletions ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# PyDDA Roadmap

In this document we show the details of the PyDDA roadmap. The roadmap shows the next steps for PyDDA and what help is needed for
contributions so that we can acheive our goal of providing a quality Direct Data Assmilation framework for the meterorological community
contributions so that we can achieve our goal of providing a quality Direct Data Assimilation framework for the meteorological community
to use. Right now, PyDDA currently supports retrievals from an arbitrary number of Doppler radars and can integrate in data from rawinsondes,
HRRR and WRF model runs. We would like improve how PyDDA assimilates data into a retrieval. Furthermore, we would like to make PyDDA
more accessible to the international meteorology community. Our current goals in this regard are:

* Support for a greater number of high resolution (LES) models such as CM1
* Support for integrating in data from the Rapid Refresh
* Coarser resolution reanalyses such as the NCEP reanalysis and ERA Interim would also provide useful information.
* Coarser resolution reanalyses such as the NCEP reanalysis and ERA5 would also provide useful information.
Due to PyDDA's capability in customizing the weight each dataset has in the retrieval, using a weak constraint against coarse
resolution reanalyses would provide a useful background.
* Support for individual point analyses, such as those from wind profilers and METARs
Expand All @@ -21,5 +21,5 @@ more accessible to the international meteorology community. Our current goals in
Contributions can be made from people of all skill levels in Python, even none! If there is anyone who would like to make a contribution
to any of these features, we would be more than welcoming of any additions. All we ask is that you follow the
[Code of Conduct](https://github.com/openradar/PyDDA/blob/master/CODE_OF_CONDUCT.md) and that your contributions are in accordance with
the [Contibutor's Guide](https://openradarscience.org/PyDDA/contributors_guide/index.html), complete with documentation and unit
the [Contributor's Guide](https://openradarscience.org/PyDDA/contributors_guide/index.html), complete with documentation and unit
tests where applicable.
19 changes: 9 additions & 10 deletions doc/source/contributors_guide/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ There are just some things that we ask of you. One is that your code be able
to be distributed under the BSD 3-clause license, which is available in LICENSE
in the main directory.

One, we ask, that when on the GitHub forum or making contributions to PyDDA
that all developers and users follow the PyDDA code of conduct.
We ask that all developers and users follow the PyDDA code of conduct when
participating in the GitHub forum or making contributions.


Contributor Covenant Code of Conduct
Expand Down Expand Up @@ -94,8 +94,8 @@ by other members of the project's leadership.

**Attribution**

This Code of Conduct is adapted from the Contributor Covenant, version 1.4,
available at `<http://contributor-covenant.org/version/1/4>`_
This Code of Conduct is adapted from the Contributor Covenant, version 2.1,
available at `<https://www.contributor-covenant.org/version/2/1/code_of_conduct/>`_

Code Style
----------
Expand All @@ -121,7 +121,7 @@ Python File Setup
-----------------

In a new .py file, the top of the code should have the function, sphinx comments
and the public and private functions within the .py file. Public fuunctions are
and the public and private functions within the .py file. Public functions are
listed first and then private functions and classes. Private functions should
have an underscore in front of the name. A space is needed between the last
function and the closing docstring quotation marks.
Expand All @@ -134,7 +134,7 @@ standards, modules should be added in the following order:
3. Local application imports

Following the main function def line, but before the code within it, a docstring is
needed to explain all arguments, retuns, references, and other information. Please
needed to explain all arguments, returns, references, and other information. Please
follow the NumPy documentation style:

`<https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt>`_
Expand Down Expand Up @@ -169,7 +169,7 @@ For an example format of the documentation, see this:
Returns a 3D float array containing the v component of the wind field.
The shape will be the same shape as the fields in Grid.
w: 3D float array
Returns a 3D float array containing the u component of the wind field.
Returns a 3D float array containing the w component of the wind field.
The shape will be the same shape as the fields in Grid.
"""

Expand All @@ -179,13 +179,13 @@ For an example format of the documentation, see this:
Testing
-------

When adding a new function to pyart it is important to add it to the __init__.py
When adding a new function to PyDDA it is important to add it to the __init__.py
under the corresponding folder.

Create a test function and use assert to test the calculated values against known
values. For an example, see:

`<https://github.com/openradar/PyDDA/tests/test_retrieval.py>`_
`<https://github.com/openradar/PyDDA/blob/master/pydda/tests/test_retrieval.py>`_

Pytest will run this test whenever a pull request is made to the master branch
of the openradar/PyDDA repository. This will then allow the maintainers to
Expand Down Expand Up @@ -235,7 +235,6 @@ forking the repository on GitHub, create your own branch by doing:
::

git checkout -b this_branch
git branch this_branch

::

Expand Down
17 changes: 14 additions & 3 deletions doc/source/dev_reference/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,9 @@ Visualization module for PyDDA.
:undoc-members:
:show-inheritance:

===========================
:mod:`initalization` Module
===========================
============================
:mod:`initialization` Module
============================

The module for creating custom initial states for the PyDDA retrieval.

Expand All @@ -64,3 +64,14 @@ The module for creating custom constraints (i.e. from models, satellites) for th
:members:
:undoc-members:
:show-inheritance:

================
:mod:`io` Module
================

Input/output utilities for reading and writing PyDDA Grids.

.. automodule:: pydda.io
:members:
:undoc-members:
:show-inheritance:
16 changes: 8 additions & 8 deletions doc/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ description of each of PyDDA's subroutines.

**User Guide**

The cookbook provides in-depth information on how
The user guide provides in-depth information on how
to use PyDDA, including how to get started.
This is where to look for general conceptual descriptions on how
to use parts of PyDDA, including how to make your first wind retrieval and
Expand Down Expand Up @@ -73,7 +73,7 @@ System Requirements
=========================

This works on any modern version of Linux, Mac OS X, and Windows. For Windows,
HRRR data integration is not supported. In addition, since PyDDA takes advtange
HRRR data integration is not supported. In addition, since PyDDA takes advantage
of parallelism we recommend:
::
An Intel machine with at least 4 cores
Expand Down Expand Up @@ -134,11 +134,11 @@ just type in the following commands assuming you have the above dependencies ins

git clone https://github.com/openradar/PyDDA
cd PyDDA
python setup.py install
pip install -e .
::

Finally, PyDDA now supports using `Jax <jax.readthedocs.io>`_ and `TensorFlow <tensorflow.org>`_
for solving the three dimensional wind field. PyDDA requries TensorFlow 2.6 and the
Finally, PyDDA now supports using `Jax <https://jax.readthedocs.io>`_ and `TensorFlow <https://tensorflow.org>`_
for solving the three dimensional wind field. PyDDA requires TensorFlow 2.x and the
tensorflow-probability package for TensorFlow to be enabled.
In addition, both of these packages can utilize CUDA-enabled GPUs for much faster processing. These two
dependencies are optional as the user can still use PyDDA with the SciPy ecosystem.
Expand All @@ -148,8 +148,8 @@ The Jax optimizer uses the same optimizer as SciPy's (`L-BFGS-B <https://docs.sc
Known issues
============

The TensorFlow engine uses the unbounded version of this optimizer which removes the constraint that the
the wind magnitudes must be less than 100 m/s. The removal of this constraint can sometimes
The TensorFlow engine uses the unbounded version of this optimizer which removes the constraint that
wind magnitudes must be less than 100 m/s. The removal of this constraint can sometimes
result in numerical instability, so it is recommended that the user test out both Jax and TensorFlow
if they desire GPU-accelerated retrievals.

Expand All @@ -168,7 +168,7 @@ Further support


We are now requesting that all questions related to PyDDA that are not potential software issues to be
relegated to the `openradar Discourse group <openradar.discourse.group>` with a 'pydda' tag on it. This
relegated to the `openradar Discourse group <https://openradar.discourse.group>`_ with a 'pydda' tag on it. This
enables the entire open radar science community to answer questions related to PyDDA so that both the maintainer
and users can answer questions people may have.

Expand Down
18 changes: 9 additions & 9 deletions doc/source/user_guide/dealiasing_velocities.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,14 @@ Radar Data Quality Control - Dealiasing

In this notebook, we will showcase how to perform quality control of your
radar files, specifically dealiasing velocities. By doing this we can provide
PyDDA with quality controlled doppler velocities for dual doppler analysis.
PyDDA with quality controlled Doppler velocities for dual Doppler analysis.

-------------
Read the Data
-------------

For this example, we use test data for two NEXRAD radars in
northern Oklahoma. For more information on
For this example, we use test data for two NEXRAD (WSR-88D) radars in
northern Oklahoma: KTLX (Twin Lakes, OK) and KICT (Wichita, KS). For more information on
reading the radar data, consult :ref:`reading-radar-data`.

Get test data::
Expand Down Expand Up @@ -164,7 +164,7 @@ Velocity Texture Displays

Let's see what this velocity texture looks like.
Additionally, a histogram of velocity texture values will allow for
the determination of a threshold to distingiush the hydrometeor signal
the determination of a threshold to distinguish the hydrometeor signal
from artifacts.

.. code-block:: python
Expand Down Expand Up @@ -260,7 +260,7 @@ Filter Doppler Velocity Artifacts
+++++++++++++++++++++++++++++++++

Now that we have determined which velocity texture values correspond to
artifacts within the doppler velocity data, we utilize Py-ART's GateFilter
artifacts within the Doppler velocity data, we utilize Py-ART's GateFilter
to filter out these artifacts

Py-ART's GateFilter function::
Expand All @@ -279,7 +279,7 @@ Apply Dealiasing
----------------

Now that we have removed artifacts, we can proceed with dealiasing
the doppler velocity data with Py-ART's Region-Based Dealiasing
the Doppler velocity data with Py-ART's Region-Based Dealiasing
Algorithm.

The Region-Based Dealiasing finds regions of similar velocities and
Expand Down Expand Up @@ -449,9 +449,9 @@ Summary
-------

Utilizing Py-ART, we read in two radar files within close proximity to each other.
We then corrected the radar doppler velocities to remove artifacts and clutter.
Finally, utilizing Py-ART, we applied a region-based dealiasing alogrithm to
unfold the doppler velocities.
We then corrected the radar Doppler velocities to remove artifacts and clutter.
Finally, utilizing Py-ART, we applied a region-based dealiasing algorithm to
unfold the Doppler velocities.

Now that we have corrected velocities, incorporating these radars into PyDDA
will be shown in the next notebook.
4 changes: 2 additions & 2 deletions doc/source/user_guide/gridding.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Converting the radar data to Cartesian coordinates with Py-ART
PyDDA expects radar data to be in Cartesian coordinates before it retrieves
the wind fields. Most radar data, however, is in the radar's native antenna
coordinates. Therefore, the radar data needs to be converted to Cartesian
coordinates. Py-ART's mapping toolbox contains the necessary utilties
coordinates. Py-ART's mapping toolbox contains the necessary utilities

We will assume that you have followed the steps outlined in :ref:`reading-radar-data`
for reading the radar data in its native coordinates. PyDDA requires dealiased velocities
Expand Down Expand Up @@ -46,7 +46,7 @@ The :code:`grid_limits` is a 3-tuple of 2-tuples specifying the :math:`z`, :math
limits of the grid in meters. The :code:`grid_shape` specifies the shape of the grid in number of
points. We then use PyART's `grid_from_radars <https://arm-doe.github.io/pyart/API/generated/pyart.map.grid_from_radars.html>`_
function to create the grids :code:`grid_ktlx` and :code:`grid_kict`. PyDDA requires that both grids have the same
shape and origin, so we explictly set those in the options for both grids.
shape and origin, so we explicitly set those in the options for both grids.

.. code-block:: python

Expand Down
6 changes: 3 additions & 3 deletions doc/source/user_guide/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@
User Guide
##########################

This is a place to include our user guide.
This user guide covers the full multi-Doppler wind retrieval workflow in PyDDA,
from reading and quality-controlling raw radar data through to retrieving and
visualizing the 3D wind field.

.. toctree::
:maxdepth: 3
Expand All @@ -14,6 +16,4 @@ This is a place to include our user guide.
retrieving_winds.rst
optimizing_wind_retrieval.rst
visualizing_winds.rst
retrieving_winds.rst
optimizing_wind_retrieval.rst
nesting_wind_retrieval.rst
Loading
Loading