Skip to content

fix new ci and notebooks #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
braingram merged 3 commits into from
Aug 6, 2025
Merged

fix new ci and notebooks #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
efc3625
Fix CI update converter notebooks to not write to home directory
braingram Aug 6, 2025
4366eea
update package description in converter notebook
braingram Aug 6, 2025
a2b226e
clean up package description in second converter notebook
braingram Aug 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ on:
branches:
- main
pull_request:
types:
- synchronize
branches:
- main


# Only cancel in-progress jobs or runs for the current workflow
Expand Down
279 changes: 14 additions & 265 deletions notebooks/Your_first_ASDF_converter.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"a Python object equivalent to the one it started out as.\n",
"\n",
"Note: The object to be serialized to ASDF must consist\n",
"of only elements that ADSF knows how to serialize. ASDF does\n",
"of only elements that asdf knows how to serialize. asdf does\n",
"know how to serialize numpy arrays and all standard \n",
"Python primative types (e.g., strings, numbers, booleans,\n",
"lists and dictionaries), as well as objects that have \n",
Expand All @@ -54,18 +54,15 @@
"--------------\n",
"\n",
"For ASDF to be aware of the converters, it is necessary \n",
"to include in the setup.cfg file information for an entry\n",
"to configure an entry\n",
"point for ASDF. Entry points are\n",
"a very useful Python tool for making plug-ins for packages\n",
"easy for users of the plug-in to use, both in the installation\n",
"a useful Python tool for making plug-ins easy for users, both in the installation\n",
"and usage aspect.\n",
"Entry points remove the need for the core package to be continually \n",
"updated with new extension packages that it has to be aware of.\n",
"\n",
"This information is provided in the converter package's\n",
"setup.cfg (it could be in\n",
"setup.py, but the .cfg file is the usual place to put this\n",
"information). What happens when the package is installed is\n",
"configuration. What happens when the package is installed is\n",
"that information about entry points is saved by Python. Python\n",
"provides an API to the core package for it to discover what \n",
"entry points have been designated for that package so that it can\n",
Expand All @@ -80,287 +77,45 @@
"We will name the converter package mfconverter (for My First\n",
"Converter) so this will create a package of that name in\n",
"the current directory. This tutorial will have the bare bones\n",
"files needed for that. The following will generate\n",
"such a barebones directory structure.\n",
"\n",
"Since this tutorial effectively is editing files and needs\n",
"to move between different directories, it is more awkward than\n",
"usual for a notebook\n",
"\n",
"Required Software\n",
"----------------------\n",
"\n",
"- numpy\n",
"- asdf v2.8 or higher\n",
"\n",
"Let's begin\n",
"-------------\n",
"\n",
"This will create a converter_tutorial directory in your home directory.\n",
"If you wish it to be elsewhere, make the appropriate changes to the\n",
"dependent commands, but if there is no strong reason to change it,\n",
"leave it be."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5fec10da",
"metadata": {},
"outputs": [],
"source": [
"mkdir ~/converter_tutorial"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3b1e163f",
"metadata": {},
"outputs": [],
"source": [
"cd ~/converter_tutorial"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "06a42933",
"metadata": {},
"outputs": [],
"source": [
"# Record current directory for restarts\n",
"import os\n",
"curdir = os.getcwd()\n",
"print(curdir)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0ae4a184",
"metadata": {},
"outputs": [],
"source": [
"mkdir mfconverter"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "46d22a46",
"metadata": {},
"outputs": [],
"source": [
"cd mfconverter"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a980c3f9",
"metadata": {},
"outputs": [],
"source": [
"mkdir src"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8dbb76d3",
"metadata": {},
"outputs": [],
"source": [
"mkdir src/mfconverter"
"files needed for that."
]
},
{
"cell_type": "markdown",
"id": "ede348bd",
"metadata": {},
"source": [
"And now we will create a module that has a very, very simple photo ID class and add a package `__init__.py`"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f045f111",
"metadata": {},
"outputs": [],
"source": [
"%%writefile src/mfconverter/photo_id.py\n",
"class PhotoID:\n",
" \"Holds Photo ID information\"\n",
"\n",
" def __init__(self, last_name, first_name, photo):\n",
" \"expects a monochromatic numpy array for photo\"\n",
" self.last_name = last_name\n",
" self.first_name = first_name\n",
" self.photo = photo\n",
" \n",
" def name(self):\n",
" return self.last_name + ', ' + self.first_name"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ad205d2e",
"metadata": {},
"outputs": [],
"source": [
"%%writefile src/__init__.py\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "09ee1385",
"metadata": {},
"source": [
"Next we create the file that contains the converter code"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aa345e79",
"metadata": {},
"outputs": [],
"source": [
"%%writefile src/mfconverter/converter.py\n",
"from asdf.extension import Converter\n",
"\n",
"class PhotoIDConverter(Converter):\n",
" tags = [\"asdf://stsci.edu/example-project/tags/photo_id-*\"]\n",
" types = [\"mfconverter.photo_id.PhotoID\"]\n",
" # The above registers the tag that the converter is used for, as well as\n",
" # associating the class that the converter is used for.\n",
" \n",
" # This method converts from the Python object to yaml\n",
" def to_yaml_tree(self, obj, tags, ctx):\n",
" # The yaml conversion expects a dictionary returned\n",
" node = {}\n",
" node['first_name'] = obj.first_name\n",
" node['last_name'] = obj.last_name\n",
" node['photo'] = obj.photo\n",
" return node\n",
" \n",
" # This method converts from yaml to the Python object\n",
" def from_yaml_tree(self, node, tag, ctx):\n",
" from .photo_id import PhotoID # Deferred import to avoid always importing \n",
" # when ASDF gets entry points.\n",
" return PhotoID(node['last_name'],\n",
" node['first_name'],\n",
" node['photo'])"
"The files for `mfconverter` are located in the `Your_first_ASDF_converter` directory alongside this notebook. Please review the files. A few specific details will be highlighted below."
]
},
{
"cell_type": "markdown",
"id": "55ca30cd",
"metadata": {},
"source": [
"In the above example, the class attribute `types` is a list since it is\n",
"In `PhotoIDConverter` defined in `converter.py`, the class attribute `types` is a list since it is\n",
"possible for multiple types to use the same converter. Secondly, it is\n",
"possible to supply the type as either a string (as done above) as the \n",
"actual class itself. The former is preferred for peformance reasons as\n",
"using the actual class itself forces the ASDF package to import the\n",
"module containing the class, even if it is never used in the program\n",
"using ASDF.\n",
"using asdf.\n",
"\n",
"The wildcard for the tag entry indicates that this converter will work\n",
"with all versions of the tag. It isn't strictly needed here, but generally \n",
"a good practice if one wants the converter code to handle multiple versions.\n",
"(How to handle versioning is a topic in its own right and not covered here)\n",
"\n",
"We need to create a module for the entry point handling"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f9c03910",
"metadata": {},
"outputs": [],
"source": [
"%%writefile src/mfconverter/extensions.py\n",
"from asdf.extension import Extension\n",
"from .converter import PhotoIDConverter\n",
"\n",
"class MFExtension(Extension):\n",
" extension_uri = \"asdf://stsci.edu/example-project/photo_id-1.0.0\"\n",
" tags = [\"asdf://stsci.edu/example-project/tags/photo_id-1.0.0\"]\n",
" converters = [PhotoIDConverter()]\n",
"\n",
"# The following will be called by ASDF when looking for ASDF entry points \n",
"def get_extensions():\n",
" return [MFExtension()]"
"(How to handle versioning is a topic in its own right and not covered here)"
]
},
{
"cell_type": "markdown",
"id": "412c3e4f",
"metadata": {},
"source": [
"And finally, the entry point reference in setup.cfg, provided here as a whole file."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "feb206d7",
"metadata": {},
"outputs": [],
"source": [
"%%writefile pyproject.toml\n",
"[project]\n",
"name = \"mfconverter\"\n",
"description = \"TODO\"\n",
"version='0.1.0'\n",
"requires-python = \">=3.9\"\n",
"dependencies = [\n",
" \"jsonschema >=3.0.2\",\n",
" \"asdf >=2.8\",\n",
" \"psutil >=5.7.2\",\n",
" \"numpy>=1.16\",\n",
"]\n",
"\n",
"[build-system]\n",
"requires = [\"setuptools >=61\", \"setuptools_scm[toml] >=3.4\"]\n",
"build-backend = \"setuptools.build_meta\"\n",
"As mentioned above `mfconverter` defines an entry point in the `pyproject.toml` configurate file that is used to register `PhotoIDConverter` with asdf.\n",
"\n",
"[project.entry-points.\"asdf.extensions\"]\n",
"mfconverter = \"mfconverter.extensions:get_extensions\"\n",
"\n",
"[tool.setuptools.packages.find]\n",
"where = [\"src\"]"
]
},
{
"cell_type": "markdown",
"id": "8d0e3882",
"metadata": {},
"source": [
"If you wish to learn more about entry points, see:\n",
"https://packaging.python.org/guides/creating-and-discovering-plugins/\n",
"\n",
"We need the setup.py file too"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3ffb22d6",
"metadata": {},
"outputs": [],
"source": [
"%%writefile setup.py\n",
"#!/usr/bin/env python3\n",
"from setuptools import setup\n",
"\n",
"setup()"
"https://packaging.python.org/guides/creating-and-discovering-plugins/"
]
},
{
Expand All @@ -369,13 +124,7 @@
"metadata": {},
"source": [
"Install package\n",
"------------------\n",
"\n",
"This is best done from a terminal window using the command\n",
"\n",
"`pip install --editable .`\n",
"\n",
"But this notebook will do that, but a consequence is that if the reinstallation must be done, the Jupyter kernel must be restarted to pick up the new installation."
"------------------"
]
},
{
Expand All @@ -385,7 +134,7 @@
"metadata": {},
"outputs": [],
"source": [
"!pip install --editable ."
"!pip install ./Your_first_ASDF_converter"
]
},
{
Expand Down Expand Up @@ -539,7 +288,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.10"
"version": "3.13.4"
}
},
"nbformat": 4,
Expand Down
18 changes: 18 additions & 0 deletions notebooks/Your_first_ASDF_converter/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[project]
name = "mfconverter"
description = "TODO"
version='0.1.0'
requires-python = ">=3.9"
dependencies = [
"asdf >=2.8",
]

[build-system]
requires = ["setuptools >=61", "setuptools_scm[toml] >=3.4"]
build-backend = "setuptools.build_meta"

[project.entry-points."asdf.extensions"]
mfconverter = "mfconverter.extensions:get_extensions"

[tool.setuptools.packages.find]
where = ["src"]
Empty file added notebooks/Your_first_ASDF_converter/src/mfconverter/__init__.py
View file
Open in desktop
Empty file.
Loading