Initial Project Milestones

The following represents the initial plan for the milestones. We envision an agile approach, and when circumstances deem necessary, modifying the milestones to take advantage of prior development experience, or external inputs changing the priority of features. Nevertheless, the following are very basic needs and we do not expect large changes.

Milestone 1: low-level read-only API

Define and implement a low-level API that provides support for:

1. Reading the YAML and binary portions of an ASDF file
2. Representing the YAML and binary blocks in structures familiar to a C programmer
3. This API will not handle:
   1. Decompression of block data
   2. Mapping of tags to custom objects
   3. Validation

This milestone will be reached when all reference files for the current ASDF standard (1.6.0) can be successfully read and contents mapped to C structures.

Milestone 2: C-native API (with array support)

Using the low-level API developed in milestone 1. Develop a “C-native” API that supports:

• Mapping tags to custom objects using an internal extension framework
• Maps the standard tags (history_entry, array, integer, etc)

Milestone 3: Roman file reader

Use the low-level API to open a Roman file parsing the metadata and array data and mapping Roman WCS tags to an object that allows mapping pixel coordinates to celestial coordinates. Note that the WCS functionality will not replicate the Python WCS functionality in its generality; the goal in this case is to handle the very specific transforms that Roman uses for imaging. In this version this functionality will be part of the base library; eventually it will be moved to an extension once the extension framework is in place.

Milestone 4: low-level write API

Building off Milestone 1, develop a low-level API for writing ASDF files.

This milestone will be reached when all reference files used in Milestone 1 can be written to disk in a format that is equivalent to the existing reference file with only expected difference (different software used to produce the file recorded in the “asdf_library” and “history” portions of the tree). This version of the API will not handle all the options available in Python; just the basic functionality to write files minus features such as streaming, compression, updates to currently open files, etc.

Milestone 5: Extension API

Prototype an extension API that allows external packages to hook into the C library to map tags to custom objects. The eventual goal will be support for the Roman WCS as an extension. For this milestone the target will be more limited aiming to support mapping the model/transforms in the WCS to custom objects using the plugin/extension API.

Appendix A: Topics to consider

1. Reading
   1. Pure yaml file
      1. Anchors and aliases
      2. Custom tags
      3. JSON Pointers
   2. File with blocks
   3. Streamed blocks
   4. External blocks
2. Block access
   1. File pointer and offset
   2. Memory map
   3. Array of bytes
   4. Shaped array of datatype defined in tree
3. Writing
4. Schema Validation
5. Extensions
   1. Manifest discovery and processing
   2. Custom schema discovery and processing
   3. Tags
6. Compression
   1. “Builtin” compression
   2. Compression extensions
7. API
   1. HPC “friendly”
   2. Integration with python and other languages (perhaps at various levels)
8. Version support
   1. How far back should we support? Any version <=1.5.0 technically needs to fill defaults from the schema (which would require a schema parser, etc).
   2. For reading?
   3. For writing?