Create build_pipelines.rst (issue #25)

[ehennestad]

Adresses issue #25

I added this as a first draft to describe how the openminds pipelines work. @lzehl @olinux

[ehennestad]

Preview here:
https://github.com/openMetadataInitiative/openMINDS_documentation/blob/iss25-doc-pipeline/docs/shared/contribution_guidelines/build_pipelines.rst

[lzehl]

please adapt to #35

[lzehl]

more suggestions are in the process and coming asap

[lzehl]

Suggested change

	openMINDS pipelines
	Framework infrastructure

[lzehl]

Suggested change

openMINDS pipelines are specialized branches within a subset of the repositories in the openMetadataInitiative organization. The purpose of these branches is to build an up-to-date version of schemas or derived schema representations. Pipelines are triggered by a set of GitHub Actions workflows whose job it is to ensure the continual synchronization of interdependent repositories.

The openMINDS metadata framework is developed in distributed, pipeline connected GitHub repositories. It differentiates between frontend-facing repositories (metadata-model, instance-libraries, targets, and converters) relevant for end-users and developers as well as backend-facing repositories (schema-modules and utils) only relevant for developers. Moreover, one can distinguish between repositories that only handle schemas (schema-modules, metadata-model), instances (instance-libraries), or both (targets, converters). All repositories are managed and connected through various automated pipelines (primary, secondary, and tertiary build pipelines, validation pipelines). Actions running across various repositories are maintained in dedicated repositories (utils).

[lzehl]

Suggested change

	We’ll first outline the different repository types in the openMetadataInitiative organization before we describe the existing workflows and the pipeline branches in more detail.
	We’ll first outline the different repository types before we describe the pipelines in more detail.

[Raphael-Gazzotti]

We will first outline...

[lzehl]

Suggested change

	There are three types of repositories in the Open Metadata Initiative Organization:
	1. "Module" repositories
	2. "Central" repository
	3. "Target" repositories
	The openMINDS metadata framework identifies the following repository types (cf. GitHub topics of repositories):
	1. schema-module
	2. metadata-model
	3. instance-libraries
	4. target
	- schema-format
	- programming-package
	- documentation
	5. converter
	6. utils

[lzehl]

Suggested change

	Module Repositories
	~~~~~~~~~~~~~~~~~~~
	These repositories represent individual openMINDS metadata modules, for example `openMINDS_core`_ or `openMINDS_SANDS`_. It also includes the `openMINDS_instances`_ repository which is a library of controlled metadata instances for metadata schemas/types of the various openMINDS modules. Development of openMINDS schemas (and instances) will happen on these module repositories, and any suggestion for a schema modification should happen through a Pull Request to the respective repository.
	.. _openMINDS_core: https://github.com/openMetadataInitiative/openMINDS_core
	.. _openMINDS_SANDS: https://github.com/openMetadataInitiative/openMINDS_SANDS
	.. _openMINDS_instances: https://github.com/openMetadataInitiative/openMINDS_instances
	Repository type: schema-module
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	All schema-module repositories hosts the simple, content-focused templates (.schema.tpl.json) which are used to build the end-user relevant openMINDS schemas on the metadata-model repository. These schema templates are thematically grouped into different repositories (e.g., `openMINDS_publication`_, `openMINDS_ephys`_). The primary build pipeline is automatically triggered via webhooks on the schema-modules (``.github/workflows/openMINDS_upstream.yml``).
	.. _openMINDS_publication: https://github.com/openMetadataInitiative/openMINDS_publication
	.. _openMINDS_ephys: https://github.com/openMetadataInitiative/openMINDS_ephys

[Raphael-Gazzotti]

Suggested change

	Central repository
	~~~~~~~~~~~~~~~~~~
	The **central** repository (`openMINDS`_) contains schemas for the full openMINDS model in the omi.json format. These schemas are extended and “resolved” [link to description of the tpl.json and omi.json formats] variants of the schema templates found in the respective **module** repositories. The **central** repository should be considered the ground truth for schemas.
	.. _openMINDS: https://github.com/openMetadataInitiative/openMINDS
	Repository type: metadata-model
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	The metadata-model repository hosts the end-user relevant openMINDS schemas (.schema.omi.json) and associated vocabulary that describe types, properties and categories. These schemas are extended and “resolved” variants of the schema templates found in the corresponding schema-module repositories. They are used to build the target repositories. The secondary build pipeline is automatically triggered via webhook on the metadata-model repository (``.github/workflows/openMINDS_upstream.yml``).
	Repository type: instance-libraries
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	The instance-libraries repository hosts curated instances (.jsonld) for selected openMINDS schemas. These instances may be used to build target repositories. The secondary build pipeline is automatically triggered via webhook on the instance-libraries repository (``.github/workflows/openMINDS_upstream.yml``).

[Raphael-Gazzotti]

Suggested change

**Note**: This is a draft for text to be added to the ReadTheDocs documentation pages

[Raphael-Gazzotti]

Suggested change

	Target repositories
	~~~~~~~~~~~~~~~~~~~
	**Target** repositories are repositories that depend on the ground truth schemas in the **central** repository. Examples include `openMINDS_Python`_, `openMINDS_MATLAB`_, `openMINDS_Java`_, and `openMINDS_documentation`_. These repositories should be updated whenever changes are made to the ground truth schemas, i.e., the **central** `openMINDS`_ repository.
	.. _openMINDS_Python: https://github.com/openMetadataInitiative/openMINDS_Python
	.. _openMINDS_MATLAB: https://github.com/openMetadataInitiative/openMINDS_MATLAB
	.. _openMINDS_Java: https://github.com/openMetadataInitiative/openMINDS_Java
	.. _openMINDS_documentation: https://github.com/openMetadataInitiative/openMINDS_documentation
	Repository type: target
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	Target repositories are automatically generated using content from the metadata-model repository and, optionally, from the instance-libraries repository. They represent the final products delivered to end-users in specific formats or for specific use-cases. Depending on the purpose, a target repository may fall into one of the following subtypes:
	* **Repository type: schema-format**
	The schema-format repositories contain openMINDS schemas converted into standard formats (e.g., JSON Schema, LinkML) suitable for validation or data integration.
	* **Repository type: programming-package**
	The programming-package repositories host language-specific libraries (e.g., Python, Java) that provide programmatic access to openMINDS schemas and facilitate the creation, manipulation, and validation of metadata instances.
	* **Repository type: documentation**
	The documentation repository provides human-readable documentation derived from the schemas, including property descriptions, usage guidelines, and diagrams. This helps users understand and apply the openMINDS metadata-models effectively.
	Repository type: converter
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	Converter repositories contain tools or programs designed to translate data to and from the openMINDS-compliant metadata format.
	Repository type: utils
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	Utils repositories provide general-purpose utilities that support development and data handling across the openMINDS framework.
	Version Management
	---------------------
	There are two distinct versioning systems used for the schema-modules and the OpenMINDS metadata-model. Each schema-module repository uses branch based versioning, where the branch with the highest numerical value represents the most up-to-date version. In contrast, the OpenMINDS metadata-model manages its versions within the `schemas`_ directory. Here also, the folder with the highest numerical version number indicates the latest version.
	The schema-modules and their corresponding versions used to build the OpenMINDS metadata-model via the primary build pipeline are referenced in the `pipeline` branch of the metadata-model repository, specifically in the `versions.json`_ and `versions-dev.json`_ files.
	.. _schemas: https://github.com/openMetadataInitiative/openMINDS/tree/main/schemas
	.. _versions.json: https://github.com/openMetadataInitiative/openMINDS/blob/pipeline/versions.json
	.. _versions-dev.json: https://github.com/openMetadataInitiative/openMINDS/blob/pipeline/versions-dev.json

[Raphael-Gazzotti]

Suggested change

	Repository dependencies
	-----------------------
	The **central** repository depends on each of the **module** repositories, and the **target** repositories all depend on the **central** repository. Once any of the **module** repositories have a new commit (push), they trigger the build process of **central** which aggregates the information across all of those **modules** and merges them in a single (versioned) view. Once this is done, the **target** build pipelines are triggered to then consume the centrally provided information to build their "language-specific" products.
	Overview of Workflows
	---------------------
	There are currently two types of GitHub Actions workflows, ``openMINDS_upstream``, and ``build``.
	openMINDS_upstream (module repositories)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	Each of the module repositories has a workflow called ``openMINDS_upstream``. This workflow is triggered whenever something is pushed to the respective repository (todo: mention version branches), and it has a single job which is to trigger the build pipeline of the **central** repository. An example of such a workflow is found `here`_.
	.. _here: https://github.com/openMetadataInitiative/openMINDS_core/blob/e873947bd8cae1de08aebd3ecc59a1c81d218cb3/.github/workflows/openMINDS_upstream.yml
	openMINDS_upstream (central repository)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	When a push is made on the ``main`` branch of the **central** `openMINDS`_ repository, the **central** ``openMINDS_upstream`` workflow is triggered. This workflow has a single job which is to trigger the ``build`` workflow of each of the **target** repositories.
	Suggestion: rename ``openMINDS_upstream`` of **central** repo to ``openMINDS_trigger_targets``
	Build (central and target repositories)
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	The **central** repository (`openMINDS`_) and the **target** repositories have a branch called ``pipeline`` that contains the ``build`` workflow and the build routines for that repository. When the ``build`` workflow is triggered, the build routines will execute and the built product is pushed to the ``main`` branch of the respective repository.
	Overview of the Workflows
	---------------------
	The openMINDS framework is supported by three main categories of automated workflows:
	1. a primary build pipeline for generating end-user schemas from simplified templates, and
	2. secondary build pipelines for producing documentation, format conversions, programming-packages, and converters.
	3. tertiary build pipelines that enable integration of openMINDS into external applications, such as domain-specific converters (e.g., bids2openMINDS) and database systems (e.g., MarmotGraph).
	Primary build pipeline
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	Using the simplified templates (.schema.tpl.json) written in the schema-modules repositories, the primary build pipeline will automatically generate the end-user relevant schemas (.schema.omi.json) for each version of the openMINDS metadata-model.
	Changes to schema-module repositories trigger the primary build pipeline via webhooks (.github/workflows/openMINDS_upstream.yml), targeting:
	* the ``development`` branch of the metadata-model repository, or
	* the ``main`` branch, if the updated schema-module corresponds to the latest release.
	The pipeline is also triggered by updates to versioning files (versions.json or versions-dev.json) located in the ``pipeline`` branch.
	The workflows responsible for this process (build.yml and build-dev.yml) are also located in the ``pipeline`` branch and rely on code within the same branch to convert the templates into their final, end-user formats.
	Secondary build pipelines
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	Using the content of the metadata-model and instance-libraries repositories, the secondary build pipelines can automatically:
	1. Create the openMINDS documentation (documentation repository)
	2. Transform the openMINDS metadata-model into common target formats such as JSON Schema (.schema.json) or LinkML (.yaml) (schema-format repositories)
	3. Generate openMINDS code packages for, e.g., Python, MATLAB and Java (programming-package repositories).
	The ``build.yml`` workflows responsible for this process are located in the ``pipeline`` branch of the target repositories and rely on code within the same branch to execute the actions described above. These workflows are triggered via webhooks (.github/workflows/openMINDS_upstream.yml) when changes occur in the metadata-model or instance-libraries repositories.
	Tertiary build pipelines
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	Using the content of the metadata-model and instance-libraries reposiories, the tertiary build pipelines allows to adopt openMINDS in other applications such as database systems or converters (converter repositories).