StackQL Provider Utils

A comprehensive toolkit for transforming OpenAPI specifications into StackQL providers. This library streamlines the process of parsing, mapping, validating, testing, and generating documentation for StackQL providers.

Table of Contents

• Prerequisites
• Installation
• Directory Structure
• Provider Development Workflow
  • providerdev.split - Divide a large OpenAPI specification into smaller, service-specific files
  • providerdev.normalize - Reshape split specs for relational consumption (flatten allOf, rename oneOf/anyOf, strip misplaced keywords, lower opaque objects)
  • providerdev.analyze - Examine split API specifications to generate mapping recommendations
  • providerdev.generate - Create StackQL provider extensions from specifications and mappings
  • docgen.generateDocs - Generate comprehensive documentation for StackQL providers
• Contributing
• License
• Support

Prerequisites

• Node.js >= 20
• npm or yarn
• stackql for testing

Installation

Add @stackql/provider-utils to your package.json:

npm install @stackql/provider-utils
# or
yarn add @stackql/provider-utils

Consumer Setup: npm scripts

The package ships two CLI entry points: provider-dev-utils (for split / normalize / analyze / generate) and docgen-utils (for generate-docs / generate-docs-v2). Wrap them as npm scripts in your provider repo's package.json so every step of the workflow is a single npm run <step> invocation:

{
  "scripts": {
    "split":             "node node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs split",
    "normalize":         "node node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs normalize",
    "generate-mappings": "node node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs analyze",
    "generate-provider": "node node_modules/@stackql/provider-utils/bin/provider-dev-utils.mjs generate",
    "generate-docs":     "node node_modules/@stackql/provider-utils/bin/docgen-utils.mjs generate-docs"
  }
}

Call the bins through node (rather than relying on node_modules/.bin/ shims) because npm does not always link .bin shims for .mjs files on Windows. Pass flags through with npm's -- separator, e.g.:

npm run normalize -- --api-dir provider-dev/source --verbose

JSON-valued flags (--servers, --provider-config, --service-config, --skip-files, --svc-name-overrides) accept either an inline JSON string or a path to a JSON file.

Full workflow example

npm run split -- \
  --provider-name github \
  --api-doc provider-dev/downloaded/api.github.com.json \
  --svc-discriminator tag \
  --output-dir provider-dev/source \
  --overwrite

npm run normalize -- \
  --api-dir provider-dev/source \
  --verbose

npm run generate-mappings -- \
  --input-dir provider-dev/source \
  --output-dir provider-dev/config

npm run generate-provider -- \
  --provider-name github \
  --input-dir provider-dev/source \
  --output-dir provider-dev/openapi/src/github \
  --config-path provider-dev/config/all_services.csv \
  --servers '[{"url": "https://api.github.com"}]' \
  --provider-config '{"auth": {"type": "basic", "username_var": "STACKQL_GITHUB_USERNAME", "password_var": "STACKQL_GITHUB_PASSWORD"}}' \
  --overwrite

npm run generate-docs -- \
  --provider-name github \
  --provider-dir ./provider-dev/openapi/src/github/v00.00.00000 \
  --output-dir ./website \
  --provider-data-dir ./provider-dev/docgen/provider-data

On generate-provider, --provider-name is accepted as an alias for --provider-id so the flag name is consistent with the other commands.

Directory Structure

A typical project structure for the development of a stackql provider would be...

.
├── bin # convinience scripts
│   ├── ... 
├── provider-dev                            
│   ├── config
│   │   └── all_services.csv  # mappings generated or updated by the `providerdev.analyze` function, used by `providerdev.generate`
│   ├── docgen
│   │   └── provider-data  # provider metadata used by `docgen.generateDocs`
│   │       ├── headerContent1.txt
│   │       └── headerContent2.txt
│   ├── downloaded # used to store the original spec for the provider
│   │   └── management-minimal.yaml
│   ├── openapi # output from `providerdev.generate`, this is the stackql provider
│   │   └── src
│   │       └── okta
│   │           └── v00.00.00000
│   │               ├── provider.yaml
│   │               └── services
│   │                   ├── agentpools.yaml
│   │                   ├── ...
│   ├── scripts # optional scripts for pre or post processing if required
│   │   └── post_processing.sh
│   └── source  # output from `providerdev.split` if used, this is the source used with the mappings to generate the provider
│       ├── agentpools.yaml
│       ├── ...
└── website # docusaurus site
    ├── docs # output from `docgen.generateDocs`
    │   ├── ...

see stackql-provider-okta for a working example.

Provider Development Workflow

The library provides a streamlined workflow for creating StackQL providers from OpenAPI specifications:

1. providerdev.split - Divide a large OpenAPI specification into smaller, service-specific files
2. providerdev.normalize - Reshape split specs for relational consumption (flatten allOf, rename oneOf/anyOf, strip misplaced keywords, lower opaque objects)
3. providerdev.analyze - Examine split API specifications to generate mapping recommendations
4. providerdev.generate - Create StackQL provider extensions from specifications and mappings
5. docgen.generateDocs - Generate comprehensive documentation for StackQL providers

Contributing

Contributions are welcome!

License

MIT

Support

• StackQL Documentation
• GitHub Issues
• StackQL Community Slack
• StackQL Discord