fix(docs): install SDK packages in V2 deploy for legacy builder#1432
Closed
hkad98 wants to merge 138 commits intogooddata:masterfrom
Closed
fix(docs): install SDK packages in V2 deploy for legacy builder#1432hkad98 wants to merge 138 commits intogooddata:masterfrom
hkad98 wants to merge 138 commits intogooddata:masterfrom
Conversation
Lock `uv.lock` after version bump and also exclude `uv.lock` from copyright.
It is necessary to install specific uv version – the same that is specified in pyproject.toml. This is just a workaround to unblock PR. It is should be redesign eventually to read the information about versions form pyproject.toml
There was a perf degradation in tests for Python 3.12 with coverage. Setting to COVERAGE_CORE=sysmon helped.
Note that docker-compose.yaml file approach does not work anymore. Therefore, this client was regenerated against the main repo docker-compose. We will need to fix this in the future.
There was a breaking change on the backend that rendered `update_oidc_parameters` unusable. OIDC was moved from the organization to the IdentityProvider entity; therefore, this API stopped working. Use `update_identity_provider` instead.
risk: low JIRA: WEB-4462
It is necessary to specify test dependencies in the root pyproject.toml; otherwise, they won't be installed using `make dev`. It is necessary to keep those dependencies in packages as well; otherwise, tox will not work.
risk: low JIRA: DP-3290
risk: low JIRA: 3290
See [link](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license). `license expression` in classifiers is deprecated.
risk: low JIRA: DP-3290
We do not need `.envrc` anymore. Everything is managed by uv now. Migrate `tox.ini` in gooddata-flexconnect to uv runner.
risk: low JIRA: WEB-4462
Since the InlineFilters can occur in the ReportExecutionRequest object, the converter needs to support those instead of raising errors. JIRA: GDAI-842 risk: low
During uv migration CLIs were migrated incorrectly. JIRA: PSDK-225 risk: low
Since a lot of people have uv installed so that it gets updated automatically, let's relax the version spec a bit. JIRA: TRIVIAL risk: low
jira: CQ-1650 risk: low
Add partialCached to three hot paths identified via --templateMetrics. head-css.html: SCSS compilation was repeated for all 860+ pages despite identical output. Pass .Site as context so partialCached shares a single cache entry (100% hit rate, was 0%). structureddata.looker.html: iterated ALL site pages to build a title list on every page render, O(n^2) total work. Extract the computation into a cached capital-titles.html partial (84s to 4s). sidebar.html: SVG chevron icon was loaded, minified, and string-replaced inside the recursive nav template (~15k invocations). Extract into cached chevron-svg.html partial (100% hit rate). Verified locally: before/after HTML output is byte-identical (diff -rq). Build time on 864 pages: 18.5s to 13.1s (29% faster). JIRA: trivial risk: nonprod
…nfig field Replace raw dict[str, dict[str, str]] with the typed CatalogGeoAreaConfig from gooddata-sdk. This enables cattrs validation at dbt manifest parse time, so malformed or misnamed config keys (e.g. camelCase geoAreaConfig) fail early with a clear error instead of silently producing None. jira: TRIVIAL risk: nonprod
Add SDK abstraction for the LLM Provider entity with full CRUD operations via the entities API. The implementation uses dedicated, provider-scoped types for both configuration and authentication: - Provider configs: CatalogOpenAiProviderConfig, CatalogAwsBedrockProviderConfig, CatalogAzureFoundryProviderConfig - Provider-scoped auth: CatalogOpenAiApiKeyAuth, CatalogBedrockAccessKeyAuth, CatalogAzureFoundryApiKeyAuth - Per-provider auth unions allow adding new auth types per provider independently Service methods on CatalogOrganizationService: get, list, create, update (PATCH), and delete for LLM providers. This feature is Beta — the public API may change in future releases. jira: trivial risk: nonprod
Reintroduce the agent guidance files and AIDA selection defaults generated by the latest aida-mcp init run. Include hook-required normalization (copyright in Codex config and EOF cleanup) to keep onboarding files policy-compliant. risk: low
`create_directory` uses a check-then-create pattern (`os.path.exists` + `os.makedirs`) which is not atomic. When multiple threads call `load_from_disk` on the same path concurrently (e.g. parallel workspace uploads), they race on creating directories like `analytical_dashboard_extensions`, causing `FileExistsError`. Replace with `os.makedirs(path, exist_ok=True)` which handles this atomically. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…hortcodes Hugo was spending ~8 minutes processing ~4,140 API ref shortcodes per version (O(n²) regex in api-ref-link-all-partial). This moves HTML rendering into python_ref_builder.py using Jinja2 templates and a pre-compiled LinkResolver with O(1) dict lookups, eliminating all shortcode processing time. Key changes: - Two-pass generation: pass 1 builds links dict, pass 2 renders HTML - Jinja2 templates replicate the exact output of Hugo shortcodes/partials - Markdown templates now receive pre-rendered CONTENT instead of shortcodes - jinja2 added to script-requirements.txt - CI workflow: use master's scripts for all versions, hash-based cache key - Dockerfile: fix COPY paths, persist Go modules in image layer, remove unnecessary data.json/links.json copies to versioned_docs - Restore api-reference/_index.md and pandas/_index.md for Hugo navigation Result: ~3x faster Hugo build (57s → 19s per version), API ref template time eliminated entirely. jira: trivial risk: high
Add 33 tests covering LinkResolver, template rendering, context builders, and file structure creation. Fix pre-existing type annotation issues (RefHolder.packages, parse_toml return type) and remove dead links.json generation that is no longer consumed after the pre-render change. Add Makefile test-docs-scripts target and CI docs-scripts-tests job with scripts/docs/** path filter. JIRA: trivial risk: nonprod
Fix two bugs found during review: (1) generate.sh crashes on missing links.json after the pre-render change removed its generation — add if-guard around the sed/mv. (2) V2 cache key included test files via scripts/docs/** glob, causing unnecessary cache busts — narrow to scripts/docs/*.py + templates/**. Also add scripts/docs/README.md documenting how the three documentation workflows operate. JIRA: trivial risk: high
uv sync --locked does not install pytest since it lives in the test dependency group. Changed to --group test to make pytest available. JIRA: trivial risk: nonprod
python_ref_builder imports toml which is not a transitive dependency of any workspace package. Adding it to the test group so the docs-scripts-tests CI job can find it. JIRA: trivial risk: nonprod
Remove leftover data.json and links.json handling that is no longer needed after pre-rendering API reference HTML in Python. The link resolution is now baked into the HTML at generation time, so these intermediate files serve no purpose. Also remove --baseURL from the V2 draft workflow so Netlify draft deploys are browsable instead of redirecting to production. jira: trivial risk: nonprod
The latest gateway-api-gw image requires GRPC_MDSINK_HOST/PORT and GEN_AI_HTTP_HOST/PORT environment variables. Without these, api-gw crashes on startup, blocking the bootstrap sequence. Also regenerates VCR cassettes to match current API responses.
Add the `result` spec (`/api/v1/schemas/result`) to the Makefile download target and merge it into the combined api-client schema. Regenerate gooddata-api-client, which brings typed endpoints and models for CSV staging upload, CSV analysis, CSV import, file listing, file deletion, and CSV file manifest reading. jira: TRIVIAL risk: low
Add five new methods to CatalogDataSourceService: - staging_upload — upload a CSV to the staging area - analyze_csv — detect columns, types, and parse config - import_csv — import a staged CSV into a GDSTORAGE data source - delete_csv_files — remove files from a data source - upload_csv — end-to-end convenience wrapper Also add CatalogDataSourceGdStorage entity model with _NoCredentials support (GDSTORAGE sources require no authentication), export the new symbol, and add documentation pages for all new methods. jira: TRIVIAL risk: low
…OpenAPI Generator OpenAPI Generator v6.6.0 silently drops the \x00 literal from regex patterns in the generated Python code. The DeclarativeColumn model's `name` field has a spec pattern of `^[^\x00]*$` (reject null bytes, important for Postgres TEXT columns), but the generator produced the invalid Python regex `^[^]*$`, causing `re.PatternError: unterminated character set` at runtime. Fix the generated file and add a post-generation sed fixup to the Makefile `api-client` target so future regenerations are patched automatically. jira: TRIVIAL risk: low
The OpenAPI Generator regeneration in e47faf7 introduced a pyproject.toml for gooddata-api-client. Since tbump has no entry for this file, uv build picks up the unbumped version from pyproject.toml instead of the bumped VERSION in setup.py, producing a wheel whose version already exists on PyPI. Remove the file so the build continues to use setup.py as the sole version source, matching the tbump configuration. jira: TRIVIAL risk: nonprod
Refresh the lockfile so gooddata-api-client dependencies are resolved from setup.py instead of the removed pyproject.toml. jira: TRIVIAL risk: nonprod
JIRA: DX-340 risk: nonprod
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> JIRA: DX-340 risk: low
…palette management Add sdk.catalog_appearance with CRUD operations (list, get, create, update, delete) for themes and color palettes via the AppearanceApi. Follows the existing service-based facade pattern. The API already exposed these endpoints but the SDK had no wrapper. jira: trivial risk: low
…ervice Update copyright from 2024 to 2026 in all new appearance module files. Migrate documentation from markdown tables to Hugo shortcodes (parameters-block/parameter) to match the existing documentation style. jira: trivial risk: nonprod
The docs json_builder.py script expects Google-style docstrings but several functions used RST-style (:param/:return:), had missing summary lines before Args sections, or had improperly indented continuation lines — all causing "Invalid docstring" warnings during docs generation. jira: TRIVIAL risk: low
…rer pipeline Replace 136 manually maintained method documentation pages with a directive-based system where signature, parameters, and returns are auto-generated from code docstrings via griffe static AST analysis. Key changes: - Add griffe_builder.py: static AST replacement for json_builder.py — produces identical data.json without requiring package installation (~1.1s vs ~1.8s) - Add method_page_renderer.py: pre-renders method pages from api_ref frontmatter, with type names hyperlinked to API reference pages via links.json - Add migrate_method_pages.py: one-time migration that converted 136 pages to api_ref directive format, preserving hand-written Example sections - Update python_ref_builder.py: fix dedup bug that skipped recursion into duplicate-named modules (e.g. CatalogWorkspace was unreachable), add --export-links flag to export type->URL mapping for method_page_renderer - Update generate.sh and generate-single-version.sh: prefer griffe over legacy json_builder, call method_page_renderer with --links-json - Fix 9 docstrings with invalid "Args: None" pattern and 3 with missing blank lines before Args/Returns sections - Delete 12 unused Hugo shortcode/partial templates (api-ref-*, parameter*) - Add 91 unit tests (up from 33) covering griffe builder, method renderer, link resolution, frontmatter parsing, idempotency, and edge cases - Update scripts/docs/README.md with new pipeline documentation jira: trivial risk: nonprod
…rer pipeline Replace 136 manually maintained method documentation pages with a directive-based system where signature, parameters, and returns are auto-generated from code docstrings via griffe static AST analysis. Key changes: - Add griffe_builder.py: static AST replacement for json_builder.py — produces identical data.json without requiring package installation (~1.1s vs ~1.8s) - Add method_page_renderer.py: pre-renders method pages from api_ref frontmatter, with type names hyperlinked to API reference pages via links.json - Add migrate_method_pages.py: one-time migration that converted 136 pages to api_ref directive format, preserving hand-written Example sections - Update python_ref_builder.py: fix dedup bug that skipped recursion into duplicate-named modules (e.g. CatalogWorkspace was unreachable), add --export-links flag to export type->URL mapping for method_page_renderer - Update generate.sh and generate-single-version.sh: prefer griffe over legacy json_builder, call method_page_renderer with --links-json - Fix 9 docstrings with invalid "Args: None" pattern and 3 with missing blank lines before Args/Returns sections - Delete 12 unused Hugo shortcode/partial templates (api-ref-*, parameter*) - Add 91 unit tests (up from 33) covering griffe builder, method renderer, link resolution, frontmatter parsing, idempotency, and edge cases - Update scripts/docs/README.md with new pipeline documentation jira: trivial risk: nonprod
fix(docs): install SDK packages in V2 deploy for legacy builder Older release branches lack griffe_builder.py, so the build falls back to json_builder.py which imports gooddata_sdk and gooddata_pandas at runtime. The V2 workflow only installed script-requirements.txt, causing ModuleNotFoundError for branches using the legacy path. JIRA: trivial risk: nonprod
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
griffe_builder.py, so the Netlify V2 build falls back tojson_builder.pywhich importsgooddata_sdkandgooddata_pandasat runtime.script-requirements.txt, missing the actual SDK packages — causingModuleNotFoundError: No module named 'gooddata_pandas'.pip install -eforgooddata-api-client,gooddata-sdk, andgooddata-pandasin the generate-version job.