Skip to content

feat: tiles deletion creation (MAPCO-5735)#95

Merged
CL-SHLOMIKONCHA merged 36 commits into
masterfrom
feat/tiles-deletion-skeleton
May 20, 2026
Merged

feat: tiles deletion creation (MAPCO-5735)#95
CL-SHLOMIKONCHA merged 36 commits into
masterfrom
feat/tiles-deletion-skeleton

Conversation

@razbroc
Copy link
Copy Markdown
Contributor

@razbroc razbroc commented May 7, 2026

Question Answer
Bug fix
New feature
Breaking change
Deprecations
Documentation
Tests added
Chore

Related issues: #MAPCO-5735

This pull request introduces several enhancements and refactors to support a new report storage provider, improve S3 integration, and add new ingestion task types. The most significant changes include configuration and infrastructure updates for report storage, the addition of a new tile deletion ingestion task, and improvements to error handling and external service integration.

Storage Provider and S3 Integration:

  • Added support for a reportStorageProvider option in configuration files (config/default.json, config/custom-environment-variables.json) and Helm charts (helm/templates/configmap.yaml, helm/values.yaml) to allow the selection of storage backends for reports, similar to existing GPKG and tile storage providers. [1] [2] [3] [4]
  • Updated Helm configuration logic to ensure S3-related environment variables are set if either the GPKG or report storage providers use S3.
  • Refactored the StorageProvider type in src/common/constants.ts to use values from @map-colonies/raster-shared, ensuring consistency across the codebase. [1] [2] [3]

Ingestion Task Enhancements:

  • Introduced a new tilesDeletion ingestion task type, including configuration options (type, tileBatchSize, taskBatchSize) and corresponding TypeScript interfaces. [1] [2] [3]
  • Updated the ingestion job handler to build and push tile deletion tasks using the new TileDeletionTaskManager, ensuring deletion tasks are processed before merge tasks. [1] [2] [3]

Dependency and Infrastructure Updates:

  • Bumped the @map-colonies/raster-shared package to version 8.1.0-alpha.3, enabling new features and types.
  • Added the unzip utility to the Docker image to support expanded data handling requirements.
  • Ensured S3 configuration is injected for both ingestion and export service instances in the dependency container setup.

Error Handling and External Service Integration:

  • Added a new IntersectionError class for improved error reporting when handling intersection operations with the polygon parts manager.
  • Extended the PolygonPartsMangerClient to support intersection requests, including error handling and logging for intersection failures. [1] [2]

razbroc and others added 10 commits April 26, 2026 18:07
…gurations

Co-authored-by: Copilot <copilot@github.com>
…ion in task building

Co-authored-by: Copilot <copilot@github.com>
…e for downloading reports

Co-authored-by: Copilot <copilot@github.com>
… refactor related interfaces and task parameters

Co-authored-by: Copilot <copilot@github.com>
…efining deletion task creation

Co-authored-by: Copilot <copilot@github.com>
razbroc and others added 6 commits May 7, 2026 14:37
Co-authored-by: Copilot <copilot@github.com>
…and deletion task manager

Co-authored-by: Copilot <copilot@github.com>
…URL handling

- Removed S3Service dependency from TileDeletionTaskManager and updated report handling to use a URL instead of a path.
- Modified readConflictFeatures function to download ZIP reports from a URL and extract shapefiles.
- Enhanced logging messages for clarity in TileMergeTaskManager.
- Cleaned up FSService constructor and type annotations for better readability.

Co-authored-by: Copilot <copilot@github.com>
@CL-SHLOMIKONCHA CL-SHLOMIKONCHA changed the title feat: tiles deletion creation MAPCO-5735 feat: tiles deletion creation (MAPCO-5735) May 11, 2026
Comment thread helm/templates/configmap.yaml Outdated
Comment thread helm/values.yaml
Comment thread src/common/constants.ts Outdated
Comment thread src/httpClients/polygonPartsMangerClient.ts
Comment thread src/job/models/ingestion/updateJobHandler.ts Outdated
Comment thread src/task/models/deletionTaskManager.ts Outdated
Comment thread src/utils/storage/fsService.ts Outdated
Comment thread package.json Outdated
Comment thread src/containerConfig.ts
Comment thread src/utils/storage/s3Service.ts
razbroc and others added 11 commits May 12, 2026 09:51
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
…skManager to use it

Co-authored-by: Copilot <copilot@github.com>
…eation.

Co-authored-by: Copilot <copilot@github.com>
…andling in deletion task creation

Co-authored-by: Copilot <copilot@github.com>
…processing in FSService

Co-authored-by: Copilot <copilot@github.com>
- Added "@types/node" dependency to package.json for better type support.
- Refactored UpdateJobHandler to improve code readability.
- Modified FSService constructor for consistency and updated file hashing logic to use 'data' event.
- Enhanced unit tests for updateJobHandler to include validation task handling.
- Updated jobManagerMocks to include findTasks mock for better test coverage.
- Added validationTaskForIngestionUpdate to tasksMockData for testing purposes.
- Adjusted Dirent type in fsService.spec.ts for improved type safety.

Co-authored-by: Copilot <copilot@github.com>
@razbroc razbroc requested a review from CL-SHLOMIKONCHA May 12, 2026 15:33
Comment thread src/common/interfaces.ts Outdated
Comment thread src/job/models/export/exportJobHandler.ts Outdated
Comment thread src/job/models/ingestion/updateJobHandler.ts Outdated
Comment thread src/job/models/ingestion/updateJobHandler.ts Outdated
Comment thread src/job/models/ingestion/updateJobHandler.ts Outdated
Comment thread src/task/models/deletionTaskManager.ts Outdated
Comment thread tests/unit/task/exportTaskManager/exportTaskManager.spec.ts
Comment thread tests/unit/mocks/tasksMockData.ts
Comment thread package.json Outdated
Comment thread package.json Outdated
razbroc and others added 6 commits May 18, 2026 15:34
Co-authored-by: Copilot <copilot@github.com>
…teJobHandler

Co-authored-by: Copilot <copilot@github.com>
…nd update related imports in TileDeletionTaskManager

Co-authored-by: Copilot <copilot@github.com>
@razbroc razbroc requested a review from CL-SHLOMIKONCHA May 19, 2026 20:35
Comment thread src/task/models/deletionTaskManager.ts
@CL-SHLOMIKONCHA CL-SHLOMIKONCHA merged commit 4474784 into master May 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants