Fetch Main #2
Open
Fetch Main #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
integrate new button into datasource overview
complete restructure of data flow inside the pipeline configuration all data flows over the vuex module, except flow up towards the PipelineEdit component, which gets handled by the PipelineTransformationConfig component.
…escheduling into own method
…nv variable declarations
Scheduler remove pipeline
Use adapter data in transformation configuration
This reverts commit c3838c6
* Allow self signed certs * Allow self sigend certs for storage-mq
* Update package-lock.json through npm install to new lockfileVersion * Applied new eslintrc to scheduler * pipeline: - Added new eslint and fixed all linter issues that were raised - Added and run prettier (also through lint) * Added files created by Local History extension to gitignore * notification: - Updated eslint + fixed all issues that arouse - added and executed prettier * small eslint fixes * scheduler: Upgraded typescript to 4.4.2 * pipeline: Upgraded typescript to 4.4.2 * notification: Upgraded typescript to 4.4.2 * storage-mq: - Updated eslint + fixed issues - added and executed prettier - Upgraded typescript to 4.4.2 * Updated node-dry-amqp and node-dry-pg for dependencies to fix bad engine version * updated vscode settings json * Added vetur config for monorepro vetur support * added prettier config * Applied most eslint updates with temporary manually updated eslintrc * Fixed monaco-editor import error * Fixed linter issues caused by merge * Added storage-mq as workingDirectory for eslint * Adjusted dockerfiles to also copy prettierrc * Fixed scheduler docker linting issue * Fixed notification docker linting issue * Fixed pipeline docker linting issue * Fixed storage-mq docker linting issue * Fixed ui docker linting issue * Fixed build issues in ui * Switched to jvalue/eslint-config package * Fixed linter issues caused by merge * - Updated jvalue/eslint-config package - fixed linter merge issues * Updated jvalue/eslint-config package * - Updated jvalue/eslint-config package - fixed linter merge issues * Updated jvalue/eslint-config package
) * Implement first basic contract test using pact * Fix lint errors * Integrate CDCT into GitHub Actions Workflow This approach uses the local file system for contract exchange * Fix error in CDCT-related docker files The contract files are now properly shared between providers and consumers. Previously, the provider did not use the newly generated contract files. * Delete all contract files This commit serves as a test to see if GitHub Actions commits the auto-generated contract files * Update auto-generated contracts * Create separate GitHub Actions Workflow for CDCT * Update auto-generated contracts * Improve mocking in pipeline provider test Now environment variables are properly mocked Also the main method is invoked to start the provider This removes the duplicate code in the provider test * Change pactfile write mode to overwrite Now existing pact files are entirely overwritten when consumer test execution brings changes to the files * Add CDCT for all interactions of UI and Pipeline * Replace static call to TransformationRest It is necessary to call the constructor and pass the pipeline service url * Add global test setup for UI Now when running CDCT, the previously existing contract files are deleted. Without this change, deprecated interactions would have to be deleted manually. * Fix error in Dockerfile.consumer of UI Previously the Dockerfile did not copy the pact folder. * Update auto-generated contracts * Move all interactions into a single test file With the previous approach, the order of the interactions in the contract file could be random. Now the order of the interactions in the contract file is determined by the order of the test cases. * Update auto-generated contracts * Update auto-generated contracts * Delete deprecated pact test This file was unintentionally added during merge resolution * Update auto-generated contracts * Remove obsolete common fixtures file * Implement first message-based CDC-Test The test covers an interaction between the notification service and the pipeline service * Update auto-generated contracts * Fix errors in GitHub Actions Workflow for CDCT Now the Consumer-Side fails if any Consumer-Test fails. The Provider-Side should run properly now. * Update auto-generated contracts * Make CDCT run in a single Workflow Job Previously newly generated contract files were not available to the Provider-Side Job. * Update auto-generated contracts * Add jest config file for pipeline provider test * Add new environment variable to test setup * Fix lint errors Co-authored-by: felix-oq <felix-oq@users.noreply.github.com>
#364) * Switched from local .prettierrc.js to using the @jvalue/eslint-config prettierrc through package.json * removed .prettierrc.js from Docker build
… docker compose (#370)
* Implement first basic contract test using pact * Fix lint errors * Integrate CDCT into GitHub Actions Workflow This approach uses the local file system for contract exchange * Fix error in CDCT-related docker files The contract files are now properly shared between providers and consumers. Previously, the provider did not use the newly generated contract files. * Delete all contract files This commit serves as a test to see if GitHub Actions commits the auto-generated contract files * Update auto-generated contracts * Create separate GitHub Actions Workflow for CDCT * Update auto-generated contracts * Improve mocking in pipeline provider test Now environment variables are properly mocked Also the main method is invoked to start the provider This removes the duplicate code in the provider test * Change pactfile write mode to overwrite Now existing pact files are entirely overwritten when consumer test execution brings changes to the files * Add CDCT for all interactions of UI and Pipeline * Replace static call to TransformationRest It is necessary to call the constructor and pass the pipeline service url * Add global test setup for UI Now when running CDCT, the previously existing contract files are deleted. Without this change, deprecated interactions would have to be deleted manually. * Fix error in Dockerfile.consumer of UI Previously the Dockerfile did not copy the pact folder. * Update auto-generated contracts * Move all interactions into a single test file With the previous approach, the order of the interactions in the contract file could be random. Now the order of the interactions in the contract file is determined by the order of the test cases. * Update auto-generated contracts * Update auto-generated contracts * Delete deprecated pact test This file was unintentionally added during merge resolution * Update auto-generated contracts * Remove obsolete common fixtures file * Implement first message-based CDC-Test The test covers an interaction between the notification service and the pipeline service * Update auto-generated contracts * Fix errors in GitHub Actions Workflow for CDCT Now the Consumer-Side fails if any Consumer-Test fails. The Provider-Side should run properly now. * Update auto-generated contracts * Make CDCT run in a single Workflow Job Previously newly generated contract files were not available to the Provider-Side Job. * Update auto-generated contracts * Add jest config file for pipeline provider test * Integrate CDCT into the regular ODS workflow Pacts are no longer committed to the repository. They are uploaded as an artifact instead. No separate CDCT workflow is required anymore. Reduces the number of docker-compose files by introducing additional stages in dockerfiles and overriding docker-compose files for testing. * Add build command to before running tests * Fix building step of integration test in notification * Fix command for provider-side test execution * Add new environment variable to test setup * Fix lint errors * Add bash scripts for executing CDC-Tests * Add Readme for CDCT * Add comparison to integration tests in CDCT Readme * Move all CDCT test files into their src directories This makes the location of the test files uniform and easier to find * Resolve merge conflict with dockerfiles * Extend build stage in dockerfiles Linting is now moved to the build stage to further speed up the building of the base stage. Unit test execution is added to the build stage again. This ensures, that docker images can only be built, if their respective unit test is passing. * Improve CDCT test scripts The scripts abort now, if any command fails. The script for the notification serivce now uses the scale option to prevent dependent but unnecessary services from being started during CDCT. * Use script files for CDCT in workflow * Fix exit codes of CDCT-scripts * Use alternative option to don't start dependent services Uses --no-deps instead of possibly multiple --scale options * Mark scripts for CDCT executable * Fix error in CDCT scripts The CDCT scripts can now be run from any directory. * Delete docker-compose.test.yml It is currently unnecessary to run unit tests with Docker Co-authored-by: felix-oq <felix-oq@users.noreply.github.com>
* Implement first basic contract test using pact * Fix lint errors * Integrate CDCT into GitHub Actions Workflow This approach uses the local file system for contract exchange * Fix error in CDCT-related docker files The contract files are now properly shared between providers and consumers. Previously, the provider did not use the newly generated contract files. * Delete all contract files This commit serves as a test to see if GitHub Actions commits the auto-generated contract files * Update auto-generated contracts * Create separate GitHub Actions Workflow for CDCT * Update auto-generated contracts * Improve mocking in pipeline provider test Now environment variables are properly mocked Also the main method is invoked to start the provider This removes the duplicate code in the provider test * Change pactfile write mode to overwrite Now existing pact files are entirely overwritten when consumer test execution brings changes to the files * Add CDCT for all interactions of UI and Pipeline * Replace static call to TransformationRest It is necessary to call the constructor and pass the pipeline service url * Add global test setup for UI Now when running CDCT, the previously existing contract files are deleted. Without this change, deprecated interactions would have to be deleted manually. * Fix error in Dockerfile.consumer of UI Previously the Dockerfile did not copy the pact folder. * Update auto-generated contracts * Move all interactions into a single test file With the previous approach, the order of the interactions in the contract file could be random. Now the order of the interactions in the contract file is determined by the order of the test cases. * Update auto-generated contracts * Update auto-generated contracts * Delete deprecated pact test This file was unintentionally added during merge resolution * Update auto-generated contracts * Remove obsolete common fixtures file * Implement first message-based CDC-Test The test covers an interaction between the notification service and the pipeline service * Update auto-generated contracts * Fix errors in GitHub Actions Workflow for CDCT Now the Consumer-Side fails if any Consumer-Test fails. The Provider-Side should run properly now. * Update auto-generated contracts * Make CDCT run in a single Workflow Job Previously newly generated contract files were not available to the Provider-Side Job. * Update auto-generated contracts * Add jest config file for pipeline provider test * Integrate CDCT into the regular ODS workflow Pacts are no longer committed to the repository. They are uploaded as an artifact instead. No separate CDCT workflow is required anymore. Reduces the number of docker-compose files by introducing additional stages in dockerfiles and overriding docker-compose files for testing. * Add build command to before running tests * Fix building step of integration test in notification * Fix command for provider-side test execution * Add new environment variable to test setup * Fix lint errors * Add bash scripts for executing CDC-Tests * Add Readme for CDCT * Add comparison to integration tests in CDCT Readme * Move all CDCT test files into their src directories This makes the location of the test files uniform and easier to find * Resolve merge conflict with dockerfiles * Extend build stage in dockerfiles Linting is now moved to the build stage to further speed up the building of the base stage. Unit test execution is added to the build stage again. This ensures, that docker images can only be built, if their respective unit test is passing. * Improve CDCT test scripts The scripts abort now, if any command fails. The script for the notification serivce now uses the scale option to prevent dependent but unnecessary services from being started during CDCT. * Use script files for CDCT in workflow * Fix exit codes of CDCT-scripts * Use alternative option to don't start dependent services Uses --no-deps instead of possibly multiple --scale options * Mark scripts for CDCT executable * Fix error in CDCT scripts The CDCT scripts can now be run from any directory. * Delete docker-compose.test.yml It is currently unnecessary to run unit tests with Docker * Execute CDCT after uploading docker images * Fix duplicate CDCT step due to faulty merge * Fix another duplicate CDCT step due to faulty merge * Add an explanation why CDC-Tests are executed after image upload * Remove unnecessary build for integration test of notification Due to the step reordering, the notification image does not have to be rebuilt for the integration test Co-authored-by: felix-oq <felix-oq@users.noreply.github.com>
* Implement CDCT for interactions between UI and Storage * Replace the type null by undefined in StorageItemView * Adjust storage consumer test in UI to adapt previous type changes
* Implement CDCT for interactions between UI and Storage * Implement CDCT for interactions between Storage and Pipeline * Fix amqp consumer initialization Previously the amqp consumer was configured with an incorrect topic. * Refactor provider test to handle different consumers in separate tests
This improves build times when the source code changes but the dependencies stay the same
* Added import-trigger amqp to scheduler + refactored tests
* Refactored integration tests for scheduler datasource trigger amqp
* - Added amqp consumer logic to adapter
- replaced import trigger REST api with amqp
* Refactored adapter integration tests to match the new import trigger api using amqp
* Added new services to ALL_SERVICES and SERVICES_TO_KILL
* Re-added datasource import trigger endpoint
* - Re-added old REST tests under new (second) describe
- Renamed AMQP tests header to no longer include POST {endpoint}
- Increased test timeout for AMQP tests
* Increased sleep delay until amqp consumer check due to switch from REST to AMQP
* Increased delay due to outboxer not getting called in pipeline, but locally works without issues
* Testing whether pipeline has timing issue
* Added debug prints
* Fixed prettier in test
* Added debug prints
* Last delay increase test
* Increased test timeout for problematic test
* Added import-trigger amqp to scheduler + refactored tests
* Refactored integration tests for scheduler datasource trigger amqp
* - Added amqp consumer logic to adapter
- replaced import trigger REST api with amqp
* Refactored adapter integration tests to match the new import trigger api using amqp
* Added new services to ALL_SERVICES and SERVICES_TO_KILL
* Re-added datasource import trigger endpoint
* - Re-added old REST tests under new (second) describe
- Renamed AMQP tests header to no longer include POST {endpoint}
- Increased test timeout for AMQP tests
* Increased sleep delay until amqp consumer check due to switch from REST to AMQP
* Increased delay due to outboxer not getting called in pipeline, but locally works without issues
* Testing whether pipeline has timing issue
* Added debug prints
* Fixed prettier in test
* Added debug prints
* Last delay increase test
* Increased test timeout for problematic test
* Removed debug logs
* Fixed the bug by added key triggered update and v-if * PipelineStorageView: use null instead of undefined for unloaded element => avoids error Co-authored-by: Georg Schwarz <schwarz@group.riehle.org>
* Fixed REST endpoint to expect pipeline array * Refactored loadPipelinesByDatasourceId to - expect an array - update the corresponding elements inside the state attribute 'pipeline' - added getter for retrieving pipelines by datasourceId from state * Adjusted and enabled pact tests for getPipelinesByDatasourceId * Fixed pact tests for getByDatasourceId: - in case of non found: switch to 200 + empty array - in case of NaN: switch to testing getAll (due to BE impl of getByDatasourceId) * Fixed getter naming * Small pact fixes * Removed unused VUEX getter * Refactored getAll endpoint to return 400 in case of datasourceId is NaN * Fixed pact tests to reflect 400 on datasource is NaN Co-authored-by: Georg Schwarz <schwarz@group.riehle.org>
* Remove unnecessary timestamp attribute * Cover optional schema attribute in CDC-Tests It is covered in interactions between Notification & Pipeline and UI & Pipeline The skipped interactions between UI & Pipeline are not adjusted, this will be done in a future commit * Fix lint error * Remove unused return value when updating and deleting pipelines * Fix lint error * Cover health status enum values in CDC-Tests * Let CDCT script remove previously existing containers This change should reduce the flakiness of the provider-side CDC-Tests of the pipeline service. Old entries in the database and old messages in the message queue are avoided that way. * Remove unnecessary timestamp attribute in provider-side CDC-Test * Cover schema attribute for previously skipped interactions * Improve script so it determines the project folder name * Remove unnecessary methods for state handling
* Fix update and delete interactions to handle absent ids Previously, the provider-side mocking did not mimic the actual behaviour for update and delete correctly. * Fix mocking in unit test of pipeline config manager * Fix integration test to handle changed interactions * Fix system test to handle changed interactions * Change status code to 404 for deleting absent pipelines
* Added POST endpoint for manual datasource trigger * - Added new action button to manually trigger datasource - Added catch to loadStatus in order to prevent leaving loop just because one has no data (-> returns 404 -> error)
#397) * Retention period to 1 day for all uploaded unpublished container artifacts * Fix typo
* Pipeline: Fix error position parsing in VM2 * Pipeline: upgrade vm2 version
* Adapt welcome message and app name * Datasource: overview adapt to pipeline overview * Pipeline: action items to icons with tooltip * Streamline datasource create and edit view * Pipeline: remove pipeline state and refactor edit into form and create components * Storage: adapt to changes in routing and presentation * UI: undo changes to REST classes and instantiate in views instead
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.