-
Notifications
You must be signed in to change notification settings - Fork 26
refactor: select all performance #107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feature/migration-logging-refactor
Are you sure you want to change the base?
refactor: select all performance #107
Conversation
…are/SwagMigrationAssistant into refactor/select-all-performance
src/Resources/app/administration/src/core/service/api/swag-migration.api.service.ts
Outdated
Show resolved
Hide resolved
...ion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal/index.ts
Show resolved
Hide resolved
...lution/swag-migration-error-resolution-modal/swag-migration-error-resolution-modal.html.twig
Outdated
Show resolved
Hide resolved
...ion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal/index.ts
Show resolved
Hide resolved
...ion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal/index.ts
Show resolved
Hide resolved
...ion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal/index.ts
Outdated
Show resolved
Hide resolved
...ion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal/index.ts
Outdated
Show resolved
Hide resolved
...ion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal/index.ts
Outdated
Show resolved
Hide resolved
...ion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal/index.ts
Show resolved
Hide resolved
...ion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal/index.ts
Outdated
Show resolved
Hide resolved
...tion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal.spec.js
Show resolved
Hide resolved
...tion/component/swag-migration-error-resolution/swag-migration-error-resolution-modal.spec.js
Show resolved
Hide resolved
…are/SwagMigrationAssistant into refactor/select-all-performance
…are/SwagMigrationAssistant into refactor/select-all-performance
| $entityName, | ||
| $fieldName, | ||
| !empty($connectionId) ? $connectionId : null | ||
| $request->request->getInt('limit', $this->maxLimit), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is still no validation on this query parameter at all. Where is it validated that it's greater than zero and less than or equals to $this->maxLimit?
My previous comment on this: #107 (comment)
| code: string, | ||
| entityName: string, | ||
| fieldName: string, | ||
| connectionId?: string, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no limit query parameter here? I'm fine with leaving it as null / undefined and letting the server decide but I think this API service should mirror the PHP endpoint and show all possible parameters
| this.migrationStore.connectionId, | ||
| ); | ||
|
|
||
| const iterations = Math.ceil(count / limit); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case thought, you are assuming a certain limit is used in the backend and should also give that to the HTTP call below to ensure its the same with your assumption
| for (let i = 0; i < iterations; i += 1) { | ||
| // each batch must be completed before fetching the next | ||
| // eslint-disable-next-line no-await-in-loop | ||
| const { entityIds } = await this.migrationApiService.getLogEntityIdsWithoutFix( | ||
| this.runId, | ||
| this.selectedLog.code, | ||
| this.selectedLog.entityName, | ||
| this.selectedLog.fieldName, | ||
| this.migrationStore.connectionId, | ||
| ); | ||
|
|
||
| const entities = entityIds.map((entityId: string) => this.createResolutionEntity(entityId)); | ||
|
|
||
| // each batch must be completed before fetching the next | ||
| // eslint-disable-next-line no-await-in-loop | ||
| await this.migrationFixRepository.saveAll(entities); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nitpick, I liked the while loop and It's also possible to do it without knowing any limit (you are relying on the server for that anyways) and with proper early break if the response is empty already for whatever reason, e.g. something like this
let offset = 0;
while (offset < count) {
// each batch must be completed before fetching the next
// eslint-disable-next-line no-await-in-loop
const { entityIds, total } = await this.migrationApiService.getLogEntityIdsWithoutFix(
this.runId,
this.selectedLog.code,
this.selectedLog.entityName,
this.selectedLog.fieldName,
this.migrationStore.connectionId,
// for the limit, maybe just let the server decide on a default?
);
if (entityIds.length === 0) break;
const entities = entityIds.map((entityId: string) => this.createResolutionEntity(entityId));
// each batch must be completed before fetching the next
// eslint-disable-next-line no-await-in-loop
await this.migrationFixRepository.saveAll(entities);
offset += entityIds.length;
}| // wait until all promisses in the loop are resolved | ||
| // 3 batches with 2 promises each = max 6 iterations + some more to be safe | ||
| let iterations = 10; | ||
| while (wrapper.vm.submitLoading && iterations > 0) { | ||
| // async calls created in a loop; we need to wait for them to be resolved | ||
| // eslint-disable-next-line no-await-in-loop | ||
| await flushPromises(); | ||
| iterations -= 1; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't like this, and I'm wondering if this is needed at all or if we can find a cleaner solution.
Regarding my previous two comments on this topic, I'm sorry but my suggestion was wrong and I though about playwright, it's not easily possible to wait for an element to disappear in Jest in a clean way.
I guessed a single call to flushPromises should be fine but now, I'm not sure if it might only be flaky 🤔
sidenote: flushPromises is defined as follows:
https://github.com/vuejs/test-utils/blob/d7e8a3b35b3e4a8079a9c44d15a6cd7372b1dc37/src/utils/flushPromises.ts#L1-L8
And we likely don't recommend not awaiting in a loop for a reason 😅.
So I looked a bit deeper into this with a little experiment:
https://jsfiddle.net/o9087wty/3/
But I guess you have to do it with a real Jest and vue test environment, because in that tests flushPromises didn't wait for anything 🤔
TLDR: what happens if you just call await flushPromises here once?
1. Why is this change necessary?
Previously, clicking "Select All" fetched all log Id's from the database. For migrations with thousands of errors, this could lead to performance issues.
2. What does this change do, exactly?
before: "select-all" fetches all swag_migration_logging.id and stores them in
selectedLogIdsnow: "select-all" tracks a
selectAllModestate and just visually selects all log-rows + disables checkbox ("all-or-nothing-principle") = no data is fetchedfetching all unresolved entity ids in submission process: fetches and saves fixes in batches (limit is Shopware default API limit value = 500)
split submission logic into
submitResolutionInBatches()>selectAllModeand
submitResolutionForSelectedIds()> unchanged submission flowcloses: shopware/shopware#13672