worker: Add indexing dashboard in development#4189
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc9fd377b9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| typeof message === 'string' && | ||
| message.startsWith('progress|') | ||
| ) { | ||
| let payload = message.substring('progress|'.length); | ||
| let progressEvent = JSON.parse(payload) as IndexingProgressEvent; |
There was a problem hiding this comment.
In ECS deployments the dashboard routes are intentionally not exposed (if (!ECS_CONTAINER_METADATA_URI) above), but this branch still parses and stores every progress|... IPC payload for all workers. Because each indexing run now sends one event with the full file list plus one event per visited file, production workers incur avoidable JSON/IPC and memory overhead with no consumer, which can noticeably slow or bloat large indexing runs.
Useful? React with 👍 / 👎.
| current.#onProgress?.({ | ||
| type: 'indexing-finished', | ||
| realmURL: current.realmURL.href, | ||
| jobId: current.#jobInfo.jobId, | ||
| stats: current.stats, |
There was a problem hiding this comment.
The new progress flow emits indexing-started before work begins, but indexing-finished is only emitted on the success path here. If indexing throws (for example any non-404 visit error rethrown by tryToVisit), the sink never gets a terminal event, so the dashboard can show that realm as permanently active until another job overwrites it.
Useful? React with 👍 / 👎.
1 participant
I’m often wondering when developing how much is left to index:
This adds an event sink for index runners to report on their progress and a worker manager dashboard to report that.