Full Async Force Rescan #1038

emanuelGitCodes · 2025-06-19T18:39:34Z

✨ Pull Request: Asynchronous Scanning Pipeline & Issue-Persistence Overhaul

“From our very first Supervisor queue to a fully-parallel rescan workflow.”

📋 Summary

This PR rolls up every change made since we introduced Symfony Messenger workers and Supervisor—culminating in a brand-new, end-to-end pipeline that refreshes LMS data, scans content items in parallel, and reliably persists Equal Access issues.

🗺️ Architecture at a Glance

Layer	New Components	Notes
Messaging	ScanContentItem, FullRescanMessage, FinishRescanMessage	Clear unit-of-work messages; supports batching.
Handlers	FullRescanHandler, ScanContentItemHandler, FinishRescanHandler	Orchestrate refresh → scan → aggregate.
Services	LocalApiAccessibilityService, major updates to LmsFetchService, EqualAccessService, CanvasLms, ScannerService	Async HTTP, rate-limit back-off, generic-issue path, report math, etc.
Entities / DB	New Issue creation paths (createIssue, createGenericIssue)	Handles both PhpAllyIssue and generic Equal Access DTOs.

✅ How to Test

Full Rescan → queue FullRescanMessage; observe workers process items in parallel and FinishRescanHandler mark scan ✔️.
Database → verify issue rows populate for each content item.
Canvas API Throttle → throttle Canvas to 1 req/sec; worker should self-sleep without 429s.

📝 Deployment Notes

# supervisorctl 
supervisorctl status

# New ENV
SCAN_POOL_SIZE=6

Removed the disabled locks on the menu tabs to allow user to click to report once scan started.

API now calls the aysnc refresh function instead of normal. Adds aysnc function which removes steps and calls correct function to start async scanning

- Gets course content asynchronously and then triggers scan as content is finished downloading Co-authored-by: Emanuel <emanuelcodes@gmail.com>

In the update course content in CanvasLMS it previously was creating new item, but also updating it later in code. Durning creation added the rest of the fields needed to avoid duplicate assignments.

Adds local scanner support and adds more print statements to test output

Adds to gitignore the scanner files that are downloaded from another repo. Removes usage of AuthToken in functions and passing it through etc.

…ng. This files are not complete, there more like blueprints to what to do.

…rker configuration

Co-authored-by: Emanuel <emanuelcodes@gmail.com>

Adds messenger system to queue scans. Currently queues print statement. Messenger system starts workers and runs concurrently.

Fixes issue with logging not appearing by routing log output to normal stdout and stderr

Updates the messenger to scan the content item.

Adds wait code to wait until all the message workers have completed before finishing the report.

Adds delete function to the MessageHandler instead. Adds new separatate async function to be called by the message to avoid editing current one. Updates config and adds more detaches and flushes to try and fix database saving

…nges Add Local Scanner Fixes To Branch

Re-adds routing back to the async there. Removes random return in the async refresh.

Removed equal access server to help future merges. Uncommeted the ace container since removed the files for it.

…do async method.

Base for supervisors

…ync-process-upgrade

…to dispatch full rescan jobs async. Force Full Rescan works and scans everything, but it runs twice. First runs scans everything, second is redundant.

…ontent scanning - Consolidate messenger worker configurations into a single directory. - Introduce new messenger handlers for concurrent and high-priority content scanning. - Update `ScanContentItem` to include content item ID and user ID. - Modify `FullRescanHandler` to dispatch `ScanContentItem` messages for updated items. - Adjust `LmsFetchService` to return updated content items for processing. Working towards having the `Force Full Rescan` process to be divided between different workers to make it faster.

…umber of browsers available on the local scanner.

…logging in rescan processes.

…h processing

…code in handlers

…sing; mark items as pending and dispatch FinishRescanMessage when all items are processed.

…redis Have to confirm all issues are being downloaded

…tent item count, and improve ScanContentItemHandler to persist issues. Update LocalApiAccessibilityService and ScannerService for better data handling.

Refactor ScannerService to include document parsing in report generation for content items.

…figurations

…onent

…ent, enhance error handling in aceCheck, update scan route for better error reporting, and improve logging in LocalApiAccessibilityService and ScannerService.

…lean up error handling

…ApiAccessibilityService

…treamline error handling, and enhance logging in LocalApiAccessibilityService and ScannerService.

…redis

…lRescanHandler

emanuelGitCodes · 2025-07-17T14:18:44Z

Updated the new env variables to

###> Equal Access Local Num Browsers ###
###> One thread per browser instance, so this is the number of browsers to run in parallel.
ASYNC_POOL_SIZE=2

##> Threads focus on the start and end of the scan, so this is the number of scans to run in parallel.
ASYNC_HIGH_PRIORITY_POOL_SIZE=2

…dler to utilize it

…redis

emanuelGitCodes and others added 30 commits February 4, 2025 08:23

Create Async version of updateCoursesContent

44a3f51

Removed Disabled Locks on Menu Tabs

d5eb23b

Removed the disabled locks on the menu tabs to allow user to click to report once scan started.

Adds Aysnc Calls From API

5e26e86

API now calls the aysnc refresh function instead of normal. Adds aysnc function which removes steps and calls correct function to start async scanning

Add Async Downloading Files & Scanning

eb26161

- Gets course content asynchronously and then triggers scan as content is finished downloading Co-authored-by: Emanuel <emanuelcodes@gmail.com>

Removed Update for New Items To Creation

9d30133

In the update course content in CanvasLMS it previously was creating new item, but also updating it later in code. Durning creation added the rest of the fields needed to avoid duplicate assignments.

Adds Local Scanner & Print Output

06ba0e1

Adds local scanner support and adds more print statements to test output

Temp Work on Concurrent Scanning

5d2a6bc

Removes AuthToken Passing & Ignores Scanner Files

6dc3f65

Adds to gitignore the scanner files that are downloaded from another repo. Removes usage of AuthToken in functions and passing it through etc.

Add FullRescanMessage and FullRescanHandler for asynchronous processi…

0ad4efb

…ng. This files are not complete, there more like blueprints to what to do.

Enhance Docker setup with Supervisor for PHP-FPM and add messenger wo…

52328e7

…rker configuration

Add Supervisor Configurations for Messenger

313a391

Co-authored-by: Emanuel <emanuelcodes@gmail.com>

Adds Messenger System

588e77d

Adds messenger system to queue scans. Currently queues print statement. Messenger system starts workers and runs concurrently.

local scan fix

258bbac

Fixes Logging Disappearing Issue

a995bae

Fixes issue with logging not appearing by routing log output to normal stdout and stderr

local scan levels

54de3ec

Implement Scanning Function for Messenger

2058614

Updates the messenger to scan the content item.

fix default lvl

f61bfb0

Adds Wait For All Scans to Complete

e7c310c

Adds wait code to wait until all the message workers have completed before finishing the report.

Printing to terminal back on

8b03a75

Merge branch 'async-equal-access' into async-equal-access-evan-changes

1f290a3

Separate Scan Function and Enhancements

30fc9ec

Adds delete function to the MessageHandler instead. Adds new separatate async function to be called by the message to avoid editing current one. Updates config and adds more detaches and flushes to try and fix database saving

Merge pull request #1 from SavvyDolphin77/async-equal-access-evan-cha…

f67a49e

…nges Add Local Scanner Fixes To Branch

Fixes Issues with Async

33c0013

Re-adds routing back to the async there. Removes random return in the async refresh.

Ignore equal-access-server directory from branch

dc70049

Removed Equal Access Server To help Merge

ee0ad8d

Removed equal access server to help future merges. Uncommeted the ace container since removed the files for it.

Focus on making updateCourseContent() faster. Currently applying a su…

da4b63c

…do async method.

Merge branch 'async-process-upgrade' into equal-access

7888128

Merge pull request #1025 from emanuelGitCodes/equal-access

5caf183

Base for supervisors

Merge remote-tracking branch 'conner-fork/async-equal-access' into as…

dc197f7

…ync-process-upgrade

Implement FullRescanMessage and its handler; refactor SyncController …

7532023

…to dispatch full rescan jobs async. Force Full Rescan works and scans everything, but it runs twice. First runs scans everything, second is redundant.

emanuelGitCodes added 9 commits May 19, 2025 14:56

Create env variable so the number of workers running is the same as n…

22b30d4

…umber of browsers available on the local scanner.

Add SCAN_POOL_SIZE variable for equal access local configuration

17afc0f

Enhance rate limit handling for when running out of tokens and error …

fc52f3a

…logging in rescan processes.

Add FinishRescanMessage and handler; enhance ScanContentItem for batc…

f09ab07

…h processing

Update FullRescanMessage transport to high priority; clean up unused …

c10a164

…code in handlers

Enhance FullRescanHandler and ScanContentItemHandler for batch proces…

5e48ea0

…sing; mark items as pending and dispatch FinishRescanMessage when all items are processed.

Merge remote-tracking branch 'origin/new-main-page-UX' into async-no-…

b5e21f4

…redis Have to confirm all issues are being downloaded

Refactor PagePool logging, enhance FinishRescanHandler to include con…

cd55a32

…tent item count, and improve ScanContentItemHandler to persist issues. Update LocalApiAccessibilityService and ScannerService for better data handling.

emanuelGitCodes added enhancement php Pull requests that update Php code review labels Jun 19, 2025

emanuelGitCodes added 12 commits June 23, 2025 12:53

On completing full rescan the summary page is updated.

4469dab

Refactor ScannerService to include document parsing in report generation for content items.

The Homepage refreshes on completion of a full rescan.

6224788

On full rescan completing, the frontend refreshes with the new changes.

7836bea

Update env variable names for async processing and adjust related con…

86d8a85

…figurations

Add sync report endpoint and update report fetching logic in App comp…

6259ac1

…onent

Add rebuild target to Makefile, adjust polling interval in App compon…

6390179

…ent, enhance error handling in aceCheck, update scan route for better error reporting, and improve logging in LocalApiAccessibilityService and ScannerService.

Refactor aceCheck to use SCAN_TIMEOUT_MS for setContent timeout and c…

0dd62f3

…lean up error handling

Add supervisor command to Makefile and remove debug output from Local…

16e7a22

…ApiAccessibilityService

Refactor aceCheck to use SCAN_TIMEOUT_MS for timeout configuration, s…

332b6d1

…treamline error handling, and enhance logging in LocalApiAccessibilityService and ScannerService.

Merge remote-tracking branch 'origin/new-main-page-UX' into async-no-…

27f9607

…redis

Create a new Function call fullRefreshLmsContent which is used by Ful…

81238d3

…lRescanHandler

Mistake when solving merge conflicts and miss this file.

7222673

emanuelGitCodes and others added 2 commits July 17, 2025 13:31

Add force rescan option to FullRescanMessage and update FullRescanHan…

9433394

…dler to utilize it

Merge remote-tracking branch 'origin/new-main-page-UX' into async-no-…

87a7736

…redis

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Full Async Force Rescan #1038

Full Async Force Rescan #1038

Uh oh!

emanuelGitCodes commented Jun 19, 2025

Uh oh!

emanuelGitCodes commented Jul 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Full Async Force Rescan #1038

Are you sure you want to change the base?

Full Async Force Rescan #1038

Uh oh!

Conversation

emanuelGitCodes commented Jun 19, 2025

✨ Pull Request: Asynchronous Scanning Pipeline & Issue-Persistence Overhaul

📋 Summary

🗺️ Architecture at a Glance

✅ How to Test

📝 Deployment Notes

Uh oh!

emanuelGitCodes commented Jul 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants