Skip to content

Feature/nrmi 244 update cleanup job #1221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
reemi1510 merged 3 commits into from
Nov 24, 2025
Merged

Feature/nrmi 244 update cleanup job #1221

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3cc5932
changed cleanup jobs default entry batch size and set to run only aft…
Nov 14, 2025
2be9e33
further update to prevent empty errors pages for unfinished tasks
Nov 18, 2025
0ef3b23
Merge branch 'develop' into feature/nrmi-244-update-cleanup-job
reemi1510 Nov 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions app/jobs/cleanup_submission_entries_job.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
class CleanupSubmissionEntriesJob < ApplicationJob
ENTRIES_BATCH_SIZE = 1000

# rubocop:disable Metrics/CyclomaticComplexity
# rubocop:disable Metrics/PerceivedComplexity
def perform(dry_run: false, max_run_time: 5.hours, task_batch_sixe: 100)
def perform(dry_run: false, max_run_time: 5.hours, task_batch_size: 100, entries_batch_size: 500)
start_time = Time.current
total_deleted_entries = 0

Expand All @@ -12,21 +10,23 @@ def perform(dry_run: false, max_run_time: 5.hours, task_batch_sixe: 100)
cleanup_processed: false })
.distinct

tasks_with_unprocessed_submissions.find_in_batches(batch_size: task_batch_sixe) do |tasks_batch|
tasks_with_unprocessed_submissions.find_in_batches(batch_size: task_batch_size) do |tasks_batch|
tasks_batch.each do |task|
active_id = task.active_submission&.id
latest_id = task.latest_submission&.id

failed_submissions = task.submissions
.where(aasm_state: 'validation_failed', cleanup_processed: false)
.where.not(id: active_id)
.where.not(id: latest_id)

next if failed_submissions.empty?

deleted_for_task = 0

failed_submissions.find_each do |submission|
begin
submission.entries.in_batches(of: ENTRIES_BATCH_SIZE) do |entries_batch|
submission.entries.in_batches(of: entries_batch_size) do |entries_batch|
if dry_run
Rollbar.info("Dry run: would delete #{entries_batch.count} entries for Submission ID #{submission.id}.")
else
Expand Down
1 change: 1 addition & 0 deletions app/jobs/data_warehouse_export_job.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
class DataWarehouseExportJob < ApplicationJob
def perform
DataWarehouseExport.generate!
CleanupSubmissionEntriesJob.perform_later
end
end
4 changes: 0 additions & 4 deletions config/sidekiq_schedule.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,3 @@ kill_stuck_submissions:
cron: '*/5 * * * *'
class: KillStuckSubmissionsJob
queue: default
cleanup_submission_entries:
cron: '0 1 * * *'
class: CleanupSubmissionEntriesJob
queue: default