feat: ingest logic to use FileService - BED-8317,BED-8318

[mykeelium]

Description

This change migrates the ingestion logic to use the new FileService. In addition, the cleanup daemon was updated to be able to delete files using the new FileService.

Motivation and Context

Resolves BED-8317,BED-8318

How Has This Been Tested?

The application was started and multiple ingested files were created. The data from these files were ingested correctly. In addition new tests were added.

For the cleanup, files were added to the directories, and both with exemptions and without PruneData was run to ensure the proper files were deleted.

Screenshots (optional):

Types of changes

• Chore (a change that does not modify the application functionality)
• New feature (non-breaking change which adds functionality)

Checklist:

• I have met the contributing prerequisites
  • Assigned myself to this PR
  • Added the appropriate labels
  • Associated an issue: Issues and Pull Requests README #672
  • Read the Contributing guide: https://github.com/SpecterOps/BloodHound/wiki/Contributing
• I have ensured that related documentation is up-to-date
  • Open API docs
  • Code comments (GoDocs / JSDocs)
• I have followed proper test practices
  • Added/updated tests to cover my changes
  • All new and existing tests passed

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: b5ee1b4e-e000-4674-94c7-33cef970ce67

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch mcuomo/BED-8317

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[wes-mil]

I am a little confused why we're doing direct OS file operations here, doesn't file service have a local storage driver?

[mykeelium]

It does, but this is supposed to be a stage earlier than the fileService, in this case spooling is done to take the users file and validate it prior to saving it to the file service. In the event that we have a file service that is not on the os, this handles the resources ensuring they are available for the fileService. Not in this scope, but maybe the upload could be shifted to a presigned url so we do not need this step at all.