Move PSD back to process #1507
Open
Move PSD back to process #1507
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While investigating compressing the preprocessing data in #1348, I found that the PSDs in the saved preprocess axis managers take up the vast majority of the space (~71 MB each for SAT ISO v3 data). We moved the PSD calculations into
calc_and_savein #1041 to improve preprocessing's performance, but this seems like it is probably not worth it especially since we now running things in large parallel jobs and are more concerned about space. This should make the uncompressed files go down from 250 MB for one obs_id, wafer, and band to something like 40 MB.Tested by running preprocessing with SATp3 ISO v3 init config and loading it back. Will break existing preprocessing configuration files.