Use GitHub's Large File Storage (LFS) for model files

[aufdenkampe]

GitHub has a 100 MB file size limit, but even including 10 MB binary files that frequently change can cause a Git repo to explode in size, because every change (including the entire new binary file) is stored in a database and needs to be synced.

This has limited our ability to have model files in the repo as part of the testing system.

An excellent solution is to use Git's Large File Storage (LFS) system, which points to files stored outside the repo but is transparent to the user.
More info: https://docs.github.com/en/github/managing-large-files/working-with-large-files

We need to do this as a first step toward building the automated testing system we've been discussing.

[aufdenkampe]

As a case study, let's try this on some of the large files that @bcous has in his branch: https://github.com/LimnoTech/HSPsquared/tree/develop-WaterQuality-BC.

Note that we don't want to merge this branch as it is, because this would introduce a large number of large files that would then forever be in the repo's database.

Rather, we want to manually incorporate these file links using LFS in a new branch (forked from develop).

Once this is completed, we then delete (not merge!) this branch with those very large files: https://github.com/LimnoTech/HSPsquared/tree/develop-WaterQuality-BC.

[ptomasula]

@aufdenkampe, I started looking into Git's LFS this morning and set up the feature_testing_lfs branch. I made a one general commit which consolidates the testing data into a single subdirectory. This will make it easier to LFS track test data since we can just track everything in that subdirectory, but also cleans up the test directory for the next phase of work when we actually start writing unit tests. I think unit tests should align one-to-one with the code modules so structuring the directory around the code rather than the testing data makes more sense to me.

From there I created a second feature branch (feature_testing_lfs_sandbox) so I can test out some of the LFS commands before actually taking the next steps on the branch that I intend to merge back into develop. I've been following the tutorial on LFS's github, and got to the step where I convert and upload existing files that are now lfs tracked. However, I received the error below when I tried to commit. The error suggests that we don't have space to upload these files, but I only tried this initial lfs conversion with the handful of wdm files (which doesn't even Branden's larger files yet). The wdm files according to the git lfs migrate info command total to about 38MB and I believe we get 1GB of storage for free. Maybe not though, I could be misinterpreting the information in my github account. It's also possible that storage limit applies to repositories associated with my account and since this is a LimnoTech repo that my storage limit doesn't apply. Anyway I think you and I will need to touch base and figure out the settings here, and perhaps purchase some data packs.

Uploading LFS objects: 0% (0/3), 0 B | 0 B/s, done. batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access. error: failed to push some refs to 'https://github.com/LimnoTech/HSPsquared.git'

[aufdenkampe]

@ptomasula, thanks for getting started on this! I just checked LimnoTech's account settings for Billing and Plans, and it looks like we should also get 1GB free. Hmm. I wonder if it might be shut down because of our legacy GitHub account?