infinite upload loop with .nc file

[jsadler2]

When I upload a csv file everything is hunky dory. When I upload a nc file, scipiper goes into an infinite loop on the upload step. When I realized this I killed it and opened the SB item to find 8 copies of my nc files. One thing I noticed, when I was deleting them, was that they were under a separate tab. The csv file is under the "files" tab, the nc files were under the "extensions" tab. Here's my guess to what is happening: the task-table functions are looking for some indicator that a "file" was successfully uploaded and since the .nc files are going into the "extensions" bucket scipiper doesn't think they've been uploaded so it keeps uploading them.

[jordansread]

With the current/new version of sbtools, I added

  # Second, upload the file
  scrape_file <- tools::file_ext(filepath) %in% c('xml')
  item_replace_files(sb_id, files = filepath, scrape_files = scrape_file)

to the sb_utils.R file and was able to work around this. Perhaps we need to add that change to this template and also specify a version number for sbtools

[jordansread]

@limnoliver do you need this change ☝️ in the repo too for your data release?

[limnoliver]

Mine are zipped in a folder, so I think avoids this issue.