-
Notifications
You must be signed in to change notification settings - Fork 17
Open
Labels
bugSomething isn't workingSomething isn't working
Description
In #198, I'm attempting to use multiple uris to collect assets under two different prefixes into the same collection
collections:
- id: cil-gdpcir-cc-by
template: ${{ local.path(./collection/cil-gdpcir-cc-by) }}
class: pctasks.dataset.collection:PremadeItemCollection
asset_storage:
- uri: blob://rhgeuwest/cil-gdpcir-stac
token: ${{ pc.get_token(rhgeuwest, cil-gdpcir-stac) }}
chunks:
options:
name_starts_with: CC-BY-4.0/
- uri: blob://rhgeuwest/cil-gdpcir-stac
token: ${{ pc.get_token(rhgeuwest, cil-gdpcir-stac) }}
chunks:
options:
name_starts_with: CC-BY-SA-4.0/
When I run this, I noticed that both output to the same file. Here are their outputs:
{
"uri": "blob://rhgeuwest/cil-gdpcir-etl-data/chunks/cc-by/2023-05-03-cc-by-fix/assets/all/rhgeuwest/cil-gdpcir-stac/0/uris-list.csv",
"chunk_id": "rhgeuwest/cil-gdpcir-stac/0/uris-list.csv"
}
and
{
"uri": "blob://rhgeuwest/cil-gdpcir-etl-data/chunks/cc-by/2023-05-03-cc-by-fix/assets/all/rhgeuwest/cil-gdpcir-stac/0/uris-list.csv",
"chunk_id": "rhgeuwest/cil-gdpcir-stac/0/uris-list.csv"
}
Perhaps we need to coordinate partition numbers among the two? Or add some extra level to the path; something like the index of the mapping under assset_storage?
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working