download with package name as folder

[patrickwolf]

we are using this to download packages
ascli aoc packages recv ALL --once-only=yes

but some vendors place their files into the root folder and some into subfolders
if files are in root folders all the packages get mixed up in the root folder which makes the automatic download not possible

would be very important for us that each package gets its own folder with the name of the package being the folder name

thanks @laurent-martin !

[laurent-martin]

using the package name as folder can have these problems:

• a package name is not unique
• a package name can contain characters not allowed by the operating system in file names (e.g. : on Windows, or /)

If the feature is to have one folder per package , then the package identifier as folder name ensures unicity of folder.

[patrickwolf]

hey @laurent-martin our pipeline requires each package name to be unique (#204) so unless there is an error all package already have a unique name.

so what would be great for us is the following.

ascli only prepends the package id if the folder already exists and is for a different package (ie a name clash). this way our packages could keep their original name without having a package id in each folder name..

the feature could become quite complex but this might make it easier

• if the listing of packages in the inbox shows that the name was used more then one time automatically append the package id (also helps alter us there was duplication)
• if there is no duplication in the listing just download it without package id

this behavior would result in

• downloads/WOS_SUB_250703_001 - first package (has a clean folder name as its the first one)
• downloads/WOS_SUB_250703_001 (45897uadf) - second package with this name has the package id
• downloads/WOS_SUB_250703_001 (fasd322f) - third package with this name

in our case 99% we'll only have the first package ie clean folder names and every time there is a package id we know it needs manual ingestion or rejection

[patrickwolf]

@laurent-martin just wanted to check-in if there has been progress on this one? thank you!

[laurent-martin]

Hi,
The last beta here: https://ibm.biz/aspera-cli-beta
implements a new option: package_folder with default value id, so that by default each package is downloaded in a folder named with the package id.

See: https://github.com/IBM/aspera-cli/tree/develop?tab=readme-ov-file#receive-packages

package id is unique and has only letters and numbers, but package name may be the same for 2 packages, and can cause problems like: too long, non-permitted characters (/, etc..)
Do you think that solves the issue ?

[laurent-martin]

Delivered option package_folder, et to name if you want that the folder has the name of package. But I recommend using id instead.

[patrickwolf]

Do you think that solves the issue ?

hey @laurent-martin no it does not solve the problem unfortunately. the option i outlined above with name_id would solve it though (especially if name[_id] would keep id optional and only use it if name leads to duplication)

but package name may be the same for 2 packages, and can cause problems like: too long, non-permitted characters (/, etc..)

customers who pay for the aspera services do so so it takes care of issues (speed, reliability, security etc.)

so it makes sense that ascli takes care of automatically sanitizing package names so they work in the file system (ie replace [/\ ] with [_] etc.) without customers needing to worry about it. this can happen on the AoC level (see idea1 at IBM and idea2 ) but since that is a bigger change it makes sense for it to start happen on the ascli level for now.

it's really a single regex that would turn a package name into a compatible file system name so my request is that its added please 🙏 (even though i do understand that the code needs to be able track both AoCpackageName and fileSystemPackageName independently and that that might create more work)

and the name_id and name[_id] would be very helpful too as just id does not help (not human readable) and just name does not work (duplication).

Thank you @laurent-martin

[laurent-martin]

Yes, the next version actually addresses that point.

[patrickwolf]

Great. Can you please confirm how its going to be addressed @laurent-martin ?

Are you going to support name[_id] ie the id is optional and only used if the name is duplicated?

The VFX team is not going to accept having an ID in every folder (as our package name already has a strict naming convention) which means without the id being optional (see solution above) we will not be able to use the feature.

Thanks a lot!

[laurent-martin]

It could be several solutions:
1- add id if folder already exists
2- add an incremental number, like .1, .2 , etc...
3- let the user provide a Proc i.e. some small ruby code that allow customization: this one is flexible: maybe this is too complex.

for 1 and 2, one can imagine a syntax to tell ascli to add that extension only when the folder already exists.

For example:
--package-folder=name+id? for 1
--package-folder=name+inc? for 2

[laurent-martin]

See beta here:
https://ibm.biz/aspera-cli-beta
Doc here:
https://github.com/IBM/aspera-cli/tree/develop?tab=readme-ov-file#receive-packages
and here:
https://github.com/IBM/aspera-cli/tree/develop?tab=readme-ov-file#invalid-filename-characters