Add support for multiple cartridge sources

[danechitoaie]

Right now there seems to be support fo just one place where cartridges are loaded and so the config json looks something like:

{
  "hostname": "1234",
  "username": "user",
  "password": "1234",
  "version": "dev",
  "cartridges": "cartridges"
}

In some cases we have multiple cartridges that we need to take from multiple repos cloned in different places so it would be nice if there would be support for this.

What I propose is to also add support for "cartridges" to be an array.

So if typeof config.cartridges === "string" then logic is as before and you take cartridges from a single place.

If it's not string then you iterate over the array and watch/clean cartridges from all sources.

{
  "hostname": "1234",
  "username": "user",
  "password": "1234",
  "version": "dev",
  "cartridges": ["/path/to/cartridges1", "/path/to/cartridges/2"]
}

[MobiusHorizons]

I will look into it, although it will probably have to be implemented separately for each component, so it may take a while.

I agree this feature would be a great addition here. Maybe we could contribute with a PR to help here.

[MobiusHorizons]

I would definitely be interested in accepting pull requests. I would start by updating clean to work with multiple directories, since that would be easiest. watch will take slightly more work.

[osapishchuk]

plus one to this feature

[CodrutTapu]

This feature will be nice to have.

[Jakobud]

I have been using Demandware/SFCC's own dwupload tool. You simply specify an array of cartridges in your dw.json file. Does dw-util not support this? I just started messing with it so I'm not clear on that limitation or not.

[danechitoaie]

@Jakobud It does, but in a different way.

dw-utils requires you to specify a folder where your cartridges are and it will watch for changes and upload all cartridges in that folder.

dwupload requires you to list each cartridge you want uploaded from current folder where dw.json file is.

This issue is about dw-utils supporting to watch/upload cartridges from multiple sources (for cases when you keep some cartridges in one GIT repo, other in other GIT repo, and so on. Feature that dwupload also does not have.

[Jakobud]

Okay that makes sense now. So if the option supported multiple relative/absolute paths to cartridges, how would that work with the clean and bulk upload functionality? Would it loop through all the cartridge paths and zip them together for a bulk upload or what?

[danechitoaie]

Would it loop through all the cartridge paths and zip them together for a bulk upload or what?
Yes, that's what I would expect. So if the config.cartridges is an array (we also support it as as string as it's now so we have backwards compatibility - if having backwards compatibility is important):

• For watch, we watch all cartridges inside the sources provided in the array and just push to remote instance as we do now. Ideally with the option to ignore some paths using the glob syntax ( https://github.com/isaacs/node-glob ).
• For clean we do the same, we iterate over all sources and add cartridges we find there. Ideally we would push each cartridge as an individual zip instead of zipping all cartridges in a single huge zip as we have cases when the resulted zip is larger than 100MB and it can't be pushed to DW (due to this archaic 100MB limitation). So if cartridges are zipped and pushed individually this could be a workaround to bypass this DW limitation.

And we can also add some checks before watch and clean like to see if same cartridge exists in two places and give an error to the user. This covers cases when you keep your code in multiple GIT repos and somehow you have a cartridge with same name in two places. If we just upload it we will get unexpected side effects. So just a simple error like "Error! Cartridge app_x already exists in path/to/source1".

All the uses cases I covered above (code version bigger than 100MB, or same cartridge in multiple places) might seem strange but I encountered these issues in the past so they happen. Having support for this edge cases would make the tool work nicely for every possible use case.

[Jakobud]

Ok I'm already in the process of adding in the ignore option for the watch functionality. I'll PR when I got it working. Running into issues with the underlying watch library chokidar.

[Jakobud]

Okay I got a branch done here with ignore implementation. I could use any feedback.

https://github.com/Jakobud/dw-utils/tree/feature/watch-ignore-patterns

This accepts either a --ignore pattern or patterns listed in a standalone file: --ignore-from option. I might pull out the ignore pattern file option though, since you can essentially do the same thing by specifying an array of patterns in the dw.json anyways.

Let me know what you guys think.