Unclear how to handle partially-shared filesystems when designing HTCondor executor

[jhiemstrawisc]

I was unsure of the most appropriate place to open this issue/discussion, so hopefully I've landed close to where I should have. If I should move the conversation somewhere else, let me know.

I've been developing on the HTCondor+Snakemake executor from this repo. In particular, I'd been working to get the executor functioning in HTCondor pools where there's no shared filesystem between the HTCondor Access Point (AP) and the pool's Execution Points (EPs). I succeeded at this by having the executor explicitly manage which files are transferred with HTCondor as well as setting Snakemake's --shared-fs-usage argument to none.

Now I'm in the process of getting the executor to work in environments where the AP shares only specific filesystem paths with EPs. For example, it might be the case that only files under /foo/bar are common to both environments. For any input/output not under this path, HTCondor needs to transfer the file from AP-->EP, but the AP does not need to manually transfer files when they are under the path. For now, these shared filesystem paths should only be used for input/output, and it's assumed the Snakemake process itself is never run under one of these paths.

I put together a proof of concept that handles the partially-shared filesystem case on this branch of my fork, but it currently introduces a very awkward configuration requirement that sets shared-fs-usage: none to prevent the local Snakemake process from trying to communicate directly with the remote process, and then further sets shared-fs-prefixes: /foo/bar so that the Executor knows which files it's not responsible for explicitly transferring. Even though I can demonstrate that this works end-to-end, I can't imagine it's the way I should be thinking about the design.

Is there a better way to plumb this logic through existing Snakemake executor options? I'm aware of the –default-storage-provider option, but it's unclear to me whether I can tap into that in a way that works with the overall setup.

[johanneskoester]

So, would the following make sense?
Extend snakemake_interface_executor_plugins.settings.SharedFSUsage to offer --shared-fs-usage path-dependent, and add another option --shared-fs-paths ... to snakemake.settings.types.StorageSettings that takes nargs of paths where the filesystem is shared? Then, Snakemake could dynamically decide which of the usual items of --shared-fs-usage (persistence, input-output, software-deployment, sources, storage-local-copies, source-cache) are actually shared depending on whether they are children of the given paths or not, upon startup.

Then, snakemake.workflow.Workflow can get a method is_on_shared_fs that takes a file path and returns True if the file is on a shared part of the filesystem. The htcondor plugin could then just transfer files where that method returns False.
It would have to check and do that for any job input and output files and for dag.source_files().

[jhiemstrawisc]

That sounds like a reasonable approach to move a lot of the half-baked logic out of my HTCondor executor and into Snakemake itself.

One other thing I'd like to point out is that the submit node and the execution point might share a partial filesystem, but they might be mounted at different paths. For example, the submit node might have /mnt/foo/my-directory while the execution point might have /mnt/bar/my-directory. Although this isn't super common, I know of at least one HTCondor cluster that does this. In HTCondor's submit claim language, this is usually referred to as a "transfer remap". That being said, this is trickier because of the way the HTCondor executor works -- it's not obvious to me how the job should know whether it's in the local or the remote environment.

[johanneskoester]

There is Workflow.exec_mode, which gives you ExecMode.DEFAULT for jobs that run locally (on the main process) and ExecMode.Remote for jobs that run within the remote executor (htcondor).

[jhiemstrawisc]

As an update, I recently merged in some of the changes I needed to get partially-shared filesystems working with the HTCondor executor. I don't love all my solutions and would welcome any suggestions to improve the user interface, which currently requires setting something along the lines of:

shared-fs-usage: none
htcondor-shared-fs-prefixes: "/foo,/bar"

Maybe I can see about attending the March Hackathon to contribute to some of this.