Feature request webDAV: support multilevel directory creation via MKCOL method

[airnandez]

According to RFC 4918, an implementation of the MKCOL method must not automatically create a directory hierarchy:

if a request to create collection /a/b/c/d/ is made, and /a/b/c/ does not exist, the request must fail.

Although dCache's implementation of MKCOL is compliant to the RFC, from the client perspective it would be advantageous to send a single MKCOL request to the server for creating a directory hierarchy. We noticed that the Rubin image processing pipelines create deep hierarchies and, as a result, can induce significant load on the dCache webDAV door.

I wonder if you would consider extending dCache to support this behavior as other implementations of webDAV already do. If necessary, the client could include some dCache-specific header to let the door know that it is requesting a behavior not part of the official protocol.

Note that dCache implementation of webDAV PUT automatically creates all the ancestor directories when serving the request, even if this is not compliant to RFC 4918 PUT which says:

A PUT that would result in the creation of a resource without an appropriately scoped parent collection must fail with a 409 (Conflict).

This feature is very useful and I am not asking any modification about that. I only mention this to illustrate a very useful extension to the official RFC already supported by dCache as well as other implementations.

cc: @ageorget

[paulmillar]

Hi @airnandez ,

Although I'm spending less time working on dCache these days, perhaps I can add some context here that might be useful.

dCache's WebDAV support comes from an open-source library called Milton. For the most part, its the Milton code that parses the WebDAV requests and provides a lot of the behaviour. We've implemented the storage-system abstractions (that Milton expects). These interface with the rest of the dCache system to provide the behaviour that Milton expects.

The behaviour you describe (RFC-compliant MKCOL and non-RFC-compliant PUT) comes from the Milton library, rather than from dCache itself.

At certain places in the Milton code base, I've seen comments describing how non-RFC behaviour is required (in Milton) in order to be compatible with other servers. In other words, some another server (e.g., Microsoft) exhibits some non-RFC-compatible behaviour and some clients (also from Microsoft) depend on that behaviour. Therefore, in order to be compatible, Milton must also implement that behaviour.

I haven't seen any specific comments in Milton on why, when handling a PUT request, it auto-creates parent directories. The code to do this is involved, so I believe it is intentional. To be honest, I'm rather surprised with RFC 4918, as the WebDAV PUT semantics seems to break HTTP PUT semantics (although honouring filesystem-like behaviour), which is odd given the WebDAV DELETE semantics follows HTTP DELETE semantics (which doesn't honour filesystem-like behaviour).

Again, just for context, the FTS service (from CERN, used to orchestrate site-to-site data transfers), when transferring a file will walk up the directory tree (on the destination) to discover an existing ancestor directory and then walks back down using MKCOL to create any missing directories. This works, but (as you observe) there is the potential for many requests.

You mentioned creating high load on the WebDAV door. At the risk of saying something obvious, have you checked whether the client is maintaining the TCP connection over these operations? Making many HTTP requests (as required to create parent directories) could be slow if the network connection has a high RTT (i.e., client is far away), but I wouldn't expect significant load on the door itself, as it's mostly parsing text and submitting messages to the rest of dCache. However, if the client is establishing a TLS-protected TCP connection for each HTTP request then I could imagine seeing high load on the door. If so, one solution could be to update the client so it issues all HTTP requests over the same TCP connection.

In terms of changing the behaviour of MKCOL, yes, of course this would technically be possible: it's all just code, after all.

However, I would have some concerns about doing so:

• Introducing code that modifies Milton behaviour and that would need to be maintained.
• Non-standard behaviour: how should clients know the server supports this? how would other servers provide compatible implementations?
• Limit benefits: only helping clients that know about this "trick". RFC-compliant clients cannot take advantage of the improvement.
• The semantics isn't clear; for example, should auto-created parent directories have the same ownership as the MKCOL target?
• Moving away from RFC-compliant behaviour risks breaking RFC-compliant clients.

So, although it's possible, I wouldn't say that it's necessarily the best way to solve your problem.

If you do want to fix MKCOL behaviour with "mkdir -p"-like semantics (something that I think could have merit) then I'd suggest first coming up with a spec on how this should work, getting feedback and endorsement from different people (server developers, client developers). This feedback might reveal the reasons why RFC 4918 has the stated MKCOL semantics.

That said, @kofemann may see this differently.

[airnandez]

Hi @paulmillar, thanks for this thoughtful response. I now appreciate that implementing this feature may not be obvious given how dCache's implementation of webDAV is architected.

Regarding your questions about the clients, I confirm that the client keeps its TLS connection with the door open. The issue we observed is that when 10k simultaneous clients need to create 4 to 5 level-deep directories, the clients have to send several MKCOL and HEAD (or PROPFIND) requests to the door.

To mitigate this issue, we modified the client to send a PUT request to create a temporary empty file and get the directory hierarchy created as a "side effect" and then DELETE the file. This means a constant cost of two requests for creation of a directory hierarchy of any depth. This is effective even if not very clean.

Before opening this issue we looked at the possibility of using dCache' frontend API but we noticed that it does not support creation of a directory hierarchy in a single request either. Such a request results in HTTP/1.1 400 Bad Request with response body:

{
   "detail" : "Name must not contain '/'.",
   "title" : "Bad Request",
   "status" : "400"
}

We could adapt our client to use the frontend API instead of the webDAV API for this particular operation if it was supported. I wonder if that would be a more acceptable path forward from your perspective.

[DmitryLitvintsev]

IMO, we could deviate from RFCs as long as there is a switch to disable it. So per site admin discretion and understanding of the risks. This is, in fact, how GFTP was handled to support overwrite if target exists. Despite RFC this was allowed provided admin enabled the behavior.

[paulmillar]

Hi @airnandez,

The PUT/DELETE on an empty file is a clever trick, although I agree it's unsatisfactory.

A few thoughts:

• Do you need to create empty directories? Otherwise, wouldn't simply uploading the files (issuing PUT requests for each file) create the desired directory structure without requiring MKCOL requests? This should work because of dCache's existing, non-RFC-conforming behaviour on PUT requests.
• As a general rule, we have frontend as a door to support dCache-specific (i.e., non standard) behaviour. It would, perhaps, make more sense to implement something like mkdir -p in that interface. You're right: this feature isn't available currently.
• Dmitry is right: we can just implement the behaviour in the WebDAV and add a dCache configuration property to control whether it is enabled. The admin can choose whether to support the optimisation.
• We could also use an HTTP request header to switch on the mkdir -p-like behaviour per request. By expecting mkdir -p-like semantics, the client is already requiring a non-RFC conforming server, so requiring an extra HTTP header isn't a "big ask". (IMHO, this would be a cleaner approach).

HTH,
Paul.

[airnandez]

Do you need to create empty directories?

In some situations yes, creating a directory hierarchy is a legitimate request. However, we are looking at improving our code to not explicitly call MKCOL when the only purpose of this request would be to prepare a directory to upload a file via PUT.

We could also use an HTTP request header to switch on the mkdir -p-like behaviour per request.

From the perspective of the client, it would not be an issue to add a header like X-dCache-extended-mkcol: True to the MKCOL request, to notify the dCache door that the client understands that it is requesting a non-standard behavior.