RESUMABLE: Retrieving the lost final response after a completed upload

[guoye-zhang]

The draft currently states that

The server SHOULD keep the upload resource available for a reasonable amount of time after the upload is complete.

However, it does not define a mechanism for retrieving the final response. Interacting with the temporary response would result in a failure with the "completed-upload" problem type.

For a well-behaving client, if the connection is lost before a response is received, it would attempt to perform offset retrieval followed by upload appending. If the server resends the final response for a 0-length complete append, the upload would complete successfully with no additional effort on the client side. If the server is unwilling to keep the final response for resending, I'm not sure if there is a value in keeping the temporary resource available, either.

We can eliminate the "completed-upload" problem type altogether if we allow this.

[Acconut]

Would you suggest that if an upload resource persists after the upload is complete, it should always resend the final response after completion? Should the server cache the response similar to implementations using idempotency keys to prevent duplicate requests? Or would that just be an implementation detail?

[guoye-zhang]

If the server keeps the upload resource alive, it should replay the final response when processing a 0-length append at the final offset. It doesn't need to process the request again, so I don't think it needs idempotency key since the URL is already uniquely identifying the upload.

[Acconut]

I'm open to that idea but would like to hear what others think as well.

[mertalev]

I think caching is an implementation detail. Ultimately, the important thing is that the response is the same. For instance, my server can make the same response based on DB metadata without explicit caching.

Personally, I don't see a need for an idempotency key either, but I'm unsure if there are any situations where it'd be useful.

[smatsson]

Yes please, I'm in favor of this. The "last response lost" problem has been a recurring issue in the tus protocol and if we can solve it here I would be happy. It seems like the last piece of the puzzle for from my point of view.

I agree with others that there is probably no need for idempotency keys unless anyone else can come up with a reason? Also agree that the exact implementation details (caching or reprocessing) should be left up to the server.

[awwright]

I think the proposal makes sense: Partial uploads should be idempotent, so if the server receives the final segment of the upload—even if it’s zero length—the server should be permitted to re-send the same response.

However, this expands the scope a little bit, and still wouldn't cover all cases where a response might be lost.

First, if recovery of responses is a feature, it's quite a useful one, so I think the introduction text should mention this.

And second, if the document is going to address lost responses, the client can also lose the "upload resumption supported" URL. The server may receive the full request and process it, without the client knowing if any part was received at all.

Solving that problem would require some sort of client-generated request identifier (Idempotency-Key); or (if the request content is non-empty) the use of “Expect: 100-continue” to ensure that the server does not commit any changes until the client acknowledges it has received a resumption URL.

Overall, how comprehensive should resumable uploads be in recovering from lost responses?

[guoye-zhang]

I agree that we do not need to solve all cases of lost responses. The draft currently recommends that the temporary resource being kept alive after upload finishes in case the final response is lost, but it's not super useful since there is no way to retrieve the final response, and interacting with the temporary resource always leads to 409.

If we think this is not in scope of the protocol, the server should just invalidate the temporary resource as soon as the upload completes.

[danielresnick]

+1 to supporting retrieval of final response via a zero-byte PATCH with Upload-Complete: ?1. I think this should probably be SHOULD rather than MUST though for server implementors, and they should be free to decide how to implement as long as it's idempotent (e.g. save final response from first "finalization" and replay it, vs re-process the request and achieve the same outcome). It should be expected that the upload resource can still disappear after a period of time.

[Acconut]

After discussing this at the IETF 124, I think we go with this route and make the final, 0-length PATCH request idempotent. It should generate the same response, although the language has to agnostic of any implementation details.

We can eliminate the "completed-upload" problem type altogether if we allow this.

Yes, but we then might want to have a problem type for when the client sends a PATCH request with Upload-Complete: ?0 after the server already received a PATCH request with Upload-Complete: ?1.