Feature: Invalidate caches when a put is done to a version

[stanislav-zaprudskiy]

The idea behind the following pipeline is that each subsequent job updates the body of the same pre-release until the final job does its update and makes that pre-release to become a normal release:

pipeline.yaml

resources:
- name: github-pre-release
  type: github-release
  source:
    owner: foo
    repository: bar
    access_token: ((token))
    release: false
    pre_release: true

- name: github-release
  type: github-release
  source:
    owner: foo
    repository: bar
    access_token: ((token))
    release: true
    pre_release: false

jobs:
- name: job1
  serial: true
  plan:
  - get: github-pre-release
    trigger: true

  - task: update-release-body
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: alpine
      inputs:
      - name: github-pre-release
      outputs:
      - name: github-pre-release
      run:
        path: /bin/sh
        args:
          - -c
          - |
            echo "this is an update from job1" >> github-pre-release/body

  - put: github-pre-release
    inputs:
    - github-pre-release
    params:
      name:      github-pre-release/tag
      tag:       github-pre-release/tag
      commitish: github-pre-release/commit_sha
      body:      github-pre-release/body

- name: job2
  serial: true
  plan:
  - get: github-pre-release
    trigger: true
    passed:
    - job1

  - task: update-release-body
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: alpine
      inputs:
      - name: github-pre-release
      outputs:
      - name: github-pre-release
      run:
        path: /bin/sh
        args:
          - -c
          - |
            echo "this is an update from job2" >> github-pre-release/body

  - put: github-pre-release
    inputs:
    - github-pre-release
    params:
      name:      github-pre-release/tag
      tag:       github-pre-release/tag
      commitish: github-pre-release/commit_sha
      body:      github-pre-release/body

- name: job3
  serial: true
  plan:
  - get: github-pre-release
    trigger: true
    passed:
    - job2

  - task: update-release-body
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: alpine
      inputs:
      - name: github-pre-release
      outputs:
      - name: github-pre-release
      run:
        path: /bin/sh
        args:
          - -c
          - |
            echo "this is an update from job3" >> github-pre-release/body

  - put: github-pre-release
    inputs:
    - github-pre-release
    params:
      name:      github-pre-release/tag
      tag:       github-pre-release/tag
      commitish: github-pre-release/commit_sha
      body:      github-pre-release/body

- name: job4
  serial: true
  plan:
  - get: github-pre-release
    trigger: true
    passed:
    - job3

  - task: update-release-body
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: alpine
      inputs:
      - name: github-pre-release
      outputs:
      - name: github-pre-release
      run:
        path: /bin/sh
        args:
          - -c
          - |
            echo "this is a final update from job4" >> github-pre-release/body

  - put: github-release
    inputs:
    - github-release
    params:
      name:      github-pre-release/tag
      tag:       github-pre-release/tag
      commitish: github-pre-release/commit_sha
      body:      github-pre-release/body

However, it doesn't work as expected. The subsequent jobs don't get the updated body of the release most of the time (depending on cache accessibility/worker used, afaics), which results into a body like this:

this is an update from job2
this is a final update from job4

(in various combinations, but never a complete one from all 4 jobs).

I believe it matches the explanation from #61 (comment):

This is working as expected of the caching semantics. When a resource version changes in-place, there's no way for Concourse to know that that same version is technically different now. I don't think it's an issue specific to this resource, but you could open a general issue to invalidate caches when a put is done to a version; that may be a route we can explore.

And is also similar to #69.

Would storing more release metadata in Concourse solve the problem? E.g. not only the id, tag and timestamp, but also body, commit_sha, etc. Or perhaps some extra timestamp/hash value of the result of the last update of a resource by Concourse's put.

Concourse version: 7.6.0, resource version: bundled with Concourse (v1.6.4).

[stanislav-zaprudskiy]

I also tried adding more passed dependencies for the subsequent jobs, but that doesn't make any difference

diff

@@ -89,12 +89,13 @@
 - name: job3
   serial: true
   plan:
   - get: github-pre-release
     trigger: true
     passed:
+    - job1
     - job2

   - task: update-release-body
     config:
       platform: linux
       image_resource:
@@ -124,12 +125,14 @@
 - name: job4
   serial: true
   plan:
   - get: github-pre-release
     trigger: true
     passed:
+    - job1
+    - job2
     - job3

   - task: update-release-body
     config:
       platform: linux
       image_resource:

[taylorsilva]

More version metadata wouldn't help this. More fields on the version emitted that make the updated version distinct would resolve this issue. Or updating the timestamp field maybe?

github-release-resource/resources.go

Lines 90 to 94 in 3a0793a

	type Version struct {
	Tag string `json:"tag,omitempty"`
	ID string `json:"id"`
	Timestamp time.Time `json:"timestamp"`
	}

[taylorsilva]

This issue is actually better suited for the concourse/concourse repo since the resource cache isn't something any individual resource controls.

Resources can influence whether the cache is hit by making each version distinct. That's why I left a comment saying the timestamp field could be updated to make the version unique from Concourse's POV, therefore skipping any resource cache.