Skip to content

Improve support for larger commits#66

Merged
abannachGrafana merged 3 commits into
grafana:mainfrom
max-frank:improve-file-contents-handling
Apr 14, 2025
Merged

Improve support for larger commits#66
abannachGrafana merged 3 commits into
grafana:mainfrom
max-frank:improve-file-contents-handling

Conversation

@max-frank
Copy link
Copy Markdown
Contributor

@max-frank max-frank commented Mar 29, 2025

WHAT

Change graphql call to read file contents from the file system instead of passing it in directly as part of the CLI option.

WHY

GitHub workflows templating has a limitation on the amount of memory that is available. When passing in the file contents as part of the CLI option via GitHub workflows templating this limit is hit relatively quickly, if cumulative file content gets bigger than a few KB.

This template memory error can be seen in 3 tests of the below workflow run. Note that the modified multiple files test only writes a total of 5MB and already fails.

Additionally the shell also has its own argument size limit, which is quickly hit as well.
Both of these limits can be avoided by populating the file contents field values from the file system instead.

Note

A third limit exists which is the maximum commit size that the GitHub GraphQL API can handle. This limit seems to be at 40MB (40MiB will already fail). So the cumulative size of all files modified, added or renamed must be below this limit for commits created via the GraphQL API.

BEGIN_COMMIT_OVERRIDE
feat: improve support for larger commits
END_COMMIT_OVERRIDE

@max-frank max-frank force-pushed the improve-file-contents-handling branch from 6701b05 to 0cf1708 Compare March 29, 2025 11:24
@max-frank max-frank marked this pull request as ready for review March 29, 2025 11:26
@max-frank
Copy link
Copy Markdown
Contributor Author

@abannachGrafana sorry for not filing an issue first, but the fix kinda happened naturally during analysis so I just created the PR directly 🙇

@max-frank max-frank force-pushed the improve-file-contents-handling branch from 0cf1708 to 919fcc2 Compare March 29, 2025 13:14
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (2)

action.yml:183

  • Verify that the output 'contents_dir' from the additions-and-deletions step correctly matches the temporary directory created earlier; any inconsistency may result in temporary files not being properly cleaned up.
rm -rf "${{ steps.additions-and-deletions.outputs.contents_dir }}"

.github/workflows/test.yml:217

  • [nitpick] Consider including the --tmpdir flag when using mktemp to specify the directory explicitly, ensuring consistency with other temporary file creations in the workflow.
file_content="$(mktemp temp-random-file-XXXXXX)"

@abannachGrafana
Copy link
Copy Markdown
Contributor

Thanks for the contribution! Sorry I was out on PTO and just saw your PR. I'm not sure why the tests are failing at checking out your branch. I need to do a little investigation as to why that is. It's happening on #67 as well.

@abannachGrafana
Copy link
Copy Markdown
Contributor

@max-frank I've merged #68 to hopefully resolve the test run issue. Please update your branch and we'll see if the tests run successfully.

@max-frank max-frank force-pushed the improve-file-contents-handling branch from 919fcc2 to a71e8f9 Compare April 9, 2025 13:07
@max-frank
Copy link
Copy Markdown
Contributor Author

max-frank commented Apr 9, 2025

@max-frank I've merged #68 to hopefully resolve the test run issue. Please update your branch and we'll see if the tests run successfully.

@abannachGrafana i've rebased on top of the latest main

the new tests still have the ref explicitly set but I can update that once its confirmed the fix in #68 is working as intended

@abannachGrafana
Copy link
Copy Markdown
Contributor

Good news, problem one solved! Onto problem two!

Copy link
Copy Markdown
Contributor

@abannachGrafana abannachGrafana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Just two cleanup things on the test checkouts.

Comment thread .github/workflows/test.yml Outdated
Comment on lines +260 to +261
with:
ref: ${{ github.head_ref || github.ref }}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
with:
ref: ${{ github.head_ref || github.ref }}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update to

with:
          ref: ${{ env.CHECKOUT_REF }}

same as all the other test workflows right now

Comment thread .github/workflows/test.yml Outdated
Comment on lines +311 to +312
with:
ref: ${{ github.head_ref || github.ref }}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
with:
ref: ${{ github.head_ref || github.ref }}

@abannachGrafana
Copy link
Copy Markdown
Contributor

abannachGrafana commented Apr 10, 2025

You'll wanna pull up from main again too. The tests won't run on the PR, but will run once merged to main. I'm curious if you're able to run the tests in your fork as well. That could be an alternative process to reviewing the tests and hoping for the best.

@max-frank
Copy link
Copy Markdown
Contributor Author

You'll wanna pull up from main again too. The tests won't run on the PR, but will run once merged to main. I'm curious if you're able to run the tests in your fork as well. That could be an alternative process to reviewing the tests and hoping for the best.

Sorry for the late reply will update and do the changes now

For the tests in my fork yeah while creating the PR I also opened it against my own fork to test the workflows

Passing file contents directly as CLI arg can result in template
memory issues with larger files
@max-frank max-frank force-pushed the improve-file-contents-handling branch from a71e8f9 to 72aa6ed Compare April 14, 2025 08:10
@abannachGrafana abannachGrafana merged commit cae2afb into grafana:main Apr 14, 2025
3 checks passed
@abannachGrafana
Copy link
Copy Markdown
Contributor

@max-frank thanks for the contribution. Tests pass on main. 👍

For the tests in my fork yeah while creating the PR I also opened it against my own fork to test the workflows

This might be the best all around solution. Bit more manual on the review side of things, but it'll avoid issues where the tests fail on main after reviewing. On a positive note, I'm hoping that there won't be much need for a ton of prs on this repo as it has one job. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants