Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 156 additions & 0 deletions docs/ai-contribution-policy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
# AI/LLM tool policy

## Summary

The **stac-utils** policy is that contributors can use whatever tools they would
like to craft their contributions, but **there must be a human in the loop**.
Contributors must read and review all AI (Artificial Intelligence) / Large
Language Model (LLM)-generated code or text before they ask other project
members to review it. The contributor is always the author and is fully
accountable for their contributions.

## Warning

It is not currently a settled question within the **stac-utils** community
whether code created by LLMs is acceptable in our codebases at all. This
document only addresses the interaction aspects of LLM content, and should not
be viewed as the project taking a position on whether or not LLM code
contributions are acceptable.

## Policy

The **stac-utils** policy is that contributors can use whatever tools they would
like to craft their contributions, but there must be a **human in the loop**.
Contributors must read and review all AI (Artificial Intelligence) / Large
Language Model (LLM)-generated code or text before they ask other project
members to review it. The contributor is always the author and is fully
accountable for their contributions. Contributors should be sufficiently
confident that the contribution is high enough quality that asking for a review
is a good use of scarce maintainer time, and they should be **able to answer
questions about their work** during review.

We expect that new contributors will be less confident in their contributions,
and our guidance to them is to **start with small contributions** that they can
fully understand to build confidence. We aspire to be a welcoming community that
helps new contributors grow their expertise, but learning involves taking small
steps, getting feedback, and iterating. Passing maintainer feedback to an LLM
doesn't help anyone grow, and does not sustain our community.

Contributors **must be transparent and label contributions that contain
substantial amounts of tool-generated content**, and always mention it. The pull
request and issue templates contain a checkbox for that purpose. Failure to do
so, or lies when asked by a reviewer, will be considered as a violation. Our
policy on labeling is intended to facilitate reviews, and not to track which
parts of **stac-utils** repos are generated. Contributors should note tool usage
in their pull request description, commit message, or wherever authorship is
normally indicated for the work. For instance, use a commit message trailer like
Assisted-by: \<name of code assistant\>. This transparency helps the community
develop best practices and understand the role of these new tools.
Comment on lines +39 to +48
Copy link
Copy Markdown

@jonhealy1 jonhealy1 Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There could be better guidance here. We should have a specific approach to citing ai use. The advice here is a little vague:

"Contributors should note tool usage in their pull request description, commit message, or wherever authorship is normally indicated for the work. For instance, use a commit message trailer like
Assisted-by: <name of code assistant>."

Also, this mentions that, regarding ai use, "The pull request and issue templates contain a checkbox ". Will we enforce that across stac-utils? It's not a bad idea, but if we aren't doing it, we should remove the text referencing the approach.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels pretty specific to me:

"Contributors should note tool usage in their pull request description, commit message, or wherever authorship is normally indicated for the work. For instance, use a commit message trailer like Assisted-by: ."

@jonhealy1 Can you provide an example of language that you feel would be less vague?

"The pull request and issue templates contain a checkbox ". Will we enforce that across stac-utils? It's not a bad idea, but if we aren't doing it, we should remove the text referencing the approach.

We can create an org-wide pull request template that will be the default for repositories that don't have one. Repositories that do have a template will need to update it to include the checkbox ... I can update the repositories I maintain if/when this PR is merged, and I can post in the CNG Slack channel and Github Discussions to notify other maintainers of the change. Thanks for calling this out, I've updated the PR description to include the additional task.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about something like:

Checklist

  • I have tested these changes locally.
  • I have included / updated tests for these changes.
  • Edge Cases: I have manually verified "unhappy paths" and edge cases beyond the basic success criteria.
  • I can explain the implementation logic for every line of code submitted.

AI Disclosure

  • AI-Assisted: Some or all of this contribution was generated or refined using AI tools. I have reviewed, tested, and take full responsibility for the quality and correctness of the output.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me a lot of the danger with ai relates to testing. An ai can cleverly write a dozen tests for you and make sure that they all pass, but they may only write tests that they know will pass. Someone thinks - there's a dozen new tests here, this must cover everything! Having a human making sure that everything important - ie all of the edge cases - is being tested is key.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The checklist looks pretty good...could you open a PR on https://github.com/stac-utils/.github to add it? The only bullet I'd add would be one about documentation. And can you use the language from the GDAL PR template for the tool usage disclosure?

## AI tool usage
 - [ ] AI (Copilot or something similar) supported my development of this PR. See our [policy about AI tool use](https://stac-utils.github.io/ai-contribution-policy). Use of AI tools *must* be indicated.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would honestly prefer something like this - do we really need to know if someone used ai or not? I think as long as they understand the "human-in-the-loop" concept and have read through our ai policy.

[ ] AI Disclosure: I have read the AI Contribution Policy and confirm a "human-in-the-loop" approach. I have reviewed all tool-generated content and take full accountability for this contribution.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can have 2 versions available - I would definitely want to use the second version.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can have 2 versions available - I would definitely want to use the second version.

I think if you're maintaining a package, it's up to you how you'd like to accept contributions. My intent behind creating this stac-utils policy was to provide a useful default, not proscribe how all stac-utils projects have to be maintained.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know - I guess I was thinking more along the lines of having 2 defaults. We definitely would like to point everyone towards at least being aware of the ai policy/ recommendations.


This policy includes, but is not limited to, the following kinds of
contributions:

- Code, usually in the form of a pull request
- RFCs or design proposals
- Issue or security vulnerability reporting
- Comments and feedback on pull requests
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would support prohibiting the use of AI in comments.

This comment was marked as off-topic.

Copy link
Copy Markdown

@jonhealy1 jonhealy1 Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docstrings? Not sure how they should be viewed?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation is mentioned in the section below:

As another example, using an LLM to generate documentation, which a contributor manually reviews for correctness and relevance, edits, and then posts as a PR, is an approved use of tools under this policy.


## Details

To ensure sufficient self review and understanding of the work, it is strongly
recommended that contributors write PR descriptions themselves (if needed, using
tools for translation or copy-editing), in particular to avoid over-verbose
descriptions that LLMs are prone to generate. The description should explain the
motivation, implementation approach, expected impact, and any open questions or
uncertainties to the same extent as a contribution made without tool assistance.

An important implication of this policy is that it bans agents that take action
in our digital spaces without human approval, such as the GitHub `@claude`
agent. Similarly, automated review tools that publish comments without human
review are not allowed. However, an opt-in review tool that **keeps a human in
the loop** is acceptable under this policy. As another example, using an LLM to
generate documentation, which a contributor manually reviews for correctness and
relevance, edits, and then posts as a PR, is an approved use of tools under this
policy.

## Extractive Contributions

The reason for our "human-in-the-loop" contribution policy is that processing
patches, PRs, RFCs, comments, issues, security alerts to **stac-utils**
repositories is not free -- it takes a lot of maintainer time and energy to
review those contributions! Sending the unreviewed output of an LLM to open
source project maintainers *extracts* work from them in the form of design and
code review, so we call this kind of contribution an "extractive contribution".

Our **golden rule** is that a contribution should be worth more to the project
than the time it takes to review it. These ideas are captured by this quote from
the book [Working in Public](https://press.stripe.com/working-in-public) by
Nadia Eghbal:

> When attention is being appropriated, producers need to weigh the costs and
> benefits of the transaction. To assess whether the appropriation of attention
> is net-positive, it's useful to distinguish between *extractive* and
> *non-extractive* contributions. Extractive contributions are those where the
> marginal cost of reviewing and merging that contribution is greater than the
> marginal benefit to the project's producers. In the case of a code
> contribution, it might be a pull request that's too complex or unwieldy to
> review, given the potential upside.
>
> -- Nadia Eghbal

Prior to the advent of LLMs, open source project maintainers would often review
any and all changes sent to the project simply because posting a change for
review was a sign of interest from a potential long-term contributor. While new
tools enable more development, it shifts effort from the implementor to the
reviewer, and our policy exists to ensure that we value and do not squander
maintainer time.

## Handling Violations

If a maintainer judges that a contribution doesn't comply with this policy,
they should paste the following response to request changes:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guidance maybe on how maintainers can judge whether ai-generated code has violated these policies.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's some in the sections above, e.g.:

it is strongly recommended that contributors write PR descriptions themselves (if needed, using tools for translation or copy-editing), in particular to avoid over-verbose descriptions that LLMs are prone to generate

And

Contributors must be transparent and label contributions that contain substantial amounts of tool-generated content, and always mention it.

In my experience, LLM-generated code tends to have patterns that can be detected by a reviewer, and per these guidelines failing to mention that substantial amounts of tool-generated content without mention that it was tool-generated would be a violation.

If the provided language is not enough, @jonhealy1 could you maybe provide examples of the language you'd like to see?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are a very experienced programmer - I would be interested to know more about the patterns you see, as guidance for helping improve the projects I work on.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some common ones that I've observed:

  • Re-implementing instead of importing, aka copying code from Github rather than adding a new dependency to a project
  • Little "explainer" comments inside functions
  • Over-refactoring into small (1-4 line) functions that are only used once or twice
  • Calling functions in a unconventional way. This happens in geospatial a lot, where there's a "common" way of doing something (e.g. using core libraries like rasterio or shapely a certain way) but the code does it some other way. This isn't always bad, but sometimes can lead to strange breakages (there's some nomenclature in geospatial software that I've found LLMs don't understand correctly, e.g. stuff like buffer)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be fair, I have written code I could have imported, long before I started using ai, because I didn't understand that another library I wasn't familiar with already provided the same functionality. Explainer comments and over-refactoring are easy things to tell an ai not to do - add in a few other watch out fors, and then maybe we have no idea.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyways we should publish a document of things to watch out for/ things we don't like to see.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add in a few other watch out fors, and then maybe we have no idea.

At a certain point, we're relying on the good faith of contributors — open source is a collaborative environment where we're working together to build cool stuff. The intent of this contribution policy (to me) is to help people contribute in a positive way, not to discourage contributions.

The biggest problems I've seen with LLM-generated code w.r.t. open source libraries have been:

  • Trying to do too much at once. Small, focused PRs are so much easier to review
  • The contributor not understanding the code they're proposing
  • Expecting a quick review — even if you generated the code quickly, it can take time to review (especially on an un-funded project)

Copy link
Copy Markdown

@jonhealy1 jonhealy1 Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small, focused PRs are also a sign of experience - I always get carried away on prs - oh I'll just fix this while I'm at it. Expecting a quick review can be similar - someone may think they've done something really valuable, but then there pr can sit there forever even. An experienced programmer can be more patient, they can fork the project if they need to or install a branch in their codebase - an inexperienced programmer may not know that there are other options.

I don't know if you use ai, but I think that no matter how good ai gets, it's only going to work as good as the developer using it.

```text
This PR does not appear to comply with our policy on tool-generated content,
and requires additional justification for why it is valuable enough to the
project for us to review it. Please see our developer policy on
AI-generated contributions:
https://stac-utils.github.io/ai-contribution-policy
```

The best ways to make a change less extractive and more valuable are to reduce
its size or complexity or to increase its usefulness to the community. These
factors are impossible to weigh objectively, and our project policy leaves this
determination up to the maintainers of the project, i.e. those who are doing
the work of sustaining the project.

If/or when it becomes clear that a GitHub issue or PR is off-track and not
moving in the right direction, maintainers should apply the `extractive` label
to help other reviewers prioritize their review time.

If a contributor fails to make their change meaningfully less extractive,
maintainers may lock the conversation and/or close the pull request/issue/RFC.
In case of repeated violations of our policy, the **stac-utils** project in
question reserves itself the right to ban temporarily or definitely the
infringing person/account.

## Copyright

Artificial intelligence systems raise many questions around copyright that have
yet to be answered. Our policy on AI tools is similar to our copyright policy:
Contributors are responsible for ensuring that they have the right to
contribute code under the terms of our license, typically meaning that either
they, their employer, or their collaborators hold the copyright. Using AI tools
to regenerate copyrighted material does not remove the copyright, and
contributors are responsible for ensuring that such material does not appear in
their contributions. Contributions found to violate this policy will be removed
just like any other offending contribution. If a reviewer has doubts about the
legal aspects of a contribution, they may ask the contributor to provide more
details on the origins of a particular piece of code.

## Credits for this document

This document is a direct adaptation from the [GDAL AI/LLM tool policy](https://gdal.org/en/stable/community/ai_tool_policy.html), which itself was a quasi direct adaptation of
[LLVM software "AI Tool Use Policy"](https://github.com/llvm/llvm-project/blob/main/llvm/docs/AIToolPolicy.md),
and due credits go to its original authors: Reid Kleckner, Hubert Tong and
"maflcko"