Add an LLM policy for rust-lang/rust

[jyn514]

Rendered

View all comments

FCP link

Summary

This document establishes a policy for how LLMs can be used when contributing to rust-lang/rust. Subtrees, submodules, and dependencies from crates.io are not in scope. Other repositories in the rust-lang organization are not in scope.

This policy is intended to live in Forge as a living document, not as a dead RFC. It will be linked from CONTRIBUTING.md in rust-lang/rust as well as from the rustc- and std-dev-guides.

Ethical issues

See this thread.

Moderation guidelines

This PR is preceded by an enormous amount of discussion on Zulip. Almost every conceivable angle has been discussed to death; there have been upwards of 3000 messages, not even counting discussion on GitHub. We initially doubted whether we could reach consensus at all.

Therefore, we ask to bound the scope of this PR specifically to the policy itself. In particular, we mark several topics as out of scope below. We still consider these topics to be important, we simply do not believe this is the right place to discuss them.

So, the following are considered off topic for this PR specifically:

• Long-term social or economic impact of LLMs
• The environmental impact of LLMs
• Anything to do with the copyright status of LLM output. We have gotten initial confirmation from US lawyers that LLM output likely doesn't cause issues under US law. We still need to confirm this with EU lawyers and in other parts of the world.
• Moral judgements about people who use LLMs

We have asked the moderation team to help us enforce these rules. For an extended rationale, please see this comment.

Feedback guidelines

We are aware that parts of this policy will make some people very unhappy. As you are reading, we ask you to consider the following.

• Can you think of a concrete improvement to the policy that addresses your concern? Consider:
  • Whether your change will make the policy harder to moderate
  • Whether your change will make it harder to come to a consensus
• Does your concern need to be addressed before merging or can it be addressed in a follow-up?
  • Keep in mind the cost of not creating a policy.

If your concern is for yourself or for your team

• What are the specific parts of your workflow that will be disrupted?
  • In particular we are only interested in workflows involving rust-lang/rust. Other repositories are not affected by this policy and are therefore not in scope.
• Can you live with the disruption? Is it worth blocking the policy over?

---

Previous versions of this document were discussed on Zulip, and we have made edits in responses to suggestions there.

Motivation

• Many people find LLM-generated code and writing deeply unpleasant to read or review.
• Many people find LLMs to be a significant aid to learning and discovery.
• rust-lang/rust is currently dealing with a deluge of low-effort "slop" PRs primarily authored by LLMs.
  • Having a policy makes these easier to moderate, without having to take every single instance on a case-by-case basis.

This policy is not intended as a debate over whether LLMs are a good or bad idea, nor over the long-term impact of LLMs. It is only intended to set out the future policy of rust-lang/rust itself.

Drawbacks

• This bans some valid usages of LLMs. We intentionally err on the side of banning too much rather than too little in order to make the policy easy to understand and moderate.
• This intentionally does not address the moral, social, and environmental impacts of LLMs. These topics have been extensively discussed on Zulip without reaching consensus, but this policy is relevant regardless of the outcome of these discussions.
• This intentionally does not attempt to set a project-wide policy. We have attempted to come to a consensus for upwards of a month without significant progress. We are cutting our losses so we can have something rather than adhoc moderation decisions.
• This intentionally does not apply to subtrees of rust-lang/rust. We don't have the same moderation issues there, so we don't have time pressure to set a policy in the same way.

Rationale and alternatives

• We could create a project-wide policy, rather than scoping it to rust-lang/rust. This has the advantage that everyone knows what the policy is everywhere, and that it's easy to make things part of the mono-repo at a later date. It has the disadvantage that we think it is nigh-impossible to get everyone to agree. There are also reasons for teams to have different policies; for example, the standard for correctness is much higher within the compiler than within Clippy.
• We could have different standards for people in the Rust project than for new contributors. That would make moderation much easier, and allow us to experiment with additional LLM use. However, it reinforces existing power structures, creates more of a gap between authors and reviewers, and feels "unfriendly" to new contributors.
• We could have a more lenient policy that allows "responsible and appropriate" use of LLMs. This raises the question of what "responsible and appropriate" means. The usual suggestion is "self-review, and judging the change by the same standard as any other change"; but this neglects the reputational and social harm of work that "feels" LLM generated. It also makes our moderation policy much harder to understand, and increases the likelihood of re-litigating each moderation decision.
• We could have a more strict policy that removes the threshold of originality condition. This has the advantage that our policy becomes easier to moderate and understand. It has the disadvantage that it becomes easy for people to intend to follow the policy, but be put in a position where their only choices are to either discard the PR altogether, rewrite it from scratch, or tell "white lies" about whether an LLM was involved.
• We could have a more strict policy that bans LLMs altogether. It seems unlikely we will be able to agree on this, and we believe attempting it will cause many people to leave the project.
• We could have no policy at all. This avoids banning valid use cases; avoids implicitly legitimizing the use of LLMs by setting a policy; and saves all of us a great deal of time and effort. However, it greatly increases the baseline level of distrust within the project; makes review assignment even more of a dice roll than it is already; wastes a great deal of contributor and moderator time dealing with LLM-authored PRs on a case-by-case basis; and gives no guidance for people who really do want to use an LLM in a responsible way.

Prior art

This prior art section is taken almost entirely from Jane Lusby's summary of her research, although we have taken the liberty of moving the Rust project's prior art to the top. We thank her for her help.

Rust

• Moderation team's spam policy
• Compiler team's "burdensome PRs" policy

Other organizations

These are organized along a spectrum of AI friendliness, where top is least friendly, and bottom is most friendly.

• full ban
  • postmarketOS - also explicitly bans encouraging others to use AI for solving problems related to postmarketOS - multi point ethics based rational with citations included
  • zig
    • philosophical, cites Profession (novella)
    • rooted in concerns around the construction and origins of original thought
  • servo
    • more pragmatic, directly lists concerns around ai, fairly concise
  • qemu
    • pragmatic, focuses on copyright and licensing concerns
    • explicitly allows AI for exploring api, debugging, and other non generative assistance, other policies do not explicitly ban this or mention it in any way
  • forgejo
    • bans AI for review, code, documentation, and communication
    • mentions "legal uncertainties" as a motivating factor
    • explicitly excludes machine translation
• allowed with supervision, human is ultimately responsible
  • scipy
    • strict attribution policy including name of model
  • llvm
  • blender
  • linux kernel
    • quite concise but otherwise seems the same as many in this category
  • mesa
    • framed as a contribution policy not an AI policy, AI is listed as a tool that can be used but emphasizes same requirements that author must understand the code they contribute, seems to leave room for partial understanding from new contributors.

      Understand the code you write at least well enough to be able to explain why your changes are beneficial to the project.

  • firefox
  • ghostty
    • pro-AI but views "bad users" as the source of issues with it and the only reason for what ghostty considers a "strict AI policy"
  • fedora
    • clearly inspired and is cited by many of the above, but is definitely framed more pro-ai than the derived policies tend to be
• curl
  • does not explicitly require humans understand contributions, otherwise policy is similar to above policies
• linux foundation
  • encourages usage, focuses on legal liability, mentions that tooling exists to help automate managing legal liability, does not mention specific tools
• In progress
  • NixOS
    • How do we deal with AI-generated issues? NixOS/nixpkgs#410741

Unresolved questions

See the "Moderation guidelines" and "Drawbacks" section for a list of topics that are out of scope.

[rustbot]

r? @jieyouxu

rustbot has assigned @jieyouxu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Fallback group: @Mark-Simulacrum, internal-sites
• @Mark-Simulacrum, internal-sites expanded to Mark-Simulacrum, Urgau, ehuss, jieyouxu
• Random selection from Mark-Simulacrum, Urgau, ehuss, jieyouxu

[jyn514]

@rustbot label T-libs T-compiler T-rustdoc T-bootstrap

[jieyouxu]

I really like this version, and thanks a ton for working on it. Specifically:

• It doesn't try to dump entire walls of text, which is unfortunately a good way to be sure nobody reads it. Instead, it gives you concrete examples, and a guiding rule-of-thumb for uncovered scenarios, and acknowledges upfront that it surely cannot be exhaustive.
• I also like where it points out the nuance and recognizes the uncertainties.
• I like that it covers both "producers" and "consumers" (with nuance that reviewers can also technically use LLMs in ways that are frustrating to the PR authors!)

I left a few suggestions / nits, but even without them this is still a very good start IMO.

(Will not leave an explicit approval until we establish wider consensus, which likely will take the form of 4-team joint FCP.)

View changes since this review

[ChayimFriedman2]

The links to Zulip are project-private, FWIW.

[jyn514]

The links to Zulip are project-private, FWIW.

I'm aware. This PR is targeted towards Rust project members moreso than the broad community.

[davidtwco]

I'm happy with this as an initial policy for the rust-lang/rust repository.

View changes since this review

[apiraino]

Ok, despite all good intentions we had to lock this conversation again.

To all those coming here and point out that this policy does not consider ethics, environmental and other related issues: we hear you. We know. We totally understand. But here again the reasoning for not including these topics here at this time. This is not negotiable.

Please take a few moments to read and fully understand that comment and consider its implications.

[traviscross]

One of the stated goals of this document is to:

Make the policy enforceable and easy to moderate.

But how do we plan to enforce the "be honest" section? How are we to judge whether someone's use of an LLM exceeded what was disclosed?

That seems an important detail. This document doesn't talk anywhere about standards of proof. Other CoC items are enforceable on visible behavior. This one is harder.

False accusations of LLM use are widespread. This policy makes each accusation into a moderation matter that must be litigated. An accused Project member can't just ignore a false accusation as an annoyance when we've decided to treat this as a bannable CoC violation. We will have raised the stakes.

Surely we don't expect the accused to prove a negative. And surely we don't intend to judge people as liars based on faulty stylistic heuristics. So what is the plan for how we adjudicate this fairly, consistently, and while lowering the burden on moderators?¹

View changes since the review

Footnotes

1. For anyone wondering why I needed to momentarily unlock the thread to post this review comment, I don't know either. But GitHub refused to let me post it while the thread was locked ("Failed to save comment: Issue is locked."). ↩

[PLeVasseur]

Hrm. I'm thinking on this a bit.

This does seem like an important point. I'd worry about accusations that cannot be disproved leading to a moderation action.

What would be the mechanics of the accusation resolution?

[apiraino]

@traviscross it's about trust.

You can assume good faith and cannot bullet-proof malice but we will do our best to tell one from the other. I think the word "experimentation" in this document is also about this and I wouldn't block on this concern.

This policy makes each accusation into a moderation matter that must be litigated

Not exactly. This document indicates guidelines for project members and for us (mods) as well as expectations from contributions. Mods are called to help - and I am quoting the document - in case of "major violations or extractive PRs" while reviewers are asked to be helpful in case of "minor violations". I wouldn't read too much into it.

---

At the RustWeek Niko said something about the Rust funding initiative that I think also applies here: maybe we won't get everything right immediately but we will have something out now and we will iterate as needed.

[jyn514]

Note also that I have an entire "Don’t play detective" section dedicated to discussing this.

[traviscross]

@apiraino: I'm trying to connect it, but I don't really follow how what you're saying engages with the concern I'm raising. The lines I'm commenting on above are the only part of the policy that is made coequal with other CoC items. If I accuse a Project member of harassment (a CoC item), I'd expect the mods would investigate to see whether that occurred. Similarly, if I accuse another member of lying about the extent of covered LLM use, given the policy treats this as a CoC item, I'd presume the mods would investigate to determine whether that happened.

My question isn't about trust, or good faith, or malice — those would seem to come after we determine the facts, and I'm unsure how we plan to determine facts here. My worry is that we may only have two bad options: guessing and unenforceability. I'm wondering whether (and hoping that) we have a better idea for this. We're talking about judgments that will have real effects on people. This isn't the kind of thing I'd want to leave to experimentation.

@jyn514: The "it's not your job to play detective" section mostly gives guidance to reporters (those who believe the policy has been broken). I'm asking how we expect the moderators (to whom the reports go) to adjudicate these reports. On that, the section says only, "the mod team is free to exercise their own judgment and discretion", but I don't see how this solves the epistemological problem. And asking the mod team to decide the undecidable seems in tension with the goal of making "the policy enforceable and easy to moderate."

[lumi-me-not]

I feel the only thing you can do here is to go with "innocent until proven guilty". My stances are very anti-LLM but I would rather have some LLM outputs slip in than have witch hunts against innocent contributors.

Reports should include as much evidence as possible, so it makes LLM usage very obvious and reduces moderation burden.

Also, I believe https://cyrneko.eu/ai-policy.html has a bunch of good ideas.

[xtqqczze]

I agree that "innocent until proven guilty" is the only workable approach here. What other standard could realistically be applied?

Once failure to disclose LLM use becomes a CoC violation, the project implicitly needs a standard of proof. But allegations of undisclosed LLM use are often inherently hard to verify, so the burden of proof pretty much has to be, in other words: "beyond a reasonable doubt”.

And if a case really is obvious beyond reasonable doubt, it probably does not need to become a moderation matter in the first place, the contributor can simply be pointed at the policy and asked to correct the disclosure. The difficult cases are precisely the ones where certainty is not realistically achievable.

[jyn514]

I'm going to unlock this now that RustWeek is over and we're not all busy.

[xtqqczze]

Suggested change

	It is **not** ok to harrass a contributor for using an LLM.
	It is a [code of conduct](https://rust-lang.org/policies/code-of-conduct/) violation to harass a contributor on the basis of suspected or perceived LLM use.

View changes since the review