feat(validator): make OSS credibility volume-aware and gate-only#1359
Open
plind-junior wants to merge 3 commits into
Open
feat(validator): make OSS credibility volume-aware and gate-only#1359plind-junior wants to merge 3 commits into
plind-junior wants to merge 3 commits into
Conversation
Closes entrius#1340. Credibility (merged/(merged+closed)) did two jobs — a 0.80 hard gate and a per-PR multiplier — and both punished active contributors. A miner at 30 merged / 8 closed scored 0.79 and was rejected outright, while 3/0 earned a free 1.0; and anyone past the gate was still taxed by the same ratio on every PR (charged twice). Smoothing prior: credibility = (merged + k) / (merged + closed + 2k), k=CREDIBILITY_SMOOTHING_PRIOR=2. 3/0 -> 0.71 (no free pass), 30/8 -> 0.76 (no death sentence). MIN_CREDIBILITY retuned 0.80 -> 0.70 to match the new scale (still per-repo overridable). Gate-only: _score_eligible_repo_prs no longer multiplies earned score by credibility (credibility_multiplier=1.0). Past the gate, PRs stand on review-quality, label, and time-decay multipliers. repo_eval.credibility is still computed and reported. Scope: OSS-contribution credibility only, matching the issue's cited code. The issue-discovery analogue (calculate_issue_credibility + discovery_credibility_multiplier) has the same shape and is left as a follow-up.
Contributor
|
I appreciate the work on this. Early in my Gittensor run, the maintainer of the main project I contribute to did a major rebase / fork-network switch. That left 4 of my open PRs effectively unrecoverable, so they had to be closed despite the work still being valid. It was pretty painful to get hit that hard on credibility while also having substantially more merged PRs overall. This change feels more balanced. It still filters for contributors with a real track record, but it’s a lot more forgiving for people who are consistently shipping and occasionally get caught by repo/process churn outside their control. Cheers 🔥 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1340. Credibility (merged/(merged+closed)) did two jobs — a 0.80 hard gate and a per-PR multiplier — and both punished active contributors. A miner at 30 merged / 8 closed scored 0.79 and was rejected outright, while 3/0 earned a free 1.0; and anyone past the gate was still taxed by the same ratio on every PR (charged twice).
Smoothing prior: credibility = (merged + k) / (merged + closed + 2k), k=CREDIBILITY_SMOOTHING_PRIOR=2. 3/0 -> 0.71 (no free pass), 30/8 -> 0.76 (no death sentence). MIN_CREDIBILITY retuned 0.80 -> 0.70 to match the new scale (still per-repo overridable).
Gate-only: _score_eligible_repo_prs no longer multiplies earned score by credibility (credibility_multiplier=1.0). Past the gate, PRs stand on review-quality, label, and time-decay multipliers. repo_eval.credibility is still computed and reported.
Scope: OSS-contribution credibility only, matching the issue's cited code. The issue-discovery analogue (calculate_issue_credibility + discovery_credibility_multiplier) has the same shape and is left as a follow-up.