Skip to content

feat(citability): add answer-first hint#51

Draft
KimHyeongRae0 wants to merge 3 commits into
multivmlabs:mainfrom
KimHyeongRae0:feat/answer-first-citability
Draft

feat(citability): add answer-first hint#51
KimHyeongRae0 wants to merge 3 commits into
multivmlabs:mainfrom
KimHyeongRae0:feat/answer-first-citability

Conversation

@KimHyeongRae0
Copy link
Copy Markdown
Contributor

@KimHyeongRae0 KimHyeongRae0 commented May 12, 2026

Adds an answer-first citability hint without changing scores. Validation: npm run lint, npm run test -- --run src/core/citability.test.ts, npm run test -- --run, npm run build.

@vercel
Copy link
Copy Markdown

vercel Bot commented May 12, 2026

@KimHyeongRae0 is attempting to deploy a commit to the Cytonic Team on Vercel.

A member of the Team first needs to authorize it.

@KimHyeongRae0 KimHyeongRae0 force-pushed the feat/answer-first-citability branch from 00d8b1e to 0a96be3 Compare May 12, 2026 16:48
@KimHyeongRae0 KimHyeongRae0 changed the title [codex] add answer-first citability hint feat(citability): add answer-first hint May 12, 2026
@rubenmarcus
Copy link
Copy Markdown
Member

Queued for substantive review. Touching citability scoring deserves a careful read — I'll come back after the current review queue clears. Thanks for the patience @KimHyeongRae0.

@rubenmarcus
Copy link
Copy Markdown
Member

@greptileai review

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented May 14, 2026

Greptile Summary

This PR adds an "answer-first" citability hint that fires when a page has at least one answer-quality paragraph but doesn't lead with one. It also extracts the inline answer-quality predicate into a reusable isAnswerQualityParagraph helper.

  • The new hint is gated on answerCount > 0 (preventing double-noise with the existing zero-answer warning) and uses a bespoke startsContextually check that deliberately skips the 200-word cap from isAnswerQualityParagraph, avoiding false positives on long direct openers.
  • Three of the four new test cases exercise the intended paths precisely; the 200+ word opener regression test passes via the answerCount === 0 guard rather than through the startsContextually branch it documents.

Confidence Score: 5/5

The change adds a new informational hint only — scores are unchanged and no existing paths are altered.

The implementation is well-contained: the new hint fires only when answerCount > 0, the startsContextually check is logically correct, and the refactored isAnswerQualityParagraph helper is a straightforward extraction. The one test that passes for a slightly different reason than its comment claims does not affect production behaviour.

src/core/citability.test.ts — the 200+ word opener regression test would be more robust with an explicit answer-quality paragraph to ensure answerCount > 0.

Important Files Changed

Filename Overview
src/core/citability.ts Extracts isAnswerQualityParagraph as a named helper and adds an answer-first hint gated on answerCount > 0 with a bespoke startsContextually check that intentionally avoids the 200-word cap; addresses all three issues raised in prior reviews
src/core/citability.test.ts Adds four targeted test cases; three are well-formed, but the 200+ word opener regression test passes via the answerCount === 0 guard rather than through the startsContextually path it documents

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[scoreAnswerBlocks called] --> B[Loop paragraphs\ncount answer-quality via isAnswerQualityParagraph]
    B --> C{answerCount == 0?}
    C -->|yes| D[Push 'No direct answer\nparagraphs' warning]
    C -->|no| E[Find firstSubstantialParagraph\n≥ 20 words, non-heading]
    D --> F[Long-paragraph loop]
    E --> G{firstSubstantialParagraph\nexists?}
    G -->|no| F
    G -->|yes| H{startsContextually?\nnot capital-non-pronoun}
    H -->|yes| I[Push 'Lead with a direct\nanswer' suggestion]
    H -->|no| F
    I --> F
    F --> J[Return CitabilityDimension]
Loading
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
src/core/citability.test.ts:115-129
**Regression test passes for the wrong reason**

The test comment says it validates that a 200+ word direct-answer opener doesn't trigger the hint via `startsContextually`. However, with this content `answerCount` is always 0: the long opener (~210 words) fails `goodLength` (`<= 200`), and the 14-word second paragraph fails the `>= 20` minimum. The `answerCount > 0` guard short-circuits the check before `startsContextually` is even reached, so the test doesn't exercise the path it claims to protect.

To properly exercise `startsContextually` for a 200+ word opener, the content needs at least one independent answer-quality paragraph (20–200 words, capital non-pronoun) so `answerCount` reaches 1 before the hint check runs.

Reviews (4): Last reviewed commit: "fix(citability): don't false-positive an..." | Re-trigger Greptile

Comment thread src/core/citability.ts Outdated
Comment thread src/core/citability.ts Outdated
Greptile flagged two P2 issues on PR multivmlabs#51:

1. Threshold mismatch — firstSubstantialParagraph used >=15 words while
   isAnswerQualityParagraph requires >=20. A 15-19-word opening would
   be selected as "first substantial" but fail the answer-quality check
   even when it's actually a direct answer, firing a false-positive
   hint. Aligned both at >=20.

2. Double-hint noise — when answerCount === 0, both the existing
   warning ("No direct answer paragraphs found...") AND the new
   suggestion ("Lead with a direct answer paragraph...") fired. The
   suggestion is only useful when at least one answer paragraph
   already exists somewhere on the page. Gated the suggestion behind
   answerCount > 0.

Adjusted the positive test to use a long-enough contextual first
paragraph plus a real answer-quality second paragraph (so answerCount
> 0). Added a new test asserting that pages with zero answer
paragraphs get only the warning, not both hints.

15 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rubenmarcus
Copy link
Copy Markdown
Member

Aligned thresholds at >=20 words and gated the new suggestion behind answerCount > 0 so it doesn't double up with the existing zero-answer warning. Test updated + new test for the zero-answer case. 15 tests pass.

@greptileai re-review please.

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented May 14, 2026

Want your agent to iterate on Greptile's feedback? Try greploops.

Comment thread src/core/citability.ts Outdated
…openers

isAnswerQualityParagraph caps at 200 words. A well-formed 201+ word opener
starting with a capital noun is still a direct answer — splitting it is what
the long-paragraph hint is for. Check only the "starts contextually" half
(capital noun, not pronoun) for the answer-first suggestion.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rubenmarcus
Copy link
Copy Markdown
Member

Fixed in f19fae9 — the answer-first hint no longer gates on isAnswerQualityParagraph (which caps at 200 words). It checks only the contextual-opener heuristic (capital noun, not pronoun), so a 201+ word opener that does lead with a direct answer no longer false-positives. Added a regression test covering the >200 word case.

@greptileai review

@rubenmarcus
Copy link
Copy Markdown
Member

@greptileai review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants