Exam CCAR-F Topic 1 Question 135 Discussion

Actual exam question for Anthropic's CCAR-F exam
Question #: 135
Topic #: 1
After deploying the automated review, you notice high precision but low recall-real bugs are slipping through undetected. Investigation reveals that your review prompt instructs Claude to "only report high- confidence issues you are certain about" and "err on the side of not commenting." Developers appreciate the low noise, but a race condition that caused a production outage was visible in a reviewed pull request and went unreported. You need to substantially improve bug detection while keeping false-positive rates manageable. What is the most effective approach?

Suggested Answer: C Vote an answer

The prompt's conservative reporting policy is directly causing the low recall. Claude may discover a legitimate race condition during analysis but suppress it because it cannot satisfy the instruction to report only issues about which it is certain. Option C separates two objectives that should not be conflated: broad defect discovery and strict acceptance filtering.
Anthropic's current code-review prompting guidance explicitly recommends reporting every issue, including uncertain or lower-severity findings, assigning confidence and severity metadata, and allowing a separate verification stage to filter them. This maximizes recall while retaining control over developer-facing noise.
Option A leaves the suppression policy intact, so additional examples cannot guarantee that discovered problems will be reported. Option B improves recall but relies on historical category-level suppression, which can discard genuine findings that happen to belong to noisy categories. Option D can improve analysis quality but does not remove the instruction responsible for withholding findings. A dedicated finding stage followed by an independently configurable verification or thresholding stage produces measurable recall and precision controls without forcing one model call to optimize competing objectives.

by Barbara at Aug 05, 2026, 02:03 AM

Comments

Chosen Answer:
This is a voting comment (?) , you can switch to a simple comment.
Switch to a voting comment New
Nick name: Submit Cancel
A voting comment increases the vote count for the chosen answer by one.

Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.

0
0
0
10