Exam AI-500 Topic 1 Question 9 Discussion
Actual exam question for Microsoft's AI-500 exam
Question #: 9
Topic #: 1
Question #: 9
Topic #: 1
You have a Microsoft Foundry multi-agent customer support solution that routes requests from an intake agent to a retrieval agent, and then to a resolution agent. A custom guardrail is assigned to the agents.
You discover that some legitimate support requests are blocked, and some injected instructions in retrieved documents are allowed.
You need to validate the updated guardrail. The solution must meet the following requirements:
* Identify false positives and false negatives.
* Verify policy coverage across the agent solution
* Prevent changing the production agent behavior during testing
* Measure intervention accuracy by using a control and intervention point.
What should you do?
You discover that some legitimate support requests are blocked, and some injected instructions in retrieved documents are allowed.
You need to validate the updated guardrail. The solution must meet the following requirements:
* Identify false positives and false negatives.
* Verify policy coverage across the agent solution
* Prevent changing the production agent behavior during testing
* Measure intervention accuracy by using a control and intervention point.
What should you do?
Suggested Answer: B Vote an answer
The updated guardrail must be evaluated with known benign and adversarial cases so false positives and false negatives can be measured explicitly. Running labeled cases through every relevant workflow path also tests whether the guardrail is applied consistently at the intended intervention points. Microsoft AI-500 guidance includes guardrail testing with synthetic or curated data and emphasizes evaluation before production rollout.
Playground-only spot checks are too narrow to establish coverage. Switching production to annotate-only would change live behavior and use customers as the test population, violating the requirement to avoid production changes. A compliance configuration review verifies that a policy exists but does not measure whether it correctly detects or misses real inputs. Option B is therefore the only approach that produces repeatable evidence of intervention accuracy and policy coverage without changing production behavior. A robust evaluation program separates process metrics from final-response metrics. The selected answer measures the layer where the stated failure actually occurs, which is essential for deciding whether to change retrieval, orchestration, prompt behavior, or the final generator.
Official Microsoft reference: AI-500 Study Guide - guardrail testing and evaluation
Playground-only spot checks are too narrow to establish coverage. Switching production to annotate-only would change live behavior and use customers as the test population, violating the requirement to avoid production changes. A compliance configuration review verifies that a policy exists but does not measure whether it correctly detects or misses real inputs. Option B is therefore the only approach that produces repeatable evidence of intervention accuracy and policy coverage without changing production behavior. A robust evaluation program separates process metrics from final-response metrics. The selected answer measures the layer where the stated failure actually occurs, which is essential for deciding whether to change retrieval, orchestration, prompt behavior, or the final generator.
Official Microsoft reference: AI-500 Study Guide - guardrail testing and evaluation
by August at Sep 27, 2026, 04:40 AM
0
0
0
10
Comments
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
Report Comment
Commenting
You can sign-up / login (it's free).