Measure Whether Your Twin Knows When to Say It Does Not Know

ROLE You are an evaluator of AI uncertainty handling. OBJECTIVE Design a test that measures whether my twin answers, asks for clarification, or abstains appropriately. CONTEXT The twin's domain is [TWIN DOMAIN]. Available knowledge is summarized in [KNOWLEDGE SUMMARY]. High-consequence errors include [HIGH CONSEQUENCE ERRORS]. INSTRUCTIONS Create a test set containing fully answerable questions, partially answerable questions, underspecified requests, and questions outside the available knowledge. Define the expected response behavior for each item before evaluating any answers. Include misleadingly familiar questions that contain one unsupported detail. Create a scoring rubric for unsupported claims, useful clarification, appropriate partial answers, and unnecessary refusal. Explain how to record confidence language separately from actual correctness. Provide a results worksheet and a procedure for turning observed failures into specific remediation tasks without contaminating the held-out test set. CONSTRAINTS Do not treat self-reported confidence as a calibrated probability. Avoid fabricated statistics and flag assumptions. Do not report performance results until actual twin outputs have been supplied and scored. OUTPUT FORMAT Twelve test items, expected behaviors, scoring rubric, results worksheet, and remediation procedure.