Library › AI Evals › Product managers
Spot why a judge scored 10 of 10 and tell your team
- For Product managers
- You practice Catching mistakes and made-up facts
- Time about 5 minutes
- Level Intermediate
About this exercise
Omar wants to switch Pedalbot's grading to the judge from the trial with the best score. Reply to Omar in a plain text Slack message of at most 4 sentences that starts with "Omar,". Name the trial with the setup flaw and the flaw, say why its score does not measure the judge, give the honest agreement score as a fraction and a percent, and end by saying what to rerun. Under 90 words.
Common mistakes when you ask AI for this
- Picking the best score. The AI agrees Trial 2 is best because 10 of 10 is the highest number. Try: Say: "Check what each trial's judge was shown before trusting its score."
- Reporting the wrong number. The AI repeats 10 of 10 or averages the trials. Try: Say: "Give the honest score: the fair trial with the policy, as a fraction and a percent."
- No next step. The AI explains the flaw but doesn't say what to rerun. Try: Add: "End by saying what to rerun."
- Too long and formatted. AI defaults to bold, bullets and a long explanation. Try: Ask for "plain text, at most 4 sentences, under 90 words."
What a strong prompt includes
- Asks to check what the judge saw
- Asks for fraction and percent
- Names the reader
- Asks for a next step
- Asks for plain text
Related exercises
- Accrue unpaid wages for the working days after the last pay period (Finance AI · Accountants)
- Age a returns queue in business days and flag the ones over the limit (Customer Support & Success AI · Customer support teams)
- Alert sales to a competitor's price change, facts only (Research & AI Search · Product managers)
- Annualize recurring revenue without the one-time fee (Data AI · Finance and accounting)
- Answer a credit dispute with the full-hour rule and the math (Customer Support & Success AI · Customer support teams)