Library › AI Evals

AI Evals

Rubrics, test cases and judge prompts that tell you when AI output is actually good. 90 exercises, about 5 minutes each.

Start the AI Evals track →

• Check AI variance commentary against the actual numbersFinance and accounting · Catching mistakes and made-up factsPremium
• Check AI-written release notes against the shipped ticketsProduct managers · Catching mistakes and made-up factsPremium
• Check an AI blog intro against your own survey reportMarketing and content · Catching mistakes and made-up factsPremium
• Check an AI lit-review paragraph against 3 abstractsCollege students · Catching mistakes and made-up factsPremium
• Check an AI reading summary against the source textTeachers · Catching mistakes and made-up factsPremium
• Check an AI research summary against your interview notesProduct managers · Catching mistakes and made-up factsPremium
• Check an AI team update against your project trackerManagers and team leads · Catching mistakes and made-up factsPremium
• Check an AI travel itinerary against the bookingsOffice and admin · Catching mistakes and made-up factsPremium
• Check an AI's API answer against the real docsSoftware developers · Catching mistakes and made-up factsPremium
• Check if an AI grader agrees with you on short answersTeachers · Catching mistakes and made-up factsPremium
• Check if two PMs agree before you build an AI judgeProduct managers · Catching mistakes and made-up factsPremium
• Check your restaurant chatbot's reply against the menuSmall business owners · Catching mistakes and made-up factsPremium
• Decide if an LLM judge is good enough to gate CISoftware developers · Catching mistakes and made-up factsPremium
• Draft 5 edge-case tests for an AI trip plannerProduct managers · Giving the right contextPremium
• File a bug report for an AI translation that broke a priceProduct managers · Asking for a format you can usePremium
• File a bug report when an AI PR summary misses a breaking changeSoftware developers · Asking for a format you can usePremium
• File a bug report when the bot quotes the wrong feeCustomer support teams · Asking for a format you can usePremium
• Grade 4 AI commit messages against your team's conventionSoftware developers · Asking for a format you can usePremium
• Grade 4 AI feedback comments against your district's rulesTeachers · Asking for a format you can usePremium
• Grade 4 AI onboarding emails against 3 criteriaProduct managers · Asking for a format you can usePremium
• Grade 4 AI social captions with quoted evidenceMarketing and content · Asking for a format you can usePremium
• Grade 4 bot refund replies against your rubricCustomer support teams · Asking for a format you can usePremium
• Grade 4 bot replies after an escalation rule changedCustomer support teams · Asking for a format you can usePremium
• Grade 4 scheduling-bot replies after a clinic moveNurses and healthcare workers · Asking for a format you can usePremium
• Grade an AI's expense categories against current policyFinance and accounting · Keeping the AI to your sourcesPremium
• Group 13 failing code-assistant runs into 3 causesSoftware developers · Asking for a format you can usePremium
• Group 13 failing shopping-bot traces into 3 failure typesProduct managers · Asking for a format you can usePremium
• List what to re-test before switching your team's AI modelManagers and team leads · Giving the right contextPremium
• Make a launch call from an AI scheduler's eval resultsManagers and team leads · Catching mistakes and made-up factsPremium
• Make a per-language launch call from eval resultsProduct managers · Catching mistakes and made-up factsPremium
• Measure how often an AI judge agrees with human labelsCollege students · Catching mistakes and made-up factsPremium
• Measure real resolution, not deflection, for a support botProduct managers · Giving the right contextPremium
• Pick the metric that shows if AI-written tests catch bugsSoftware developers · Giving the right contextPremium
• Pick the right metric and bar for an urgent-ticket taggerProduct managers · Giving the right contextPremium
• Pick the right metric for an AI flashcard generatorCollege students · Giving the right contextPremium
• Plan a monthly review of your team's AI meeting notesManagers and team leads · Setting limits and rulesPremium
• Plan a weekly human check of AI review summariesProduct managers · Setting limits and rulesPremium
• Plan a weekly review of AI-written product descriptionsMarketing and content · Setting limits and rulesPremium
• Plan which AI feedback comments you read each weekTeachers · Setting limits and rulesPremium
• Plan which bot chats your team reviews each weekCustomer support teams · Setting limits and rulesPremium
• Report a wrong calculus answer from the campus AI tutorCollege students · Asking for a format you can usePremium
• Sort 13 AI resume-parsing errors into 3 failure typesHR and recruiting · Asking for a format you can usePremium
• Sort 13 bad AI research answers into 3 failure typesCollege students · Asking for a format you can usePremium
• Sort 13 bad AI room bookings into 3 failure typesOffice and admin · Asking for a format you can usePremium
• Sort 13 failing AI meeting-note traces into 3 failure typesProduct managers · Asking for a format you can usePremium
• Sort 13 flagged AI promo emails into 3 failure typesMarketing and content · Asking for a format you can usePremium
• Sort 13 flagged AI quiz questions into 3 failure typesTeachers · Asking for a format you can usePremium
• Sort 13 flawed AI weekly reports into 3 failure typesManagers and team leads · Asking for a format you can usePremium
• Sort 13 low-rated bot chats into 3 failure typesCustomer support teams · Asking for a format you can usePremium
• Sort 13 trip-planner complaints into 3 failure typesProduct managers · Asking for a format you can usePremium
• Turn "the bot feels pushy" into 4 testable checksCustomer support teams · Catching mistakes and made-up factsPremium
• Turn "the bot sounds weird" into 4 testable checksManagers and team leads · Catching mistakes and made-up factsPremium
• Turn "the digest feels generic" into 4 testable checksProduct managers · Catching mistakes and made-up factsPremium
• Turn "the review bot is noisy" into 4 testable checksSoftware developers · Catching mistakes and made-up factsPremium
• Turn front-desk complaints about a reminder bot into checksNurses and healthcare workers · Catching mistakes and made-up factsPremium
• Write 4 pass/fail checks for AI newsletter draftsMarketing and content · Saying who it's forPremium
• Write 4 pass/fail checks for AI-drafted status updatesManagers and team leads · Saying who it's forPremium
• Write 4 safe tests for your salon's booking chatbotSmall business owners · Giving the right contextPremium
• Write 4 safe tests for your team's internal AI assistantManagers and team leads · Giving the right contextPremium
• Write 4 test cases for an AI accounts-payable assistantFinance and accounting · Giving the right contextPremium
• Write 4 yes/no checks for an AI essay feedback toolTeachers · Setting limits and rulesPremium
• Write 5 edge-case tests for a campus citation botCollege students · Giving the right contextPremium
• Write 5 edge-case tests for an internal docs Slack botSoftware developers · Giving the right contextPremium
• Write 5 red-team tests for an HR benefits botHR and recruiting · Giving the right contextPremium
• Write 5 safe red-team tests for a car rental support botProduct managers · Giving the right contextPremium
• Write 5 safe red-team tests for a skincare brand's chatbotMarketing and content · Giving the right contextPremium
• Write a 4-line pass/fail rubric for AI search answersProduct managers · Setting limits and rulesPremium
• Write a 4-line pass/fail rubric for AI study guidesCollege students · Setting limits and rulesPremium
• Write a 4-line pass/fail rubric for bot handoff notesCustomer support teams · Setting limits and rulesPremium
• Write a 5-line pass/fail rubric for an AI meeting recapProduct managers · Setting limits and rulesPremium
• Write a bug report for a summary that dropped a chargeSoftware developers · Asking for a format you can usePremium
• Write a golden reference answer for a stats tutor botCollege students · Keeping the AI to your sourcesPremium
• Write a JSON judge for banned hype words in ad copyMarketing and content · Setting limits and rulesPremium
• Write a JSON judge for replies in the wrong languageSoftware developers · Setting limits and rulesPremium
• Write a JSON judge that catches made-up SQL columnsSoftware developers · Setting limits and rulesPremium
• Write a JSON judge that catches needless refusalsProduct managers · Setting limits and rulesPremium
• Write a JSON judge that catches personal data in summariesSoftware developers · Setting limits and rulesPremium
• Write a JSON judge that catches wrong quiz answer keysTeachers · Setting limits and rulesPremium
• Write a JSON judge that checks ad prices against the listMarketing and content · Keeping the AI to your sourcesPremium
• Write a judge that catches answers not in the help articleProduct managers · Keeping the AI to your sourcesPremium
• Write a regression checklist before changing a review botSoftware developers · Giving the right contextPremium
• Write a regression checklist before shrinking RAG chunksSoftware developers · Giving the right contextPremium
• Write the golden answer for a clinic portal records questionNurses and healthcare workers · Giving the right contextPremium
• Write the golden answer for a dev-helper bot's git questionSoftware developers · Giving the right contextPremium
• Write the golden answer for a fractions tutor testTeachers · Giving the right contextPremium
• Write the golden answer for a return question with a new ruleCustomer support teams · Giving the right contextPremium
• Write the golden answer for a subscription pause questionProduct managers · Giving the right contextPremium
• Write the golden answer for an HR bot's PTO questionHR and recruiting · Giving the right contextPremium
• Write the golden answer for an IT helpdesk bot questionOffice and admin · Giving the right contextPremium
• Write the golden answer your bakery chatbot is graded onSmall business owners · Giving the right contextPremium