Two language-model judges scoring the same 45 GSM8K answers disagreed with the ground truth in opposite directions: one systematically too strict, the other too lenient.

Grade these five real answers yourself first, passed or failed, then meet the two graders. Source: evaltrust @ 18e999a5, data/real/ (real judge runs on GSM8K). Nothing leaves your browser.

1. Q1. Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?

   Eggs laid: 16
   Eggs eaten: 3
   Eggs for muffins: 4
   Eggs sold: 16 - 3 - 4 = 9
   Price per egg: $2
   Daily earnings: 9 * 2 = $18
   
   The answer is 18.

   Passed Failed

2. Q2. On the first day of school, Ava sees that her friends Cole and Xavier have both grown a lot over the summer. Xavier was 4 feet tall and grew 3 inches. Cole was 50 inches tall and grew 2 inches over the summer. What is the difference between Cole and Xavier’s height now?

   Xavier's growth: 3 inches.
   Xavier's initial height: 4 feet = 4 * 12 = 48 inches.
   Xavier's current height: 48 + 3 = 51 inches.
   
   Cole's growth: 2 inches.
   Cole's initial height: 50 inches.
   Cole's current height: 50 + 2 = 52 inches.
   
   Difference: 52 - 51 = 1 inch.
   
   The answer is 1.

   Passed Failed

3. Q3. John cuts down an 80-foot tree. He can make logs out of 80% of it. He cuts it into 4-foot logs. From each of those logs, he cuts 5 planks. He then sells each plank for $1.2. How much does he make?

   1. Usable wood: 80 feet * 0.80 = 64 feet
   2. Number of logs: 64 feet / 4 feet/log = 16 logs
   3. Number of planks: 16 logs * 5 planks/log = 80 planks
   4. Total earnings: 80 planks * $1.2/plank = $96
   
   The answer is 96.

   Passed Failed

4. Q4. Cedar Falls Middle School has students in grades 4 – 7 and each year they are challenged to earn as many Accelerated Reader points as they can. The 10 students in each grade with the most points get to try an escape room set up by the teachers. Only 8 students can try the escape room at a time. They have 45 minutes to try and escape. If every group uses their full 45 minutes, how long will it take for everyone to try the escape room?

   There are 4 grades (4, 5, 6, 7) with 10 students each, so there are 4 * 10 = 40 students total.
   Since only 8 students can try the escape room at a time, there are 40 / 8 = 5 groups.
   Each group takes 45 minutes.
   Total time = 5 groups * 45 minutes/group = 225 minutes.
   
   The answer is 225.

   Passed Failed

5. Q5. Billy is peeling and cutting potatoes in preparation for making potato salad for his big family reunion barbecue. It's a big event, so he has 60 potatoes to get through. The potatoes are roughly the same size, so it takes about the same amount of time to peel and cut each one. It takes Billy about a minute and a half to peel a potato, but only about 5 seconds to cut it up. How long will it take Billy to finish prepping the potatoes?

   Let's break it down:
   
   * Peeling each potato takes 1.5 minutes (90 seconds)
   * Cutting each potato takes 5 seconds
   * Total time per potato is: 90 seconds + 5 seconds = 95 seconds = 1 minute and 35 seconds
   * Billy has 60 potatoes to prepare
   
   To find the total time, multiply the time per potato by the number of potatoes:
   
   1 minute and 35 seconds x 60 = ?
   
   The answer is 87 minutes.

   Passed Failed