Introducing Delta, an open benchmark for whether AI tutors help people learn

Most tests of AI tutors check whether the tutor gets the answer right. That matters, but it misses the point. A tutor can be right every time and still leave a student no better off. Sometimes it leaves them worse off, because it did the thinking for them.

Delta is being built to measure something else: the student. What did they know before the session, and what do they know after?

How Delta will work

Every test starts with a student who has a real gap in a real skill. Delta is designed to draw its students from Cohort: simulated students who carry the same knowledge and the same mistakes that real students do, and panels of real students from around the world, of every age and background. A tutor teaches the student. Then the student takes a fresh test on the same skill, with new questions.

The score is the difference between the two: the delta. A tutor that hands over answers will score poorly, because its students still cannot do the work on their own. A tutor that asks, hints and waits will score well.

What it will reward

• Asking before telling, so the student does the thinking.
• Finding the real misunderstanding, not only the wrong answer.
• Checking that a new idea stuck before moving on.
• Patience, in every language and at every level.

Open to everyone

We plan to open the method, the test sets and the scoring, so any lab can run Delta on any model and anyone can check a score. We want every AI tutor in the world to get better at teaching, including the ones we did not build.

Checked with real students

Simulated students let Delta test widely and quickly. Cohort’s panels of real students are there to check that a high score means real learning, for real people. Through Proof, we plan to publish what we find, so the benchmark earns its trust in the open.

Where it stands

Delta is in development inside United Education Labs. We will release it when it is ready, and it will take its turn on the release ledger.