AI Scoring Accuracy: Method, Calibration and Limits

Ways to get the most from it:

• Compare several answers, not one. A pattern across three attempts says more than any single estimate.
• Watch the criterion that comes out lowest each time — that is where practice pays off.
• Read the quoted evidence. If a quote does not match what you meant, give that estimate less weight.
• Before booking the test, combine AI estimates with full timed practice, including the official CELPIP practice tests.

1. 1

   Transcribe with timings

   Speech-to-text keeps fillers such as “um” and gives every word a start and end time.

2. 2

   Measure delivery in code

   Words per minute, pauses of 0.6 s or more, long pauses of 1.5 s or more, fillers per minute and the share of the response time you used are calculated — not guessed by the model.

3. 3

   Rate four criteria as a range

   The model rates content and coherence, vocabulary, listenability and task fulfillment on the 3–12 scale, as a range one or two levels wide, against a checklist for the task. Writing is rated on readability instead of listenability.

4. 4

   Check every quote

   Each piece of evidence must appear word for word in your answer. Anything that does not is removed before you see the report.

5. 5

   Apply fixed rules

   Answers that use less than 40% of the time or have fewer than 50 words are capped at level 6. If the audio itself could not be analysed, the report says pronunciation was judged from the text only.

6. 6

   Discard the recording

   Audio is processed in memory and never stored. Only the transcript and the report are kept — visible to you alone, until you delete them.

Calibration means comparing our estimates with levels that trained human raters give the same answers.

• Answers: practice recordings and essays from volunteers who opt in separately — covering all eight speaking tasks and both writing tasks, recorder and uploaded audio, and a range of first-language backgrounds.
• Raters: teachers with CELPIP or TESL teaching experience score each answer independently, without seeing our estimate.
• Fixed settings: the model, prompt version and settings are frozen before the comparison, so the figures describe the scoring you actually get.
• Consent: answers are used for calibration only with separate, opt-in consent, which can be withdrawn at any time.

• Pronunciation in detail. It hears overall clarity, rhythm and intonation, not individual sounds the way a dedicated pronunciation engine would.
• Every accent equally. Speech-to-text makes more mistakes with some accents, which can pull an estimate down. That is why we will report results by first-language background.
• What a transcript hides. Automatic transcripts can smooth over errors or add punctuation you did not speak.
• Very short or off-topic answers. These get low confidence and capped levels.
• Memorised templates. We flag answers that sound rehearsed and generic, but this detection is approximate.
• Your official result. Real CELPIP raters, test-day conditions and the full four-skill test all differ from one practice answer.