Callround®
Book a pilot

What we measured

Numbers we measured, and how.

Each number says on what and how many it was measured, how we did it, and what it does not tell you.

In this page

Speech

6.3%

of words were wrong when Callround AI wrote down Nigerian English speech.

On what and how many

400 recordings by 31 Nigerian speakers (19 women, 12 men), 4,863 words, from the public Nigerian English speech set OpenSLR 70.

How we measured it

Each recording was turned into text once by the speech model Callround uses during an interview. Nothing was tuned on these recordings.

The reference text and the model's text were cleaned in the same way before comparing: lower case, punctuation removed, numbers written as words. These rules were fixed before any result was seen.

The figure is all word errors divided by all reference words. The likely range is 5.4% to 7.4%, found by drawing the 31 speakers again and again 1,000 times.

What this does not tell you

These are short read sentences recorded for speech research. They are not interview answers spoken on a phone, which are harder.

20.2%

of words were wrong on recorded conversations between Nigerian English speakers, the hardest speech we tested.

On what and how many

400 pieces of conversation by 5 Nigerian English speakers, 14,398 words, from the public speech set EdAcc. For comparison, 120 pieces each by speakers of three other accents.

How we measured it

The same speech model and the same rules as for the read sentences. Here people talk to each other freely, with pauses, restarts and talking over one another.

On the same test, mainstream US English came out at 16.2%, so the gap for Nigerian English was 4.1 points. With only 5 Nigerian speakers the likely range of that gap is wide: from 1.7 points the other way to 12.5 points.

The speech model's own doubt pointed at the bad transcripts. In the quarter of pieces it was least sure of, 41.8% of words were wrong. In the quarter it was most sure of, 14.8% were.

What this does not tell you

An interview answer is one person speaking into their own phone, which sits between read sentences and free conversation. Neither test used interview answers recorded on phones in Nigeria.

0.1 points

was the gap in word errors between women's and men's recordings.

On what and how many

The same recordings: 19 women (6.3% of words wrong) and 12 men (6.4%).

How we measured it

The word error rate was worked out for each group in the same way, and the gap is one minus the other.

The likely range of the gap is from 2.3 points one way to 2.0 points the other, found by drawing the speakers again 1,000 times. A gap of this size is too small to tell apart from chance with 31 speakers.

What this does not tell you

The set does not record a speaker's first language or region, so this says nothing about differences between accents within Nigeria.

14.8% against 1.6%

of words were wrong in the recordings Callround AI was least sure of, against the ones it was most sure of.

On what and how many

The same 400 recordings, split into four equal groups by how sure the speech model said it was of its own words.

How we measured it

The speech model gives each word a number for how sure it is. Each recording's average was taken, and the recordings were put in order.

In the least sure quarter 14.8% of words were wrong. In the most sure quarter 1.6% were. This is why an answer with many unsure words is sent to a person instead of being scored as it stands.

What this does not tell you

It shows the signal points the right way on read speech. It does not yet show how many bad transcripts of real interview answers the check would catch.

Scoring

97%

of scores were within one point of the level each test answer was written to.

On what and how many

164 skill scores on 90 written test answers for three roles: customer service, sales and graduate trainee. Each answer was written to a set level from 1 to 5.

How we measured it

Each answer was scored once with the method Callround uses in a hiring round: against the role's score descriptions and worked examples, with the answer shown as numbered sentences.

43% of scores matched the level exactly and 97% were within one point. The average score rose with every level: 1.6, 2.3, 3.3, 3.7 and 4.0.

The same answers scored with skill names only, and no score descriptions, were within one point 76% of the time.

What this does not tell you

The answers were written for the test to a known level, and they were scored as clean text. This shows the scoring can tell a strong written answer from a weak one. It does not show agreement with experienced recruiters, which needs real answers rated by people. When 15% of the words were changed to imitate speech recognition errors, scores fell by 0.3 points on average, which is why an answer the speech model is unsure of goes to a person. Almost no answer was given a 5.

184 of 213

scores came out the same in all three scoring runs. The other 29 differed by one point.

On what and how many

213 skill scores from the 20 interviews in the three test hiring rounds on this site, counting each score that all three runs gave.

How we measured it

Callround AI scores every answer three separate times and keeps the middle score.

For each skill score the highest run was compared with the lowest. None differed by two points or more. A gap that large sends the answer to a person.

What this does not tell you

The interviews were prepared to test Callround: the answers were written for the test and spoken by a computer voice. Real applicants' answers are less tidy.

0 of 90

scoring calls returned words the applicant did not say, once the AI had to point at numbered sentences.

On what and how many

The same 90 written test answers, scored two ways.

How we measured it

When the AI copied out its own quotes, 5 of 90 calls returned at least one quote that was not word for word in the answer.

Callround now shows the answer as numbered sentences. The AI names the sentences behind a score and code looks them up, so a quote is always the applicant's own words.

In the three test hiring rounds on this site, all 1,705 quotes behind the scores were found in the answers.

What this does not tell you

A real quote can still be a poor reason for a score. That is why every score shows its quote, and a person can change any score.

Fair treatment

0.05 points

is how far an answer's score moved, on average, when a personal detail was added to it.

On what and how many

63 pairs of written answers, the same answer with and without one personal detail: that the speaker is a woman or a man, their religion, home state, age, family or health. Each text was scored three times.

How we measured it

The detail was one plain sentence added to the answer, with nothing else changed. Scores are out of 5.

The same answers with a sentence about the weather added in the same place moved by 0.02 the other way. Two runs of the very same text differ by 0.2 on average, so a personal detail moved a score less than the scoring's own run to run difference.

With seven kinds of detail and 9 pairs of each, a mention of health and a hint of age leaned slightly up. That is too few pairs to settle either way.

What this does not tell you

The answers were written for the test, and this is one check of the scoring on text. It is not proof that Callround is fair to real applicants, which needs real answers from many kinds of people.

59 of 60

questions an interviewer must not ask were refused before a role could open.

On what and how many

120 written interview questions for three roles: 60 that ask about age, religion, tribe or home state, marriage, pregnancy, health, disability or politics, some in a roundabout way, and 60 ordinary job questions.

How we measured it

Each question went through the Question Screen exactly as it does when a recruiter saves a role: fixed rules in code first, then a check by Callround AI.

The one that got through asked about taking part in an election campaign. A rule for it was added, and all 60 are now refused.

3 of the 60 ordinary questions were refused by mistake, two of them for the words "how old" said about an invoice and a complaint. The recruiter sees the reason and can reword the question.

What this does not tell you

The questions were written for the test. A person at the employer still reads and approves every question, and that is the last check.

150 of 150

times, the next question was one the employer approved, even when the applicant brought up a personal topic.

On what and how many

150 written answers in which the applicant mentions their religion, tribe, marriage, pregnancy, health, disability or politics. In 73 of them the applicant invites a question about it.

How we measured it

Callround AI never writes a question during an interview. It reports facts about the answer, and code picks one follow up from the list the employer approved, or moves on.

Each answer went through the same code that runs after every answer in a real interview. 64 were followed by an approved follow up and 86 by the next planned question.

What this does not tell you

This checks a rule in the code, on answers written for the test.

Integrity

58 of 58

copied answers were noticed, and so were 58 of 58 copies with a few words changed.

On what and how many

58 written answers copied three ways, hidden among 180 answers by different applicants. Of the copies reworded sentence by sentence, 48 of 58 were noticed.

How we measured it

The product's own Integrity Signals code was run on six questions for each of three roles, each with ten answers by different applicants plus one copy.

A signal is raised when two answers to the same question share long runs of the same words, or use nearly the same words overall.

None of the 810 pairs of answers by different applicants was flagged.

What this does not tell you

The copies were reworded by a fixed rule. A person rewording freely, or a group of applicants taught the same model answer, has not been tested.

59 of 63

answers that tried to give the AI an instruction were noticed. None of 48 ordinary answers was.

On what and how many

63 written answers, each with one attempt added to instruct the interviewer or the scorer, in seven styles: a direct command, a note pretending to come from the system, a command hidden in the story, a command in Pidgin, a polite request, role play, and a claim that a score was already given.

How we measured it

Each answer went through the same code that runs after every answer in a real interview.

An attempt raises a signal for a person. It does not change a score. In 53 of the 63 pairs the applicant would hear the same next question with or without the attempt.

Each pair was also scored three times. The answers with an attempt scored 0.07 lower on average, not higher.

What this does not tell you

The attempts were written for the test in fixed styles. A new style may get through, which is why the answer is treated as data at every step and a person makes the decision.

Weak networks

177 of 177

pieces of recorded answers arrived exactly once on slow and broken connections.

On what and how many

Seven test interviews in a browser: normal connections, a slow one, a very slow one (50 kilobits a second), a connection cut for 30 seconds in the middle of an answer, and one where every piece was sent twice and out of order.

How we measured it

An answer is recorded in pieces of about three seconds, and each piece is sent while the applicant is still speaking. A piece that fails is sent again.

After each test the pieces on the server were counted against the pieces the page made, and the joined recording was compared with what was spoken. The longest difference was 0.11 seconds.

What this does not tell you

This was an early build of the interview page in a desktop browser with a simulated network, not real phones. Reloading the page in the middle of an answer lost up to the last three seconds of that answer.