Can AI Really Assess English Proficiency? CEFR, Communicative Competence and Human Judgement.

Can AI Really Assess English Proficiency? CEFR, Communicative Competence and Human Judgement.

There is still a fair amount of scepticism among teachers about using artificial intelligence to assess English. That is understandable. Checking grammar or counting pauses is one thing; judging whether a learner can communicate successfully is quite another.

A human examiner listens to more than pronunciation. They notice whether the speaker has understood the situation, responded appropriately, developed an idea and achieved the purpose of the task. In writing, the same principle applies. Correct grammar matters, but so do relevance, organisation, clarity and the ability to communicate meaning.

AI-based assessment is now attempting to make these judgements as well. Some systems can analyse spoken or written responses, apply detailed scoring criteria and produce results within seconds. Early evidence from particular tests shows that automated scores can agree closely with scores from trained human raters. That is promising, although it does not mean that every AI assessment is equally accurate or suitable for every purpose.

A useful way to understand the issue is to look at what the CEFR expects language users to do, what AI can measure reliably, and where human expertise is still needed.

CEFR Is About What Learners Can Do With Language

The Common European Framework of Reference for Languages, usually known as the CEFR, describes proficiency through levels from A1 to C2. Its descriptors are built around what learners can do with language in different situations. The framework covers reception, production, interaction and mediation, rather than treating language proficiency as a collection of grammar rules and vocabulary items.

That difference is important when designing an assessment.

A learner may know a wide range of vocabulary but struggle to explain a simple problem clearly. Another learner may make occasional grammatical mistakes but handle a conversation confidently, give relevant information and respond appropriately. Both linguistic accuracy and communicative effectiveness matter.

The CEFR also places considerable emphasis on language users as social agents. In practical terms, this means that assessment should examine how well learners use language to achieve a purpose, not only whether individual sentences are technically correct.

Why Communicative Competence Is Harder to Measure

Consider a speaking task in which a learner is offered something they cannot accept. They are asked to decline the offer, explain why and suggest an alternative.

A response might contain excellent pronunciation and accurate grammar but still fail to complete the task. Perhaps the learner gives a long explanation without clearly declining the offer. Perhaps no alternative is suggested.

Another learner may make two or three small language errors but respond naturally: they decline the offer, give a clear reason and suggest another arrangement. From a communicative point of view, the second response has achieved more.

This is why a speaking test cannot rely on pronunciation and fluency alone. It also needs to consider whether the learner understood the task and responded in a way that fulfilled its purpose.

The same principle applies to writing. An essay can be grammatically accurate and still be irrelevant, poorly organised or unconvincing. Conversely, a learner can make minor errors while presenting a clear and well-developed argument.

Language assessment therefore has to consider the quality of the language and the success of the communication.

What AI Can Measure Well

Modern automated systems can examine many aspects of spoken English. Depending on the technology and the design of the test, these may include pronunciation, intelligibility, speech rate, pauses, fluency, vocabulary and grammar. Written responses can also be analysed for grammatical patterns, vocabulary range, sentence structure, organisation and relevance.

There are obvious practical advantages. A computer can process hundreds or thousands of responses without becoming tired. The same scoring rules can be applied repeatedly, and results can be produced far more quickly than in traditional manual marking.

Human raters work under different conditions. Listening carefully while judging several criteria at once requires concentration. Fatigue, differences in interpretation and changes in severity can create some variation between raters.

Automation can reduce part of that variation, but consistency on its own does not make an assessment good. A system could consistently reward the wrong feature. If a speaking test gave too much weight to speed, for example, a fast speaker might receive a higher score even when their message was unclear.

What matters is whether the system is measuring the right things.

How AI Can Judge Task Achievement

This is where newer AI systems go beyond traditional automated scoring. If the assessment criteria are clearly defined, the system can examine the content of a response as well as its linguistic features.

In the earlier example, the system could be asked to identify whether the learner:

This kind of evaluation is sometimes described mainly in terms of prompt engineering, but that description can be misleading if it suggests that the prompt is the most important part of the process. It is not.

Before a prompt is written, assessment specialists need to decide what ability is being tested. They need suitable tasks, clear scoring criteria and a definition of what stronger and weaker performances look like. The system then has to be tested against real learner responses.

Prompt design can help the AI apply those decisions. It cannot replace them. A well-written instruction given to a poorly designed test will still produce a poorly designed assessment.

What the Dynamic Speaking Test Evidence Shows

Clarity’s Dynamic Speaking Test, or DST, offers a useful case study because it includes task achievement alongside pronunciation, fluency, grammar and vocabulary. Its published description defines task achievement as the ability to handle different communicative tasks appropriately.

A 2024 calibration study compared automated scoring with ratings produced by six human raters. The dataset contained speaking performances from 34 participants across six questions. The results showed a strong relationship between the automated and human ratings.

The correlations between the AI mean rating and individual human-rater mean ratings ranged from 0.899 to 0.942. The report also calculated a Krippendorff’s alpha of 0.92 across the eight average ratings used in that analysis. These figures show a high degree of agreement within this dataset.

They need to be interpreted carefully, however. A high correlation does not mean that the machine and the humans gave exactly the same score every time, nor does correlation alone prove that a test is valid for every use.

There is another important limitation. The study involved only 34 participants, and the report itself states that this limits the generalisability of the findings.

The results are therefore useful evidence about this particular assessment system under the conditions studied. They should not be presented as proof that AI in general can assess every learner or every type of English test as accurately as an expert examiner.

Reliability Is Only Part of the Picture

Reliability and validity are closely related, but they are not the same thing. Consistency is what reliability concerns: if similar performances are assessed under similar conditions, the resulting scores should be reasonably stable. Validity, by contrast, concerns whether the evidence supports the interpretation and use of those scores.

This distinction matters because a test can be consistent while still measuring the wrong thing. Imagine a system that measures speech rate extremely accurately. If it repeatedly gives faster speakers higher proficiency scores, its measurement of speed may be reliable. That does not mean that speed is an adequate measure of communicative ability.

Good assessment practice therefore looks at several forms of evidence. Test developers need to know whether raters agree, whether tasks represent the intended language ability, whether scores behave as expected and whether the results are suitable for the decisions being made.

Cambridge English describes validity, reliability, fairness and fitness for purpose as central elements of assessment quality.

Accent Should Not Be Confused With Ability

Fairness becomes particularly important when AI assesses speech. English is spoken with a very wide range of accents. A Pakistani, Nigerian, Indian, Chinese or Spanish speaker may sound different from a British or American speaker while remaining completely understandable.

Modern CEFR guidance reflects this distinction. Its updated approach to phonological competence places greater emphasis on intelligibility and moves away from treating similarity to an idealised native-speaker accent as the main target. An automated test should follow the same principle.

The relevant issue is not whether a candidate has an accent. Almost everyone does. The question is whether pronunciation affects the listener’s ability to understand the message.

This is also one area where claims of machine objectivity need caution. AI systems are influenced by their development data, models and scoring rules. If some accents or learner groups are poorly represented, the system may perform differently for those speakers.

Research on automated speaking assessment has found cases where automated and human scores differed for particular groups, including speakers with speech or hearing impairments. Such findings do not mean automated scoring is inherently unfair, but they show why performance has to be checked across the populations who will actually take the test.

Fairness has to be demonstrated. It cannot simply be assumed because a score was produced by software.

CEFR Labels Also Need Evidence

There is another issue that can easily be overlooked. Producing a label such as B1, B2 or C1 is technically simple. Demonstrating that the label is justified is much harder.

A CEFR-linked test needs evidence showing that its tasks, scoring system and score interpretations relate meaningfully to the abilities described by the framework.

The Council of Europe is clear that it does not itself verify the quality of the claimed link between individual examinations and CEFR levels. Responsibility for demonstrating that link lies with the organisations and authorities using the framework.

This matters because CEFR labels carry considerable authority. When a learner is told that they are at B2, they are likely to assume that the result has a clear and recognised meaning. Assessment providers therefore need to be able to explain how they arrived at that result.

Simply adding a CEFR label to an AI-generated score does not establish CEFR alignment.

Where Human Judgement Remains Essential

AI is particularly useful when large numbers of responses need to be processed quickly and consistently. It can also provide detailed information that may help teachers and learners identify particular strengths and weaknesses.

None of this removes the need for human expertise. People still have to decide what the assessment should measure. They design the tasks, set the scoring standards, review the evidence and decide whether the test is suitable for its intended use. They also need to monitor the system when new learner groups, new tasks or new versions of the technology are introduced.

The level of human oversight should also reflect the consequences of the decision. An automated tool used for classroom practice does not carry the same risk as a test used for university admission, employment or professional certification. The higher the stakes, the stronger the validation and review procedures need to be.

Recent work on responsible AI in educational measurement makes the same point, placing fairness, transparency, security, accountability and continuous monitoring alongside technical performance.

The Strongest Model Is Human-Led and AI-Assisted

The evidence so far gives good reasons to take automated language assessment seriously. AI can analyse several features of speech and writing at once, apply scoring criteria consistently and handle a volume of responses that would be difficult to manage through human marking alone. There is also evidence from individual systems that automated scores can show strong agreement with trained human raters.

That is not the same as saying that the machine has replaced the examiner.

A trustworthy assessment still starts with people who understand language, testing and the learners being assessed. They decide what should count as successful communication. Tasks are then designed to make those abilities visible. Finally, the scoring is examined to see whether it works as intended and whether it works fairly across different groups.

AI can make that process faster and more consistent. It can also provide useful forms of analysis that would be difficult for a person to carry out repeatedly at scale. Its value depends on the assessment system around it.

The most credible direction for CEFR-based testing is therefore neither fully human nor fully automated. It is a carefully designed model in which human expertise defines the meaning of good performance and technology helps evaluate the evidence efficiently.

For learners and institutions, the standard should remain simple. A score should be understandable, supported by evidence and appropriate for the decision being made. If an AI-based test can meet that standard, it has a valuable role to play.