New accents on the ASR leaderboard: test the English your customers actually speak

Your voice demo can understand the founder perfectly and still make a new customer repeat every delivery address. A label saying “English supported” does not tell either person which experience to expect.

On November 6, Hugging Face and Liva AI added Egyptian and Nigerian English conversational test sets to the Open ASR Leaderboard. They contain 1.34 and 1.55 hours respectively, with spontaneous speech and references retaining punctuation, case, and disfluencies. This adds useful evaluation coverage; it does not certify any model for an entire country or every English speaker.

For nontechnical founders and small teams building voice features, the immediate opportunity is to improve a product decision: define the speech situations you intend to support, then measure whether people can finish their task in those situations. Use a leaderboard to choose candidates, and a customer coverage pilot to decide your release scope.

Below is a reusable pilot worksheet, a worked delivery-booking scenario, and a way to separate recognition mistakes from product failures. We have not run the new private datasets, tested models, or measured customer outcomes. The news facts come from the maintainers; the pilot and fictional examples are our proposed method.

1. What the new coverage changes, and what remains unknown

ASR means automatic speech recognition: converting audio into text. It is one component of a voice product, alongside turn detection, conversation logic, interface feedback, and any downstream action. An improvement in that component can matter without making the complete product reliable.

The November update also introduces broader accent groupings in the private results. Those groupings help expose differences hidden by one English average. They are organizing categories for these evaluations, not a diagnosis of an individual speaker's accent, identity, or ability. In particular, a geographic label cannot explain all the variation inside a sample.

The release should change the questions you ask a vendor. Request the relevant split, the evaluated model version, and its conditions. Ask whether your proposed service uses that version and whether the published result covers your input mode. A batch transcription result does not establish what happens during interruptions on a live telephone call.

It should not trigger an automatic provider migration. The new hours are limited, private samples; we do not know from the announcement alone how closely their participants, recording environments, vocabulary, or task distribution match your customers. The private status can reduce opportunities to optimize directly against public test material, but it does not prove the absence of contamination or establish universal coverage.

The actionable distinction is between candidate evidence and release evidence. Candidate evidence makes a model worth examining. Release evidence connects a specific configuration to a defined customer experience. Keep both in your decision note, with separate conclusions. If the public result is promising and your customers remain untested, the honest status is “candidate selected; coverage unverified.”

2. Replace the language checkbox with a task-and-condition map

Start with the task people are hiring the feature to do. “Recognize English” is too broad. “Let a customer request a delivery slot and review the captured address before submitting” gives you observable behavior, a boundary, and a recovery path.

A coverage cell is a combination of conditions worth examining together. For example: a volunteer's self-described English speech background, a phone microphone, ordinary indoor noise, and a delivery address containing unfamiliar place names. It is a research organizing tool, not a permanent customer label. Ask participants how they describe their speech; do not infer ethnicity or nationality from an audio clip.

Choose cells from actual recruitment conversations and the intended launch audience. Include people whose speech differs from the team's, but do not recruit one person and treat them as a national representative. Record distinct speakers and sessions, not only total minutes. A long recording by one frequent tester can dominate a dataset while adding little evidence about other people.

Older research on global English ASR disparities provides a reason to inspect variation rather than assume English is one uniform condition. Its historical systems and speech collection are not results for today's vendors. Use the research to justify broader evaluation, not to assign a current failure rate to a customer group.

Prioritize the cells relevant to the first release and explicitly mark the others unknown. That can support a bounded pilot without suggesting that unsupported people should be excluded from the product. Offer an equivalent text route and invite feedback about failed speech interactions. A narrow evidence claim is useful; an opaque access restriction based on presumed identity is not.

3. Read word error rate without mistaking it for customer success

Word error rate, or WER, compares recognized text with a reference transcript. It counts substitutions, deletions, and insertions relative to the reference word count. Microsoft's evaluation documentation explains the calculation and the need for representative audio paired with human transcripts. The metric describes text differences; its business significance depends on which words changed.

Consider the invented request “deliver on Tuesday, not Thursday.” Dropping “not” may look small in a transcript while reversing the booking constraint. Conversely, a harmless filler word can increase transcription error without preventing a successful booking. Keep WER as a diagnostic measure and add critical-field correctness: day, address, item, quantity, or any other field whose error changes the requested outcome.

Also distinguish a macro-average from a pooled rate. A macro-average gives selected dataset scores equal weight; pooling sums errors and reference words before dividing. The leaderboard application code contains selection-dependent averages and completeness filters. Record the tab, date, selected datasets, and missing entries when using its numbers. A changed selection can change the comparison without any model improving.

Here is deliberately hypothetical arithmetic, not a leaderboard result: a model with WERs of 5% and 25% across two equally weighted cells has a macro-average of 15%. That average conceals the second cell's much larger error burden. A pooled calculation could differ if one cell contains far more reference words. Neither calculation tells you whether a wrong delivery day was caught before submission.

For a founder's scorecard, preserve three separate observations: transcript differences, critical-field mistakes, and completed tasks. Add correction effort as a fourth. A flow that succeeds only after exhausting repetition can meet a completion target while remaining unsuitable for customers in a hurry.

4. Build references that preserve the mistakes you care about

A reference transcript is the text your reviewer believes was spoken. A scoring policy determines which differences count. Neither should be improvised after seeing which candidate wins. Decide in advance how to handle fillers, number formats, abbreviations, interruptions, and uncertain words.

Keep the original audio, the human reference, and the scoring-normalized version distinct. Lowercasing may be harmless for one measurement but unsuitable for evaluating the final text displayed to users. Converting “twenty one” and “21” to a shared form can remove a formatting difference; it must not erase a wrong apartment number. Keep a separate field-level check for those values.

The JiWER documentation describes edit-based speech metrics and explicitly handles empty references, enabling checks for text produced during silence. That is a useful reminder to include pauses and non-speech, not only clear sentences. A booking app should not invent a destination while a customer is thinking. Your test specification should say what the interface does during that pause and whether any draft action can advance.

Have a second reviewer examine uncertain passages and important discrepancies. If neither reviewer can confidently hear an address, preserve that uncertainty instead of forcing a reference that favors a candidate. Mark the example unresolved, report how many such examples exist, and decide whether the product should ask for clarification in that situation.

Do not turn a test set into training material by repeatedly adjusting prompts against every held-out example. Keep a development collection for iteration and a separate check collection with previously unseen speakers where feasible. If you expose a check item to guide a repair, record that exposure. It remains useful for regression testing, but it no longer carries the same evidence of fresh generalization.

5. Separate speech background from microphone and environment

Suppose one pilot group records in a quiet room on a laptop while another calls from a busy street. A quality difference could reflect speech patterns, audio conditions, or both. The product still needs repair, but the evidence does not isolate the cause.

A 2024 study of provenance confounding found that recording practices materially affected an ASR disparity analysis using CORAAL and Whisper. That finding is specific to its study, not a finding about the new Liva samples. It supports a methodological caution: preserve recording context before attributing an observed gap to a population.

For your own pilot, record microphone or channel, background conditions, speaking style, and whether there was overlapping speech. Where participants are willing, collect comparable tasks through more than one relevant channel. Keep the wording natural rather than asking everyone to mimic a preferred pronunciation. A perfectly controlled reading test can miss hesitation and self-correction that appear in ordinary use.

You do not need an enormous experiment to learn that a flow breaks on a particular phone connection. You do need to avoid overstating that observation. Write “this combination failed in the observed sessions,” then test a plausible explanation. If a microphone change helps the same participant, that is evidence worth pursuing; it is not proof that all accent-related differences have disappeared.

Treat missing coverage as missing evidence. A cell with no participants should remain blank, rather than inheriting the average from a neighboring country or language variety. Repeating one person's clip many times may reveal run-to-run instability, but it does not increase the number of independent speakers. Keep those counts separate in every review.

6. Work through a delivery-booking failure before choosing a fix

Imagine a fictional small business tool called Parcel Desk. Customers dictate a delivery request, review a summary, and submit it. The founder wants to add voice to an existing text form. This example has no measured results and does not describe a real customer deployment.

A participant says, “Tuesday afternoon, not Thursday; apartment twenty one.” The draft summary shows Thursday and apartment twelve. There are at least three separate questions: what the recognizer transcribed, how the application extracted the fields, and what the customer saw before submission. Inspect each boundary. A correct transcript with wrong fields calls for a different repair from incorrect recognition.

The founder's first response should be to keep the request a draft and make the disputed fields easy to inspect and change. Ask the participant to use the whole flow, including correction. Observe whether they notice the error without coaching. A generic “Looks good?” prompt is weak evidence if the important detail is hidden below a collapsed section.

Now compare candidate repairs. A vocabulary list might help recurring place names. A clearer review screen might catch a misheard day. A text fallback might let the customer complete the request when audio remains difficult. A model change could improve recognition but introduce different errors elsewhere. Evaluate the repair against the specific failure and against previously working cases.

The feature can be useful before it earns permission to submit requests without review. Parcel Desk could first offer “dictate a draft, then confirm the details.” That is a different product promise from “tell us once and we book it.” Test the promise you actually intend to ship, and measure the burden of the confirmation step rather than treating its existence as sufficient protection.

7. Copy this speech-coverage pilot worksheet

Use one record per coverage cell and configuration. The table is a proposed working artifact, not a validated standard or a set of universal pass thresholds. The product owner chooses acceptance rules before the check run, with technical input about measurement and the consequences of mistakes.

A filled example record might read: “PD-01; delivery draft; phone browser; ordinary indoor background; speech background volunteered by participants; day, address, and apartment are critical. Model configuration saved in the test record. Submission requires explicit review. Acceptance requires every observed submitted request to match the participant's confirmed details; any unintended submission holds the release. Completion and correction burden reviewed separately. Status: planned, no results.”

That last phrase matters. A well-designed worksheet is not evidence that a system passed. After testing, replace the status with actual counts and notes. If there were no unintended submissions in a small pilot, say exactly that; do not convert it into a production guarantee. A severe observed failure can justify a hold even when an average looks good.

Assign the unresolved items. For example, the founder owns the customer promise, a reviewer owns reference disputes, and an engineer owns configuration traceability. If nobody can identify the tested version after a provider update, the prior record cannot by itself establish the new version's coverage.

8. Choose repairs by the error, then retest the complete journey

Resist the temptation to treat every failure as a reason to buy a larger model. First classify the observed problem: audio capture, recognition, interpretation, presentation, correction, or action. More than one layer may contribute. Preserve enough detail to explain why the proposed intervention matches the failure.

For recurring domain words, Google's model adaptation documentation describes phrase biasing and warns that stronger boosting can increase false positives. This is a mechanism with a tradeoff, not a general accent repair. Test both the desired place name and plausible alternatives that should remain unchanged. A repair that forces the known name into unrelated speech creates another product failure.

For poor audio, make capture feedback understandable and offer another route. For an extraction mistake, compare the captured text with the structured fields. For an unnoticed draft error, test whether the review screen helps customers find and change it. For excessive repetition, examine the whole correction path, including whether users must restart after a minor mistake.

The Open ASR evaluation repository documents reproducible evaluation environments and separate handling of private sets by maintainers. Borrow the discipline of recording conditions, without claiming your small pilot reproduces that benchmark. Save the input mode, model identifier, relevant settings, reference policy, and date alongside the result.

When comparing two candidates, use the same check inputs and the same product flow. Count a provider timeout as an unavailable result, rather than quietly removing that session from the average. Record how many requests were attempted, returned a transcript, reached review, and completed. Review the extra delay and human correction work as well as transcription quality. A candidate that produces cleaner text but frequently fails to return before customers leave may be a worse fit for this workflow. Set the waiting and fallback behavior before comparing candidates so one is not given an easier journey.

Retest a repair on the affected cell and on cases that previously worked. Then examine an untouched check collection. If you cannot afford that check yet, keep the release bounded and label the missing evidence. A configuration that was tuned until it recognized one customer's address has demonstrated a local repair, not coverage for everyone who shares a geographic label.

9. Make the rollout match the evidence you have

Hold a short decision meeting with the worksheet visible. Review the weakest relevant cell first. Ask what was observed, what remains unknown, whether people could recover, and what exact promise the interface makes. Do not let a global score override an unresolved failure in a launch-critical journey.

A reasonable outcome can be a supervised draft pilot, a limited release with text fallback, or a hold while a specific failure is repaired. Availability decisions should follow the tested task and conditions, with inclusive alternatives; they should not silently sort customers by inferred accent. Explain limitations in ordinary product language and provide a useful way to report failures.

Measure ongoing correction and abandonment alongside task completion. Look for situations where users repeatedly switch to text or stop using voice. Invite voluntary context about the interaction instead of trying to derive sensitive identity labels from recordings. Inspect representative failed journeys under the participant permissions and access rules you established. Store only the material you have a justified reason to retain.

Set a review trigger as part of the release decision: a provider model change, a new telephone channel, a different customer task, or a sustained pattern of correction complaints should reopen the relevant cells. Keep the previous record so the team can see what changed. For a small team, this can be a dated document and a named owner rather than a new analytics platform. What matters is that a version change cannot inherit a green status without someone checking whether the earlier evidence still applies.

This method fits teams adding dictation, searchable transcripts, or reviewed voice intake to a product. It is not enough to approve clinical decisions, emergency handling, financial transfers, or other consequential autonomous actions. Those contexts require deeper domain evaluation and controls beyond this founder pilot. Nor does the method establish demographic fairness from a handful of volunteers.

The November leaderboard addition is valuable because it makes another coverage gap visible. The practical next step is modest and concrete: select one customer task, recruit beyond the founding team's speech habits, preserve the conditions, and decide what customers can safely complete and correct. Publish a promise that matches those observations, then broaden it as the evidence grows.

References

1. Hugging Face and Liva AI: Egyptian and Nigerian English join the Open ASR Leaderboard, November 6, 2026.
2. Open ASR Leaderboard evaluation repository.
3. Open ASR Leaderboard application source, inspected November 7; this moving source may change.
4. Microsoft: Test accuracy of a custom speech model.
5. JiWER documentation.
6. Li, Cohen, and Pakhomov: Reexamining Racial Disparities in Automatic Speech Recognition Performance: The Role of Confounding by Provenance, 2024.
7. DiChristofano and colleagues: Global Performance Disparities Between English-Language Accents in Automatic Speech Recognition, revised 2023.
8. Google Cloud: Improve transcription results with model adaptation.