Benchmarking Fonada ASR: Fonada Kapture v2 and Fonada Kapture v2 Streaming

FonadaLabs TeamSeptember 10, 20266 min read

Automatic Speech Recognition (ASR) is a foundational component of modern voice AI systems, enabling real-time voice bots, automated transcription, and conversational analytics across diverse languages. Evaluating ASR systems accurately presents significant challenges, particularly in multilingual and code-mixed conversational environments. Traditional Word Error Rate (WER) metrics often penalize valid linguistic variations, loanwords, and script choices, leading to inflated error rates.

In this blog post, we present a comprehensive evaluation of Fonada Kapture v2 and Fonada Kapture v2 Streaming across standard open-source benchmark suites for English and Hindi. Rather than focusing on underlying model architecture, this article focuses strictly on evaluation methodology, benchmark suite composition, and empirical performance across standard WER/CER and semantic LLM-WER metrics.

The Challenge: Evaluating Speech Recognition in Multilingual & Code-Mixed Environments

Evaluating speech recognition quality is critical for building reliable voice agents. While Word Error Rate (WER) has long served as the industry standard, it relies on strict, word-for-word string comparisons between transcribed text and reference transcripts.

In multilingual and code-mixed settings (such as Hinglish), rigid string matching frequently misclassifies functionally correct transcriptions as errors:

Loanwords in Native vs. Latin Scripts: In Hindi, speakers regularly use English terms. An ASR model transcribing “doctor” in Latin script (doctor) against a Devanagari reference (डॉक्टर) receives a 100% error penalty despite identical meaning and phonetic fidelity.
Multiple Valid Orthographic Spellings: Hindi features several acceptable spellings for the same word, brand, or name.
Colloquial vs. Formal Variations: Spoken dialogue often contains colloquial phrasing that conveys the exact message of formal text but differs in exact character sequences.

To address these limitations, our benchmark incorporates both traditional deterministic metrics and LLM-WER, a semantic evaluation metric that accounts for linguistic equivalence.

Evaluation Methodology: LLM-WER Pipeline

Our evaluation methodology adopts the open-source LLM-WER framework developed by Sarvam AI. LLM-WER combines deterministic text alignment with Large Language Model adjudication to assess semantic and phonetic equivalence.

Benchmarking Fonada ASR: Fonada Kapture v2 and Fonada Kapture v2 Streaming — LLM-WER Evaluation Pipeline Methodology

Evaluation Pipeline Stages:

Reference & Hypothesis Ingestion: Ground-truth reference transcripts and model-predicted hypotheses are ingested into the evaluation pipeline.
Text Normalization & Baseline Scoring: Transcripts undergo standardized normalization (punctuation removal, lowercasing, language-specific normalizers) to compute baseline deterministic WER and CER.
SequenceMatcher Diffing: Differencing segments are isolated using difflib.SequenceMatcher opcodes to identify mismatched text spans.
LLM Judge Adjudication: Mismatched spans are evaluated by the Qwen3.5-35B-A3B model via the Fonada API to determine semantic and phonetic equivalence.
Transcript Reconstruction & Final Metrics: Equivalent spans are reconciled in the hypothesis text to calculate final LLM-WER and LLM-CER scores.

1. Text Normalization

Both reference and predicted transcripts undergo standardized normalization:

Trailing punctuation marks (,, -, ., (), [], {}, /, \) are spaced out.
Latin and Indic punctuation (danda, quotes, ZWJ/ZWNJ, dashes) are stripped.
Text is lowercased.
Language-specific tokenization/normalization is applied: Hindi (hi) uses indic-nlp-library IndicNormalizer, while English (en) uses the Whisper tokenizer normalizer (openai/whisper-small).
Extra whitespace is collapsed and trimmed.

2. Standard Metric Formulas

Standard Word Error Rate (WER) and Character Error Rate (CER) are calculated per utterance using clamped edit distance formulas:

$$\text{WER} = \frac{S + D + I}{\max(N, M)}$$

$$\text{CER} = \frac{S + D + I}{\max(\text{len}(\text{ref}), \text{len}(\text{hyp}))}$$

Where $S$ is substitutions, $D$ is deletions, $I$ is insertions, $N$ is reference word count, and $M$ is hypothesis word count. Scores are reported as macro-averages across all evaluated utterances, giving equal weight to every utterance.

3. Semantic LLM Adjudication

To calculate LLM-WER:

Differencing segments are identified using difflib.SequenceMatcher.
Mismatched spans containing both reference and hypothesis text are sent to an LLM judge via the Fonada API (https://api.fonada.ai/api/v1) using the Qwen3.5-35B-A3B model.
The LLM determines whether the two text spans are semantically equivalent (same meaning) and phonetically similar (sound alike).
Mismatches confirmed as equivalent are reconciled by copying the reference text into the hypothesis.
Final LLM-WER and LLM-CER are computed on the reconciled transcripts.

Benchmark Dataset Composition

The evaluation benchmark is compiled from two public open-source evaluation suites:

AI4Bharat Vistaar Suite (Hindi): Sourced from Kathbath, MUCS, Mozilla Common Voice, FLEURS, and IndicTTS.
Hugging Face Open ASR Leaderboard (English Suite): Covering diverse evaluation subsets including SPGISpeech, GigaSpeech, AMI, Common Voice English, Earnings-22, LibriSpeech, VoxPopuli, and TED-LIUM.

Models Evaluated

We evaluated two distinct Fonada ASR production models:

Fonada Kapture v2: High-accuracy batch ASR model optimized for offline processing and transcription workflows.
Fonada Kapture v2 Streaming: Ultra-low latency streaming ASR model engineered for real-time voice bot and interactive voice agent pipelines.

Benchmark Results & Key Findings

1. English Benchmark Results (Hugging Face Suite)

On the English benchmark suite, both models demonstrate strong performance across diverse acoustic conditions:

Fonada Kapture v2: 10.6% WER | 6.2% CER | 9.5% LLM-WER | 6.3% LLM-CER
Fonada Kapture v2 Streaming: 14.4% WER | 8.2% CER | 13.2% LLM-WER | 8.2% LLM-CER

2. Hindi Benchmark Results (Vistaar Suite)

On the Hindi benchmark suite, LLM adjudication highlights how standard WER penalizes harmless script and orthographic variations:

Fonada Kapture v2: Standard WER 12.5% $\rightarrow$ LLM-WER 5.3% | Standard CER 4.1% $\rightarrow$ LLM-CER 2.2%
Fonada Kapture v2 Streaming: Standard WER 17.9% $\rightarrow$ LLM-WER 9.6% | Standard CER 6.5% $\rightarrow$ LLM-CER 4.3%

3. Detailed Benchmark Comparison Table

Language	Model	Standard WER	LLM-WER	Standard CER	LLM-CER
English	Kapture v2	10.6%	9.5%	6.2%	6.3%
	Kapture v2 Streaming	14.4%	13.2%	8.2%	8.2%
Hindi	Kapture v2	12.5%	5.3%	4.1%	2.2%
	Kapture v2 Streaming	17.9%	9.6%	6.5%	4.3%

Key Takeaways

Batch vs. Streaming Performance: Fonada Kapture v2 leads across both English and Hindi evaluations, achieving a 10.6% English WER (9.5% LLM-WER) and an exceptional 5.3% Hindi LLM-WER. For real-time applications requiring low latency, Fonada Kapture v2 Streaming delivers competitive performance with a 14.4% English WER (13.2% LLM-WER) and 9.6% Hindi LLM-WER.
Impact of LLM Evaluation: Standard WER significantly overstates practical error rates in Hindi and code-mixed speech recognition. LLM-WER reduces perceived Hindi errors by over 57% for Kapture v2 (12.5% to 5.3%) and 46% for Streaming (17.9% to 9.6%), confirming that most discrepancies are benign spelling or transliteration variations.

Future Scope

We are actively expanding and refining our speech recognition models:

Noisy Acoustic Robustness: Enhancing performance in non-stationary noise environments such as call centers and public places.
Ultra-Low Latency Optimization: Further reducing streaming latency for seamless real-time voice bot interactions.