FonadaLabs Embedding Model for Real-Time Multilingual Voice Bot Routing

Summary

Voice bots have evolved significantly with the adoption of AI, making conversations more natural and intelligent. While Large Language Models (LLMs) have enabled remarkable conversational capabilities, not every component of a voice bot benefits from using one. Some tasks demand speed above everything else.

One such task is conversation routing in flow-based voice bots. Every time a user responds, the platform must determine the most appropriate conversational path before generating the next response. This decision needs to be accurate, consistent, and completed within a fraction of a second.

For businesses serving users across India, this challenge becomes much more complex. Users respond in different languages, switch between native scripts and Romanized text, speak in very short phrases, and are often transcribed imperfectly by speech recognition systems.

To address these challenges, we developed the FonadaLabs Embedding Model, a lightweight multilingual ranking model designed specifically for real-time voice bot routing. Instead of focusing on general language understanding benchmarks, the model was trained on data that closely represents real production conversations, making it significantly more reliable for multilingual voice applications.

The Problem

Our platform enables businesses to create both LLM-powered conversational voice bots and flow-based voice bots. In a flow-based bot, every user response determines which conversational route should be followed next.

Consider a simple example.

A bot asks whether a customer is available to speak. Depending on the user's response, the conversation may continue, end, or be transferred to another flow. Although this appears to be a straightforward intent-matching problem, production environments introduce challenges that are far more difficult than traditional text classification.

Real-Time Decision Making

Once a user finishes speaking, the routing engine has only a very small time window to identify the correct transition before the bot generates its next response.

Our latency requirement was less than 100 milliseconds. Any delay directly affects the natural flow of the conversation and creates noticeable pauses for the user.

Large Language Models were not suitable for this requirement because they require significantly more computational resources and introduce additional inference latency. We needed a compact model capable of running locally while maintaining high ranking accuracy.

Extremely Short User Responses

Unlike search engines or chat applications, users interacting with voice bots rarely speak in complete sentences.

Our production dataset consisted of 668 routing utterances across 10 routing intents, with an average utterance length of only 2.9 words. Nearly 66% of all responses contained three words or fewer.

Typical user responses include:

• Haan

• Ji

• Nahi

• Boliye

• Theek hai

These responses provide very little contextual information, making semantic matching significantly more difficult than traditional sentence similarity tasks.

Speech Recognition Noise

Speech recognition systems are rarely perfect in real-world telephony environments.

For example, a user may say:

"हाँ बताइए"

The speech recognizer may transcribe it as:

"हा बताइ"

Small transcription errors such as missing diacritics, clipped words, or altered word boundaries can completely change the surface form of an utterance. The routing model must still recognize that both expressions represent the same intent.

Multilingual Conversations

India's linguistic diversity introduces another level of complexity.

The same affirmative response may appear as:

• haan

• हाँ

• ஆம்

• అవును

• ہاں

Many users also switch naturally between Romanized text and native scripts within the same conversation. A production-ready routing model therefore needs to understand multiple languages, multiple scripts, and multiple writing styles while remaining fast enough for real-time conversations.

Where Existing Models Fall Short

Before developing our own model, we evaluated several multilingual language models that are widely used for semantic similarity and retrieval tasks. Each model demonstrated strong performance on its intended benchmarks, but none was designed for the challenges presented by production voice conversations.

IndicBERT

IndicBERT provides excellent multilingual coverage across Indian languages and performs well on language understanding tasks such as classification and named entity recognition.

The challenge was that it was never designed as a ranking model. Its embeddings are not naturally optimized for semantic similarity. During our evaluation, extremely short responses such as "haan" and "nahi" often produced embeddings that were surprisingly similar because the model primarily recognized them as common Hindi words rather than opposite conversational intents.

Indic Sentence SBERT

Indic Sentence SBERT is specifically trained for sentence similarity, making it a stronger candidate for our use case.

It performed well on clean, medium-length sentences where sufficient contextual information was available. The performance dropped noticeably once utterances became very short because single-word responses simply did not provide enough information for the model to generate highly discriminative embeddings.

The model also struggled with mixed-script inputs where users switched between Romanized Hindi and Devanagari.

BAAI BGE-Reranker-v2-M3

Among the evaluated models, BAAI BGE-Reranker-v2-M3 proved to be the closest match to our requirements.

Unlike dual-encoder architectures, this model uses a cross-encoder architecture that processes the user query and candidate response together. This allows the model to evaluate fine-grained relationships between both inputs, making it particularly effective for ranking tasks.

Its performance on multilingual retrieval benchmarks such as MTEB is among the strongest within compact multilingual models.

When evaluated on our production-style dataset, the results revealed an important limitation. Responses such as "ji", "haan", or "nai" were frequently ranked incorrectly because the model had never been exposed to noisy ASR transcripts or extremely short spoken conversations during training.

This evaluation made one thing very clear. A model can perform exceptionally well on public benchmarks while still struggling with production traffic if the training data does not resemble real user interactions.

FonadaLabs Embedding Model

Instead of designing a completely new architecture, we chose to build upon the cross-encoder approach because it was the best architectural fit for ranking conversational responses.

Our primary focus shifted from changing the architecture to changing the data the model learned from.

The objective was simple.

Train the model on the same type of conversations it would encounter after deployment.

Training for Real ASR Conditions

Our first training experiments used clean text collected from routing examples. Offline evaluation looked promising, but production performance was noticeably lower.

The reason was straightforward.

Real speech recognition output is rarely clean.

To bridge this gap, we introduced realistic ASR noise directly into the training pipeline.

The augmentation strategy included:

• Removing diacritics

• Converting native scripts into Romanized text

• Introducing word boundary variations

• Adding conversational fillers

An important design decision was to apply these augmentations only to user queries. Candidate routing patterns remained unchanged because this accurately represents production traffic where noisy user speech is compared against predefined routing phrases.

This training strategy allowed the model to learn robust semantic matching even when speech recognition introduced transcription errors.

The Benchmark

Existing multilingual benchmarks primarily evaluate models on well-formed web documents, paragraphs, and search queries.

Production voice routing looks very different.

Users speak in short phrases. Speech recognition introduces errors. Multiple scripts appear within the same conversation. Intent boundaries are often separated by only a single word.

To evaluate models under these realistic conditions, we created a dedicated benchmark that reflects actual production traffic.

Overall Performance

Top-1 Accuracy indicates how often the correct route is selected as the highest-ranked candidate.

For production routing systems, another metric becomes equally important.

Perfect Ranking Rate measures whether every candidate is ordered correctly. This directly impacts confidence thresholds and fallback decisions during live conversations.

Pairwise AUC measures how consistently the model ranks the correct route above incorrect routes across all possible candidate pairs. A higher score indicates better overall ranking quality.

MRR (Mean Reciprocal Rank) measures how early the correct route appears in the ranked list. A higher MRR means the correct intent is consistently placed near the top, reducing routing errors and improving response speed.

Our model achieved a Perfect Ranking Rate of 78.3%, nearly doubling the performance of the strongest publicly available baseline.

Performance Across Different Difficulty Levels

The results demonstrate that the performance of general-purpose models decreases as transcription quality deteriorates. Our model remained consistently accurate because it was trained using realistic ASR errors from the beginning.

Performance Across Indian Languages

We also compared multilingual performance across eleven Indian languages.

The FonadaLabs Embedding Model achieved the highest score in 10 out of 11 languages. In Gujarati, it remained highly competitive, trailing the strongest baseline by only 1.8 percentage points.

These results demonstrate that combining multilingual architecture with production-specific training data produces significantly better performance than relying solely on large-scale multilingual pretraining.

Conclusion

Building a reliable voice routing system for India requires much more than selecting a multilingual language model. The model must understand extremely short responses, remain robust to speech recognition errors, support multiple languages and scripts, and deliver predictions within a fraction of a second.

Our evaluation showed that publicly available multilingual models perform well on the tasks they were originally designed for, but production voice routing presents a very different problem. The quality of the training data proved to be just as important as the underlying architecture.

By training a cross-encoder on real routing patterns, carefully selected hard negatives, and realistic ASR noise, we developed a model that consistently outperformed existing multilingual baselines across both accuracy and ranking quality.

The broader lesson extends beyond this project. Public benchmarks, model size, and multilingual support provide useful indicators, but they cannot predict production performance on their own. The most reliable way to evaluate any model is to benchmark it against the same type of data it will encounter after deployment. For multilingual voice AI, that means testing on short, noisy conversational speech instead of clean benchmark datasets.

That principle guided the development of the FonadaLabs Embedding Model and continues to shape how we evaluate language models for real-world voice applications.

References

https://huggingface.co/BAAI/bge-reranker-v2-m3
https://huggingface.co/ai4bharat/indic-bert
https://huggingface.co/l3cube-pune/indic-sentence-similarity-sbert