Architecting Conversational AI Systems

“A voice assistant is more than a speech recognizer attached to a search engine. It is a stateful entity that must navigate the social nuances of human turn-taking and intent.”

1. Introduction: Beyond Single-Shot Commands

The early era of voice was defined by Single-Shot Commands:

• “Hey, what’s the weather?”
• “Set a timer for 10 minutes.” The system performed a task and then immediately “forgot” the user. There was no memory, no context, and no conversation.

Conversational AI is the science of Multi-turn Interaction. It requires the system to maintain a Dialog State across minutes or even hours of conversation. It must handle interruptions, corrections, and anaphora (“What about it?”).

We architect a full-stack Conversational AI system, focusing on State-Driven Transitions and Persistent Persistence.

---

2. The Functional Blueprint: The Five Core Modules

1. ASR (Acoustic-to-Text): Translating vibrations into a list of candidate sentences.
2. NLU (Intent & Entity): Decoding the user’s meaning and parameters.
3. DST (Dialog State Tracking): The “Short-term Memory” of the current flow.
4. DP (Dialog Policy): The “Decision Maker” that decides the next system action.
5. NLG & TTS (Text-to-Speech): Generating a natural response and speaking it.

---

3. High-Level Architecture: The Latency Budget

In a voice conversation, a delay of > 700ms feels “awkward.” A delay of > 2000ms is a system failure.

Module	Latency Budget
ASR	200ms
NLU & DST	100ms
External API (e.g., Weather)	300ms
TTS (First chunk)	100ms
Total	700ms

To achieve this, we use Streaming Orchestration. NLU starts processing while the user is still speaking. TTS starts speaking while the audio is still being generated.

---

4. Implementation: The Dialog State Machine

In complex domains like flight booking, simple “intent matching” fails. We use a Frame-based State Machine.

class DialogManager:
    def __init__(self):
        self.state = "IDLE"
        self.slots = {"destination": None, "date": None}

    def process_turn(self, nlu_result):
        # 1. Update Slots from NLU
        for entity in nlu_result.entities:
            if entity.type in self.slots:
                self.slots[entity.type] = entity.value

        # 2. Transition Logic (The State Machine)
        if not self.slots['destination']:
            return "Where are you flying to?", "AWAIT_DESTINATION"
        
        if not self.slots['date']:
            return f"Understood. When do you want to fly to {self.slots['destination']}?", "AWAIT_DATE"
        
        return "Everything is set. Booking now.", "FINISHED"

---

5. Advanced: Handling “Speech Interruptions” (Barge-in)

How does a system know when to stop talking if the user interrupts?

• Full-Duplex Audio: The system is always “Listening” while “Speaking.”
• Acoustic Echo Cancellation (AEC): The system must “subtract” its own voice from the incoming audio stream.
• Latency Connection: If interrupt detection is slow, the system feels less intelligent.

---

6. Real-time Implementation: Contextual ASR

The ASR model shouldn’t just be a general model. It should be State-Aware.

• If the Dialog Manager is in the AWAIT_DATE state, the ASR’s Language Model should be weighted toward numbers and months.
• This “Contextual Boosting” reduces the WER significantly for ambiguous words.

---

7. Comparative Analysis: Rule-based vs. LLM-based Dialog

Metric	Rule-based (Rasa/Dialogflow)	LLM-based (GPT-4 / Claude)
Control	Absolute	Probabilistic
Consistency	High	Medium
Complexity	High (Human setup)	Low (Zero-shot)
Best For	Banking / Booking	General Chat / Therapy

---

8. Failure Modes in Conversational AI

1. Dialog State Collapse: The user changes their mind multiple times, and the state machine gets stuck.
   • Mitigation: Implement a “Reset State” command and confidence threshold.
2. Entity Confusion: Ambiguity in multi-entity sentences.
3. Vocal Sarcasm: Misinterpreting positive words used sarcastically.

---

9. Real-World Case Study: Amazon Alexa’s Contextual Reasoning

Alexa uses a specialized component for Short-term Goal Tracking.

• If you ask “How tall is the Eiffel Tower?” and then “When was it built?”, Alexa performs Coreference Resolution to map “It” back to “Eiffel Tower” from previous turns.

---

10. Key Takeaways

1. State is Soul: A conversation is a trajectory through a state space.
2. Latency is the UI: A slow voice system is a broken voice system.
3. Hybridization wins: Use LLMs for general chat but hard state machines for transactions.
4. Feedback Loops: Use data from failed turns to refine your state transitions.

---

Originally published at: arunbaby.com/speech-tech/0058-conversational-ai-system

If you found this helpful, consider sharing it with others who might benefit.