15 minute read

The moment a voice agent’s TTS model causes an OOM on the GPU that was running fine yesterday — because the conversation got longer, because you added a new agent, because the batch size grew — you start caring about memory in a way that benchmarks never prepared you for.

Benchmarks show you MOS scores and WER. They don’t show you the 3 AM alert when a long-running session spills past the KV cache budget and your TTS process dies mid-sentence. They don’t show you the spreadsheet where you discover the next GPU tier costs $2,400 more per month, all because your TTS model needs 8 GB that your LLM backend left no room for.

MamTra (arXiv:2603.12342, KAIST and Chung-Ang University, March 2026) is an attempt to solve exactly this problem — not by quantizing weights or pruning layers, but by rethinking which architectural primitive handles which part of the sequence.

TL;DR

MamTra replaces half the Transformer decoder blocks in a TTS model with Mamba selective state-space layers. The result: 34% less inference VRAM versus CosyVoice 2, NMOS dropping 0.02 to 3.66, WER rising 0.25 percentage points to 2.28%. For co-located TTS+LLM stacks on a single GPU, that margin is often the difference between one machine and two. See TTS system fundamentals for the background on Transformer TTS architecture before diving in.

A GPU server rack with alternating cards of two distinct designs — every other slot holding a slimmer module with fewer cooling fins, representing ...

Why Transformer TTS is memory-expensive

Autoregressive Transformer TTS generates one acoustic token at a time, and every new token attends to every previous token. At 2,048 context length, that attention matrix grows to 1.64 TFLOPs of compute and a KV cache that scales quadratically with sequence length — not linearly.

The math is straightforward and unfriendly. Attention’s complexity is O(n²) in both time and memory with respect to sequence length n. For a sentence of average length, this is manageable. For a long paragraph, or a multi-turn conversation where TTS context accumulates across turns, the KV cache becomes the problem.

Modern LLM-based TTS systems like CosyVoice 2 are the worst offenders. They model speech as a sequence of discrete acoustic tokens — typically at 75–150 tokens per second of audio. A 30-second synthesis run generates 2,250–4,500 tokens. Every one of those tokens maintains keys and values for all previous tokens. The cache grows with each token generated, and on a GPU already shared with an LLM backbone, that growth has nowhere to go.

A typical co-located deployment — Whisper Large v3 at 5–8 GB FP16, an 11B LLM at roughly 22 GB, and a Transformer TTS model at 6+ GB — lands at 33–36 GB total, right at the A100 40GB limit before KV cache growth pushes you over. The fix is usually to either drop to a smaller LLM, add a second GPU, or accept degraded context. None of those are free.

Transformer TTS: KV cache growth vs sequence length

Context length:    512     1024    2048    4096
KV cache (GB):     0.8     1.6     3.2     12.8   ← quadratic explosion

MamTra’s pitch is that the quadratic growth is not inevitable — it’s a consequence of using full attention everywhere, including in layers where long-range global context isn’t what’s actually needed.

What MamTra does: where Mamba replaces attention and where it doesn’t

MamTra uses an interleaved architecture where Transformer attention blocks handle global semantic context and Mamba selective state-space blocks handle local acoustic modeling. In the default 1:1 configuration, 12 of 24 decoder layers are Mamba and 12 remain Transformer — placed with Transformer layers at the front (BlockBeg strategy).

Full Transformer TTS decoder (24 layers):
┌─────────────────────────────────────────────────────┐
│  [T] [T] [T] [T] [T] [T] [T] [T] [T] [T] [T] [T]  │
│  [T] [T] [T] [T] [T] [T] [T] [T] [T] [T] [T] [T]  │
└─────────────────────────────────────────────────────┘
All 24 layers use full self-attention → quadratic KV growth

MamTra 1:1 (BlockBeg configuration):
┌─────────────────────────────────────────────────────┐
│  [T] [T] [T] [T] [T] [T] [T] [T] [T] [T] [T] [T]  │  ← Transformer (global context)
│  [M] [M] [M] [M] [M] [M] [M] [M] [M] [M] [M] [M]  │  ← Mamba (local acoustic)
└─────────────────────────────────────────────────────┘
12 Transformer + 12 Mamba → sub-linear KV growth

T = Transformer attention block (O(n²) memory)
M = Mamba selective state-space block (O(1) memory per step)

The key design question is: why keep any Transformer blocks at all? The answer comes from an empirical finding in the paper. Pure Mamba models lose on multimodal data — specifically, the encoder-decoder cross-attention between acoustic features and textual features. Mamba processes sequences unidirectionally and struggles with the non-causal attention that TTS needs to align text representations with generated audio frames. Transformer blocks at the front of the decoder handle that alignment. Mamba blocks later in the stack handle the sequential acoustic pattern generation where the linear recurrence actually excels.

The BlockBeg placement (Transformer blocks first, Mamba after) consistently outperforms alternatives — Front, Middle, Back, Sandwich — across Mamba ratios. The intuition is that early layers need the global view to establish the semantic and prosodic trajectory, and later layers fill in the acoustic detail sequentially.

Mamba’s O(1) memory per generation step is what creates the VRAM savings. Unlike attention, which must store all previous keys and values, Mamba compresses the entire history into a fixed-size hidden state. That state updates at each step but doesn’t grow. At 2,048 context length, replacing 12 of 24 attention blocks with Mamba cuts 1.4×10¹¹ FLOPs per token and reduces average inference memory by 34% versus CosyVoice 2 and 17% versus Zonos-v0.1.

Training the hybrid without starting from scratch uses three-component distillation: cross-entropy on ground-truth tokens, Skew KL divergence between teacher and student logits, and MSE on token embeddings. This transfers the Transformer teacher’s learned representations into the hybrid student — which is why MamTra reaches competitive quality training on approximately 0.5k hours from LibriTTS, roughly 2% of the original training data.

The quality-memory tradeoff: what 34% savings costs you

MamTra 1:1 scores NMOS 3.66 vs CosyVoice 2’s 3.68 — a 0.02 difference that falls within measurement noise — with WER rising from 2.03% to 2.28%, a 0.25 percentage point absolute increase. The quality cost is real but small at the 1:1 ratio. It becomes meaningful at 1:3 and significant at 1:11.

Configuration VRAM reduction NMOS WER TFLOPs
CosyVoice 2 (teacher) baseline 3.68 ± 0.16 2.03% 1.78
MamTra 1:1 34% 3.66 ± 0.16 2.28% 1.64
MamTra 1:3 ~45% (est.) degrades 2.53% ~1.58
MamTra 1:11 ~55% (est.) audible degradation higher 1.52
Zonos-v0.1 17% vs Zonos 3.18 ± 0.18 3.42% —

The UTMOS scores — an automated naturalness metric from the Seed-TTS evaluation set — tell a similar story. MamTra 1:1 scores 4.13–4.16, CosyVoice 2 scores 4.15. That’s essentially a tie. Speaker similarity (SSIM) holds at 0.72, matching the teacher.

Where does quality go first? Word error rate. The 0.25 percentage point WER increase at 1:1 is the first casualty of replacing attention with Mamba. It’s small — and it’s still substantially better than Zonos-v0.1’s 3.42% — but it tells you that the Mamba layers are slightly less precise at articulation than full attention. At 1:3, WER creeps to 2.53%. At 1:11, the memory savings are largest but the model begins to sound mechanical in a way that’s hard to measure and easy to notice.

The honest takeaway: if you’re comparing to CosyVoice 2 as your baseline, the 1:1 hybrid is a genuine free lunch at the naturalness level. WER being 0.25% higher matters more in high-accuracy applications (medical, legal dictation) than in casual voice agent interactions where slight imprecision is acceptable.

There’s one gap in the benchmarks worth noting: the paper reports FLOPs and VRAM savings but not wall-clock RTF (real-time factor). Whether the linear-time Mamba blocks translate to measurable latency reduction on real hardware — especially on A100s or H100s where CUDA attention kernels are highly optimized — isn’t answered here. For latency-critical applications, you’d want to benchmark RTF independently.

Deployment decision guide: when MamTra changes the calculus

MamTra 1:1 saves roughly 2–3 GB of VRAM versus CosyVoice 2 at typical TTS model scales. That’s meaningful when the GPU is already under pressure — specifically in co-located TTS+LLM stacks, high-concurrency multi-stream deployments, or constrained edge hardware. It’s less compelling when TTS already runs on a dedicated node with headroom.

flowchart TD
    A[New TTS deployment] --> B{Is TTS co-located\nwith an LLM?}
    B -->|Yes| C{Does TTS+LLM\nfit current GPU tier?}
    B -->|No| D{Running 4+ concurrent\nTTS streams?}
    C -->|No - OOM risk| E[MamTra 1:1\nHigh value ✓]
    C -->|Yes - with headroom| F{How much headroom?}
    F -->|< 3 GB margin| G[MamTra 1:1\nPreventive ✓]
    F -->|> 3 GB margin| H[Full Transformer\nStay put]
    D -->|Yes| I{Quality requirement?}
    D -->|No| H
    I -->|Casual / assistant| J[MamTra 1:1\nHigh value ✓]
    I -->|Medical / legal| K[Evaluate WER carefully\nMay not be worth it]
    J --> L[Deploy with\nBlockBeg 1:1 config]
    G --> L
    E --> L

Co-located TTS + LLM (the primary use case). A voice agent stack combining Whisper Large v3, an 11B LLM, and CosyVoice 2 uses roughly 33–36 GB of VRAM before KV cache growth. An A100 40GB has 4–7 GB of headroom — which disappears as conversation context grows. Replacing CosyVoice 2 with MamTra 1:1 recovers 2–3 GB, moving you from OOM-risky territory to stable. For voice agents where long conversations are common, this isn’t a minor optimization — it’s the difference between reliable operation and intermittent failures.

Multi-stream TTS on a single GPU. If you’re running four concurrent TTS streams on an A100, 34% VRAM reduction per stream may let you run five or six instead, improving GPU utilization without adding hardware. The economics depend on your traffic patterns, but at scale the per-stream savings compound.

Edge and resource-constrained hardware. The Qwen3-TTS 1.7B model already fits on a 24 GB consumer GPU, but for higher-quality models in the 500M–1B parameter range, MamTra’s efficiency matters more at the edge than in data centers. The 2% training data requirement also matters for fine-tuning: adapting MamTra to a new speaker or language requires a fraction of what full Transformer TTS needs.

When to stay with full Transformer. If TTS runs on a dedicated GPU node with room to spare, MamTra’s savings don’t justify the 0.25% WER increase. If you’re already using Q4_K_M quantization (which reduces VRAM by roughly 75% at some quality cost), MamTra adds marginal benefit. For applications requiring high articulation accuracy — medical transcription, legal dictation, audiobooks — the WER tradeoff warrants careful evaluation rather than automatic adoption.

The broader question MamTra raises is architectural. The field has been treating Transformer-based TTS as essentially fixed — tune the training data, tune the prompting, optimize the inference kernel. MamTra is a signal that the decoder architecture itself is still in play. At Interspeech 2026, this is one of several papers (see also the TM-Speech work showing 2× size reduction and 3× training speed using Mamba-Transformer hybrids) suggesting that selective state spaces will become a standard component of production TTS, not an experimental curiosity.

For the full picture of where TTS sits in 2026 — including Voxtral, ElevenLabs v3, and the emerging voice cloning landscape — see the Voxtral and the 2026 TTS landscape post. For GPU memory budgeting in the broader voice agent stack, see voice agent architecture.

Key takeaways

  • MamTra 1:1 is 34% less VRAM than CosyVoice 2 with a 0.02 NMOS drop and 0.25% WER increase — a small quality cost for a large memory gain
  • The architecture is an interleaved hybrid: Transformer blocks at the front handle global text-speech alignment, Mamba blocks at the back handle sequential acoustic generation
  • Training uses three-component distillation from a Transformer teacher, enabling competitive quality with only 2% of the original training data
  • The 1:1 ratio is the deployment-safe choice — more Mamba-heavy variants (1:3, 1:11) save more memory but quality degrades meaningfully
  • The savings matter most for co-located stacks — dedicated TTS nodes with GPU headroom don’t benefit enough to justify the WER tradeoff
  • Wall-clock RTF isn’t reported — FLOPs and VRAM savings don’t automatically translate to lower latency on optimized hardware; benchmark before assuming

FAQ

What is MamTra and how does it differ from standard Transformer TTS?

MamTra (arXiv:2603.12342, KAIST and Chung-Ang University, March 2026) replaces a portion of Transformer attention blocks in TTS decoders with Mamba selective state-space layers. Where Transformer attention scales quadratically with sequence length, Mamba processes sequences in linear time. The default 1:1 ratio uses 12 Mamba and 12 Transformer layers from a 24-layer base, reducing inference VRAM by 34% vs CosyVoice 2 while keeping NMOS at 3.66 vs the teacher’s 3.68.

Does MamTra compromise audio quality?

The quality drop is small at the 1:1 ratio. On the Seed-TTS evaluation set, MamTra 1:1 scores NMOS 3.66 vs CosyVoice 2’s 3.68 — a 0.02 difference within measurement noise. WER rises from 2.03% to 2.28%, a 0.25 percentage point increase. UTMOS scores (4.13–4.16) match CosyVoice 2 (4.15). Speaker similarity holds at 0.72. MamTra 1:11 — the most Mamba-heavy variant — degrades more noticeably, making 1:1 the practical deployment choice.

How does MamTra’s knowledge distillation work?

MamTra trains with three simultaneous loss signals from a pretrained Transformer teacher: cross-entropy supervision on ground-truth tokens, Skew KL divergence between teacher and student logits, and MSE on token embeddings. This three-component approach lets MamTra reach competitive quality on roughly 0.5k hours of LibriTTS — about 2% of what the teacher required for full pretraining — because the student inherits learned representations rather than discovering them from scratch.

When does MamTra’s 34% VRAM saving change your deployment decision?

The saving matters most in three scenarios: (1) co-located TTS+LLM stacks where the LLM already consumes most of the GPU’s VRAM, (2) multi-stream deployments where 4–8 concurrent TTS instances share a single A100, and (3) edge or cloud instances where the next GPU tier carries a significant cost jump. It matters less when TTS runs on a dedicated node or when you’re already using Q4_K_M quantization, which provides larger memory reduction at a different quality tradeoff.

What is the Mamba-Transformer ratio, and which should I use?

The ratio describes how many Transformer layers remain relative to Mamba layers in a 24-layer decoder. MamTra 1:1 (12 Transformer, 12 Mamba) gives the best quality-memory balance: 34% VRAM reduction with negligible quality loss. MamTra 1:3 (6 Transformer, 18 Mamba) saves more but quality starts to slip. MamTra 1:11 (2 Transformer, 22 Mamba) is most memory-efficient but the quality degradation becomes audible. For production deployments, 1:1 is the safe choice.

Further reading

Want to work together?

I take on projects, advisory roles, and fractional CTO engagements in AI/ML. I also help businesses go AI-native with agentic workflows and agent orchestration.

Get in touch

Related across topics