Voice Clone Studio

Voice Clone Studio is a local voice cloning workbench for English and Urdu. Drop in a short reference clip, tag the delivery, and the app keeps a reusable voice profile: audio, transcript, language and emotion references. Then synthesise speech in that voice, or talk to it.

Cloning and synthesis run on a 16 GB RTX 4060 Ti. Qwen3-TTS handles the main English path, Chatterbox Multilingual handles Urdu, XTTS v2 is the second TTS engine, and Whisper transcribes. Chat runs on DeepSeek by default, or Qalb-urdu through Ollama for a fully local loop. Reference audio and generated speech stay on the machine; the hosted chat model receives text only when it is selected.

Models

Qwen3-TTS 1.7B is the default English engine, with 0.6B as the faster option. Both are Apache-2.0, clone from a short clip zero-shot, and stream. XTTS v2 (Coqui) is the alternate engine when its voice profile fits better.

Urdu needed a different route. Qwen does not speak Urdu, so Urdu script is converted to Devanagari with ur2dev (eSpeak phonemes plus a word lexicon) and spoken by Chatterbox Multilingual with the Hindi tag. The installed weights are an accent fine-tune trained on nine hours from 149 native speakers and two narrators. Stock Chatterbox is the fallback.

Whisper large-v3, running through faster-whisper in float16, transcribes reference clips and chat turns. Qalb-urdu is Qalb-1.0-8B Q4_K_M through Ollama; it handles local translation and the local chat option. DeepSeek v4 flash is the default chat model.

Fine-tuning is per voice. The app segments the voice's own recordings, transcribes them with Whisper, converts the transcript to Devanagari, and trains a LoRA adapter on the Chatterbox Urdu model. It needs about 30 seconds of speech, saves a 90 MB adapter, and keeps the best validation checkpoint. Zero-shot still works when a voice has no adapter or the adapter is switched off.

Two fine-tunes came before that. The pilot used 4.6 hours from three speakers and learned one narrator's accent, which was audible. The accent fine-tune used nine hours from 149 speakers and two narrators. The automatic Urdu-to-Devanagari conversion also had to prove itself: Whisper read-back error was 11.4% against 10.5% for hand-written Devanagari, close enough to skip hand transliteration.

Problem

Hosted voice tools are convenient. They also send reference audio to someone else's server, keep the voice in an account instead of on my disk, and bill every generation. For a tool I wanted to use daily, all three were the wrong default.

The first version was a CLI. It worked and it was useless for daily use: every test meant editing a path, and every voice meant remembering which clip and transcript belonged together. A clone is only reusable if the reference is stored properly. The library is the product.

There is a quality problem that has nothing to do with the model. Cloning is zero-shot: the reference clip acts as a prompt, and the model is not trained on it. If the transcript does not match the clip, the output is garbage. If the clip is noisy, the clone inherits the noise. The interface has to show those facts instead of hiding them behind one Clone button.

Architecture

The backend is FastAPI. It exposes REST for the voice library and synthesis, and a WebSocket for real-time chat. The client is a React app built with TanStack Start, compiled to static files, and served by the same FastAPI process in production.

A voice profile is a directory on disk: reference clips, metadata, transcript, language, and which clip is the neutral identity versus which emotion it represents. A voice needs at least one neutral reference. Each emotion can have one clip, and neutral clips can be combined. The clone prompt is cached per voice and model.

A provider registry resolves each synthesis request to an engine, model and language route. English goes to Qwen3-TTS, Urdu goes to Chatterbox, and XTTS is available as the second TTS provider. A request that cannot be served by a provider with a profile for that voice fails instead of silently falling back to a worse route.

Synthesis and chat reuse the same profile. Synthesis takes an upload or browser recording, normalises it to 24 kHz mono, transcribes it with Whisper when needed, and returns a WAV. Chat takes 16 kHz frames over the WebSocket, runs voice activity detection, transcribes the turn, sends the text to DeepSeek or Qalb-urdu, and streams the reply through the selected TTS engine. Playback is queued. A debug panel shows the last speech-to-text time.

Challenges

The first spoken turn is slow by design. Whisper large-v3 is lazy-loaded so Qwen can keep the GPU until someone speaks. The first utterance in a server process pays the model load, roughly eight seconds on this card, on top of normal transcription. After that it stays warm and utterances transcribe in well under a second.

Reference quality decides the result. The common failure is a transcript that does not match the clip, a reference with music or reverb, or text in a language the model does not route well. The UI asks for the transcript, tags each clip by emotion, and refuses to build a voice without a neutral reference.

Urdu adds a conversion step and an accent risk. ur2dev has to preserve meaning, not just sounds, and Chatterbox only conditions on the first few seconds of the reference. A clean reference and an accurate transcript matter more on this route than on the English one.

The machine has a budget. The 1.7B Qwen model is comfortable for inference. The 0.6B is the faster option. Fine-tuning has to unload the resident models, run one job at a time, and keep the adapter small enough to leave the GPU usable.

Results

The result is a working local studio: a persistent voice library, multi-reference profiles, English and Urdu synthesis, an optional per-voice fine-tune, and a real-time chat loop. Outputs are WAV for editing and Ogg/Opus for messaging. The repository is private, and the app runs on my machine rather than at a public URL.

The part I actually use is the library. Coming back to a voice a week later, with the same reference and transcript, is the difference between a demo and a tool. The visuals on this page are generated studies rather than screenshots; the app is local, and the repository is private.

The clone and synthesis path is local. Chat is local only when Ollama and Qalb-urdu are selected; the default DeepSeek route sends the conversation text off the machine.

Lessons learned

Data quality beats model choice. Ten clean seconds with an exact transcript will beat a larger model fed a noisy clip. The interface should teach that instead of hiding it.

Try zero-shot before fine-tuning. Training is the expensive, fragile path, and it should be a deliberate upgrade for a specific voice. The pilot fine-tune also showed that the wrong data teaches the wrong accent, even when the loss improves.

One model did not cover both languages. Qwen handles English well, Chatterbox with a Devanagari conversion handles Urdu, and the app routes by language instead of pretending a single clone is uniform.