Tencent Hunyuan Releases Hy-SimulMT for Real-Time Bilingual Speech Translation and Voice Cloning

Tencent Hunyuan has released Hy-SimulMT, a real-time simultaneous translation model supporting bidirectional speech-to-speech translation between Chinese and English alongside real-time voice cloning. Designed for live communication settings such as meetings, speeches, and interviews, the model processes incoming audio streams to generate both translated text and synthesized speech while preserving the original speaker's vocal characteristics.

Performance and Benchmarks

To balance translation quality and latency, the developers evaluated Hy-SimulMT across public benchmarks and real-world audio samples:

• Long-Form Audio: In the RealSI evaluation, Hy-SimulMT recorded a StreamLAAL of approximately 3 seconds, achieving the lowest lag among compared systems.
• Quality Scores: Its GEMBA translation quality score surpassed those of tested open-source models and four out of five commercial APIs.
• Short-Form Datasets: The model scored higher than other listed open-source models on FLEURS, ACL6060, and RealSI-Sentence benchmarks.
• Expert Comparisons: In an evaluation involving 18 online audio recordings across ten domains compared against expert human interpreters holding CATTI Level 1 and 2 certificates, the system demonstrated competitive translation performance.

Deployment and Tools

The project provides inference services and training tools via its repository:

• SimulInfer: Offers web interfaces, audio file uploads, real-time WebSockets, and optional TTS integration with input voice cloning.
• Llama.cpp Integration: A pull request (#31198) has been submitted to add architecture-specific support for GGUF model inference using `llama-mtmd-cli`.
• SGLang-Omni Runtime: Serves the `Hy-SimulMT-30B-A3B` checkpoint through OpenAI-compatible HTTP endpoints and real-time WebSockets.
• SimulTrain: Provides capabilities to continue training from downloaded checkpoints using customized environment configurations.

Verification and Expert Feedback

Academic figures, including representatives from Guangdong University of Foreign Studies, noted that the system's iterative correction mechanism resembles human self-corrections, marking progress for AI simultaneous interpreting in low-risk settings. Users can verify the official announcement and access resources through the Tencent Hunyuan GitHub repository, Hugging Face, ModelScope, or by trying the Hy Translate App.