Date of Award

6-2-2026

Document Type

Thesis

Publisher

Santa Clara : Santa Clara University, 2026

Department

Computer Science and Engineering

First Advisor

Yuhong Liu

Second Advisor

Benham Dezfouli

Abstract

Language barriers remain a significant obstacle to natural human communication, limiting access to healthcare, business, travel, and daily social interaction for billions of people worldwide. Existing translation solutions such as smartphone applications and earpiece devices address the functional problem of converting words between languages but fail to preserve the speaker’s voice identity, require hands-on interaction, or depend on proprietary device ecosystems. These limitations disrupt the natural flow of conversation and reduce the human quality of cross-lingual communication.

Mimica is a real-time wearable translation necklace designed to address these shortcomings. Built around an ESP32-S3 microcontroller with an integrated microphone and speaker, the device captures speech, streams it to a cloud pipeline via Wi-Fi, and plays back the translated audio through the built-in speaker, entirely hands-free. The cloud pipeline combines ByteDance ASR for speech recognition, Helsinki-NLP MarianMT for machine translation, and Coqui XTTS v2 for voice cloning synthesis, deployed on an AWS EC2 GPU instance inside a custom Docker environment. The system preserves the speaker’s voice identity across languages by cloning the user’s voice from a short reference sample uploaded once at setup. To optimize performance, four pipeline configurations were evaluated across latency and voice quality metrics, achieving a best-case end-to-end processing time of 3.36 seconds while maintaining a voice similarity score of 83% and a speech clarity rate of 86%. The final system demonstrates that realtime, voice-preserving, hands-free translation is achievable on embedded hardware, and establishes a foundation for future work including custom PCB miniaturization, on-device configuration, and bidirectional conversation support.

Download

Share

COinS