Date of Award

6-6-2026

Document Type

Thesis

Publisher

Santa Clara : Santa Clara University, 2026

Department

Computer Science and Engineering

First Advisor

Sean Choi

Abstract

This senior design thesis presents a machine learning research pipeline for child-parent and child-directed speech diarization. Speech diarization is the task of determining who spoke and when in an audio recording. Although this problem is well studied in adult meeting, broadcast, and conversational speech, child-parent audio is a more uneven setting. Children may speak in short turns, change pitch quickly, vocalize without producing clear words, interrupt adults, overlap with adult speech, cry, laugh, or code switch. These behaviors are normal in real family interaction, but they can confuse diarization systems that were mostly designed and evaluated on adult speech.

The project was completed as a solo senior design effort. Its goal was not to build a polished commercial product, but to create a reproducible workflow that could collect or import audio, organize files, create synthetic training mixtures, run diarization backends, evaluate model output, and document the results clearly. The original plan relied more heavily on manual labeling and fine-tuning. After early delays caused by team and project-direction changes, the actual implementation began in January, and later work was further affected by limited labeled data and the loss of some manually created labels during an SSH-related workflow problem. These constraints shifted the final design toward synthetic data and controlled experimentation.

The final pipeline uses adult speech, child speech, and infant vocalization categories to create or evaluate diarization output. It uses existing tools such as pyannote.audio and NVIDIA NeMo as diarization backends, with supporting scripts for audio numbering, synthetic stitching, review bundles, metrics, and optional dashboard-based workflow management. The project evaluates selected adult-to-adult, adult-to-child, and adult-to-infant clips using Diarization Error Rate and Jaccard Error Rate. The results were mixed: synthetic-data-based NeMo performed better on some clips, while pyannote performed better on the infant-focused case. This outcome is useful because it shows that child-parent diarization is not solved by simply applying an adult speech model to a new domain.

The main contribution of this thesis is a clean and documented senior design prototype for studying child-parent diarization under realistic constraints. It also argues that synthetic data is a practical bridge when manually labeled child-parent audio is scarce, costly, or difficult to preserve. Future work should expand the real labeled dataset, compare synthetic-only, synthetic-plus-real, and real-only training strategies more systematically, improve overlap handling, and add better support for multilingual or code-switched child-directed speech.

Download

Share

COinS