automatic-speech-recognition transcribe phonemes

[IanSweeneyAC]

Feature request

from the pipeline api examples

Example: Transcribe English w/ word-level timestamps.

would be nice to be able to transcribe audio and also get phoneme level timestamps, something like

const transcriber = await pipeline('automatic-speech-recognition', 'Xenova/whisper-tiny.en');
const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav';
const output = await transcriber(url, { return_timestamps: ['word', 'phonemes'] });

Motivation

having phoneme level timestamps would allow browser processing of audio to drive lip sync of a 3D avatar. assuming phoneme data is available as a building block of words in speech recognition.

Your contribution

I can animate a 3D character, eg ready player me, in three.js from audio samples if phoneme timestamps are available

[xenova]

Unfortunately, I don't believe whisper has this functionality. However, you could use a separate library, like https://www.npmjs.com/package/phonemizer to extract phonemes from the input text.