Whisper word level timestemp

[apssouza22]

I am wondering if it is possible to have word level timestamp when using the whisper example? I tried to include the param word_timestamps to the model options but it didn't work.

Any help in how to achieve this will be very welcome.
Thanks

[danieloi]

They don't appear to work with the distil models. the old xenova ones seem fine though less accurate for me than the sample for the distilled version here https://huggingface.co/spaces/Xenova/distil-whisper-web but it only gives sentence timestamps

[apssouza22]

@danieloi I managed to do it here https://github.com/apssouza22/video-text-edit

[danieloi]

@apssouza22 it works for word-level timestamps when you use the distil variants of the model?

[apssouza22]

Oh sorry, distilled didn't work, but I thought it was because the models are too big.
@danieloi

[danieloi]

Same here @apssouza22, I get this error when I want word-level timestamps and use the distil variants in transformers.js:

Error: Layer index 6 is out of bounds for cross attentions (length 4).
at webpack://@huggingface/transformers/./src/models.js:3498:1
at Array.map ()
at Function._extract_token_timestamps (webpack://@huggingface/transformers/./src/models.js:3496:30)
at Function.generate (webpack://@huggingface/transformers/./src/models.js:3442:1)
at async Function._call_whisper (webpack://@huggingface/transformers/./src/pipelines.js:1867:1)
... message: 'Layer index 6 is out of bounds for cross attentions (length 4).'}

[apssouza22]

@danieloi yeah. Got the same.
Let me know if you manage to get it sorted