💡 Proposal: Add temporal-grounding pipeline for video-language tasks

[mreraser]

Feature request

Hi 🤗 team and contributors,

I'm currently exploring ways to extend the transformers library to support temporal grounding — the task of identifying a [start, end] timestamp segment in a video given a natural language query.

While Hugging Face already supports pipelines like video-classification, image-to-text, and zero-shot-image-classification, it seems there is currently no pipeline or task definition for video moment retrieval / temporal grounding tasks.

Motivation

As multimodal models become increasingly capable of understanding both vision and language (e.g., BLIP2, VideoChatGPT, TimeChat), there is a growing demand for models that can not only recognize what is happening in a video, but also when it happens.

Temporal Grounding — the task of identifying a relevant moment span [start, end] in a video given a natural language query — is a fundamental step in making video-language models temporally aware.

For example:
Given a query like "the person starts cooking", a temporal grounding model is expected to localize the clip where this action occurs. --> [3.5, 8.9]

This capability is critical for a wide range of downstream tasks:

• Video Question Answering (When does X happen?)
• Video Summarization and Highlighting
• Instruction-following agents in videos

Your contribution

I’d love to know if there is any ongoing work on this?

If not, I’d love to propose an initiative to explore what it might look like to support temporal grounding as a task within the transformers library — either through a new pipeline or modular components that make it easier to build moment retrieval models using Hugging Face tools.

I’m also curious if any other contributors might be interested in collaborating on this idea. I’d be very happy to contribute and work with others who are exploring similar directions.

[Rocketknight1]

Hi @mreraser, this is an interesting proposal! I would definitely be interested in one or more pipelines for these tasks, but I recommend you start by identifying pre-trained models that have good performance at these tasks. Pipelines are intended to be an easy, high-level API that's accessible for new users, so usually we try to make sure they have a good default model that works without lots of tweaks

cc @zucchini-nlp as well, who works on vision/video LMs

[zucchini-nlp]

Great proposal! Currently we don't have video-text pipelines though we might extend an existing image-text to work with any visual input.

I am curious, for Temporal Grounding we would need models specifically tuned for this task right? AFAIK none of the models in the core library are good at temporal grounding. I would also definitely be interested if you can explore it further and find robust pre-trained models for this task or if we can use existing models. And if you need any assistance, just lmk 🤗

[mreraser]

Hi @Rocketknight1 and @zucchini-nlp!
Thank you both so much for your kind support and for showing interest in this idea — I truly appreciate it!

As you’ve pointed out, robust pre-trained models tailored for temporal grounding will be essential for building a meaningful pipeline. I’ll look into existing models that perform well on this task and also explore ways we might adapt current models to support it.

Once I’ve gathered some findings, I’ll share them here for further discussion.

Thanks again!

[AnimeshMaheshwari22]

I would like to contribute to this! Can contribute to any research/code for the task.