Kino element playing video and audio

[Janix4000]

Hey, I’m from Membrane Framework. We wanted to create interactive examples of our media pipelines in a livebook
However, as far as I checked livebook and Kino extension, there is no sufficient way to feed raw video/audio to any Kino element and show it in real-time (some type of video and audio player).
(possible workaround is to use https://hexdocs.pm/kino/Kino.html#animate/3, however Kino.Image.new accepts many standard image formats but raw YUV format. Additionally, converting raw frames to pictures would be overkill in the real-time pipeline)
Ergo, we decided to create our own Kino element supporting manual streaming (and maybe movies from files and native streams)
Should we contribute this extension (when finished) directly to Kino or create a separate package for it? (edited)

[josevalim]

Hi @Janix4000! If it is an output Kino (i.e. it is rendering audio/video), then I would definitely start as a separate package, this will give you more flexibility to iterate quickly. But please do share it once it is complete. Then once you consider it robust and stable enough, please ping us again.

Inputs are a bit different and it is best if they are part of Livebook+Kino themselves.

[jonatanklosko]

Inputs are a bit different and it is best if they are part of Livebook+Kino themselves.

This goes for stateful inputs that have a specific value at a time, but in case of streaming that's not really the case so a Kino.JS.Live with some subscription API sounds like the way to go.

@Janix4000 in case it's at all helpful there is an example camera component in this repo :) Note that it distinguishes browser clients, so if you open a notebook in two tabs (or two users collaborating in the cloud) it shows separate feed for each; multiple clients is just something to keep in mind with Kino components.

[fazibear]

Just implemented such a thing here: livebook-dev/kino#311
but wondering if it should be implemented in livebook itself?