Video

Tip

This feature requires av dependency. You can install it via pip install "docarray[full]".

Load video data

from docarray import Document

d = Document(uri='toy.mp4')
d.load_uri_to_video_tensor()

print(d.tensor.shape)

(250, 176, 320, 3)

For video data, .tensor is a 4-dim array, where the first dimension represents the frame id, or time. The last three dimensions represent the same thing as in image data. Here we got our d.tensor.shape=(250, 176, 320, 3), which means this video is sized in 176x320 and contains 250 frames. Based on the overall length of the video (10s), we can infer the framerate is around 250/10=25fps.

We can put each frame into a sub-Document in .chunks as use image sprite to visualize them.

for b in d.tensor:
    d.chunks.append(Document(tensor=b))

d.chunks.plot_image_sprites('mov.png')

Key frame extraction

From the sprite image you can observe our example video is quite redundant. Let’s extract the key frames from this video and see:

from docarray import Document

d = Document(uri='toy.mp4')
d.load_uri_to_video_tensor(only_keyframes=True)
print(d.tensor.shape)

(2, 176, 320, 3)

Looks like we only have two key frames, let’s dump them into images and see what do they look like.

for idx, c in enumerate(d.tensor):
    Document(tensor=c).save_image_tensor_to_file(f'chunk-{idx}.png')

Additionally, if only_keyframes=False the keyframe indices will be stored in .tags['keyframe_indices'] when you call .load_uri_to_video_tensor(). This way, you can easily access selected scenes:

from docarray import Document

d = Document(uri='toy.mp4')
d.load_uri_to_video_tensor()
first_scene = d.tensor[d.tags['keyframe_indices'][0]: d.tags['keyframe_indices'][1]]

print(first_scene.shape)

(95, 320, 176, 3)

Save as video file

You can also save a Document .tensor as a video file. In this example, we load our .mp4 video and store it into a 60fps video.

from docarray import Document

d = (
    Document(uri='toy.mp4')
    .load_uri_to_video_tensor()
    .save_video_tensor_to_file('60fps.mp4', 60)
)

Create Document from webcam

You can generate a stream of Documents from a webcam via generator_from_webcam():

from docarray import Document

for d in Document.generator_from_webcam():
    pass

This creates a generator that yields a Document for each frame. You can control the framerate via fps parameter. Note that the upper bound of the framerate is determined by the hardware of webcam, not the software. Press Esc to exit.