Construct

Tip

In DocArray, a Document object can contain sub-Documents in .chunks. If you’re unaware of this design, read this chapter before continuing.

Just like the Python dataclasses module, DocArray provides a dataclass() decorator and a set of type annotations in docarray.typing like Image, Text, Audio, that let you construct a multimodal Document:

from docarray import dataclass
from docarray.typing import Image, Text


@dataclass
class MyMultiModalDoc:
    avatar: Image
    description: Text


m = MyMultiModalDoc(avatar='test-1.jpeg', description='hello, world')

Each field is a modality. The above example contains two modalities: image and text.

Caution

Be careful when assigning names to your modalities.

Don’t use names that are properties of Document, like text, tensor, embedding, etc. Instead, use more specific names that fit your domain, like avatar and description in the example above.

If there’s a conflict between a modality name and a Document property, there may be unexpected behavior when accessing such a name.

To convert a MyMultiModalDoc to a Document object, simply:

from docarray import Document

d = Document(m)
d.summary()

This creates a Document object with two chunks:

Nested structure (chunks)

📄 Document: f3b193bbe8403c3ce1599b82f941f68a
└── 💠 Chunks
    ├── 📄 Document: 18c7ca1c829fe819250faa4914bc45c1
    │   ╭──────────────┬───────────────────────────────────────────────────────────────╮
    │   │ Attribute    │ Value                                                         │
    │   ├──────────────┼───────────────────────────────────────────────────────────────┤
    │   │ parent_id    │ f3b193bbe8403c3ce1599b82f941f68a                              │
    │   │ granularity  │ 1                                                             │
    │   │ tensor       │ <class 'numpy.ndarray'> in shape (504, 504, 3), dtype: uint8  │
    │   │ mime_type    │ image/jpeg                                                    │
    │   │ uri          │ test-1.jpeg                                                   │
    │   │ modality     │ image                                                         │
    │   ╰──────────────┴───────────────────────────────────────────────────────────────╯
    └── 📄 Document: 1ee7fadddc23fc72365b2069f82d4bb4
        ╭──────────────────────┬───────────────────────────────────────────────────────╮
        │ Attribute            │ Value                                                 │
        ├──────────────────────┼───────────────────────────────────────────────────────┤
        │ parent_id            │ f3b193bbe8403c3ce1599b82f941f68a                      │
        │ granularity          │ 1                                                     │
        │ text                 │ hello, world                                          │
        │ modality             │ text                                                  │
        ╰──────────────────────┴───────────────────────────────────────────────────────╯

To convert a Document object back to a MyMultiModalDoc object:

m = MyMultiModalDoc(d)

Dataclass decorator

First, import dataclass decorator from the DocArray package:

from docarray import dataclass

The look and feel of this dataclass decorator is the same as the built-in dataclass decorator. In fact, any class wrapped by docarray.dataclass is also a valid Python dataclass:

from dataclasses import is_dataclass

from docarray import dataclass, Image


@dataclass
class MMDoc:
    banner: Image = 'test-1.jpeg'


print(is_dataclass(MMDoc))
print(is_dataclass(MMDoc()))

True
True

That means, arguments accepted by standard dataclass are also accepted here. Methods that can be applied to Python’s dataclass can be also be applied to DocArray’s dataclass.

To tell if a class or object is a DocArray dataclass, you can use is_multimodal():

from docarray.typing import Image
import dataclasses

import docarray
from docarray.dataclasses import is_multimodal


@docarray.dataclass
class MMDoc1:
    banner: Image = 'test-1.jpeg'


@dataclasses.dataclass
class MMDoc2:
    banner: Image = 'test-1.jpeg'


print(is_multimodal(MMDoc1))
print(is_multimodal(MMDoc2))

True
False

Moving forwards, unless otherwise specified, dataclass always refers to docarray.dataclass, not Python’s built-in dataclass.

Annotate class fields

DocArray provides docarray.typing that allows you to annotate class fields as

• Image, Text, JSON, Audio, Video, Mesh, Tabular, Blob

• primitive Python types

• other docarray.dataclass

from docarray import dataclass
from docarray.typing import Image, Text, Audio


@dataclass
class MMDoc2:
    banner: Image = 'test-1.jpeg'
    summary: Text = 'This is an empty test image in 256 x 256 pixels'
    soundfx: Audio = 'white-noise.wav'

Converting MMDoc2 object into a Document object is easy:

from docarray import Document

m = MMDoc2()
d = Document(m)

You can look at the structure of d via d.summary():

Nested structure (chunks)

📄 Document: 90c744c5155c2356d27f8c91955f70f7
└── 💠 Chunks
    ├── 📄 Document: c9d71990088fb0d8db3c83a6bd35650d
    │   ╭──────────────┬───────────────────────────────────────────────────────────────╮
    │   │ Attribute    │ Value                                                         │
    │   ├──────────────┼───────────────────────────────────────────────────────────────┤
    │   │ parent_id    │ 90c744c5155c2356d27f8c91955f70f7                              │
    │   │ granularity  │ 1                                                             │
    │   │ tensor       │ <class 'numpy.ndarray'> in shape (504, 504, 3), dtype: uint8  │
    │   │ mime_type    │ image/jpeg                                                    │
    │   │ uri          │ test-1.jpeg                                                   │
    │   │ modality     │ image                                                         │
    │   ╰──────────────┴───────────────────────────────────────────────────────────────╯
    ├── 📄 Document: 22e27708288e70813673711c86f834ee
    │   ╭─────────────────┬────────────────────────────────────────────────────────────╮
    │   │ Attribute       │ Value                                                      │
    │   ├─────────────────┼────────────────────────────────────────────────────────────┤
    │   │ parent_id       │ 90c744c5155c2356d27f8c91955f70f7                           │
    │   │ granularity     │ 1                                                          │
    │   │ text            │ This is an empty test image in 256 x 256 pixels            │
    │   │ modality        │ text                                                       │
    │   ╰─────────────────┴────────────────────────────────────────────────────────────╯
    └── 📄 Document: 05ad36dfb0c520027b18c582d205c176
        ╭──────────────┬───────────────────────────────────────────────────────────────╮
        │ Attribute    │ Value                                                         │
        ├──────────────┼───────────────────────────────────────────────────────────────┤
        │ parent_id    │ 90c744c5155c2356d27f8c91955f70f7                              │
        │ granularity  │ 1                                                             │
        │ tensor       │ <class 'numpy.ndarray'> in shape (63248,), dtype: float32     │
        │ modality     │ audio                                                         │
        ╰──────────────┴───────────────────────────────────────────────────────────────╯

Behavior of field annotation

This section explains the behavior of field annotations in detail.

• A dataclass corresponds to a Document object, let’s call it root.

• Unannotated fields are ignored.

  Field without type annotation

  from docarray import dataclass, Document
  
  
  @dataclass
  class Doc:
      a = 1
      b = 'hello'
  
  
  Document(Doc()).summary()
  

  Document structure

  📄 Document: 17c77b443471f9d752cbcc360174b65f
  

• A class field annotated as a Python primitive data type will be put into root.tags.

  Fields in primitive type

  from docarray import dataclass, Document
  
  
  @dataclass
  class MMDoc:
      some_field: str = 'hello'
      other_field: int = 1
  
  
  Document(MMDoc()).summary()
  

  Document structure

  📄 Document: 15725d705b6c8d7e99908d380d614fa5
  ╭────────────────┬─────────────────────────────────────────────────────────────╮
  │ Attribute      │ Value                                                       │
  ├────────────────┼─────────────────────────────────────────────────────────────┤
  │ tags           │ {'some_field': 'hello', 'other_field': 1}                   │
  ╰────────────────┴─────────────────────────────────────────────────────────────╯
  

• A class field annotated as a DocArray type will create a sub-Document nested under root.chunks.

  Field in DocArray type

  from docarray import dataclass, Document
  from docarray.typing import Image
  
  
  @dataclass
  class MMDoc:
      some_field: str = 'hello'
      banner: Image = 'test-1.jpeg'
  
  
  Document(MMDoc()).summary()
  

  Document structure

  📄 Document: 48a84621d51d94383b86db89e64022a3
  ╭────────────────────────┬─────────────────────────────────────────────────────╮
  │ Attribute              │ Value                                               │
  ├────────────────────────┼─────────────────────────────────────────────────────┤
  │ tags                   │ {'some_field': 'hello'}                             │
  ╰────────────────────────┴─────────────────────────────────────────────────────╯
  └── 💠 Chunks
      └── 📄 Document: 1cb5cc74f1f986876a0c4c87201b9a28
          ╭──────────────┬───────────────────────────────────────────────────────────────╮
          │ Attribute    │ Value                                                         │
          ├──────────────┼───────────────────────────────────────────────────────────────┤
          │ parent_id    │ 48a84621d51d94383b86db89e64022a3                              │
          │ granularity  │ 1                                                             │
          │ tensor       │ <class 'numpy.ndarray'> in shape (504, 504, 3), dtype: uint8  │
          │ mime_type    │ image/jpeg                                                    │
          │ uri          │ test-1.jpeg                                                   │
          │ modality     │ image                                                         │
          ╰──────────────┴───────────────────────────────────────────────────────────────╯
  

• The annotation type determines how the sub-Document is constructed. For example, annotating a field as Image will instruct the construction to fill in doc.tensor by reading the image URI. Annotating a field as JSON will instruct the construction to fill in doc.tags. The complete behavior table can be found below:

Type annotation	Accepted value types	Behavior
Image	str, numpy.ndarray	Creates a sub-Document, fills in doc.tensor by reading the image and sets .modality='image'
Text	str	Creates a sub-Document, fills in doc.text from the given value and sets .modality='text'
URI	str	Creates a sub-Document, fills in doc.uri from the given value
Audio	str, numpy.ndarray	Creates a sub-Document, fills in doc.tensor by reading the audio and sets .modality='audio'
JSON	Dict	Creates a sub-Document, fills in doc.tags from the given value and sets .modality='json'
Video	str, numpy.ndarray	Creates a sub-Document, fills in doc.tensor by reading the video and sets .modality='video'
Mesh	str, numpy.ndarray	Creates a sub-Document, fills in doc.tensor by sub-sampling the mesh as point-cloud and sets .modality='mesh'
Blob	str, bytes	Creates a sub-Document, fills in doc.blob from the given value or reading from the path
Tabular	str (file name)	Reads a CSV file, creates a sub-Document for each line and fills in doc.tags by considering the first row as column names and mapping subsequent rows into corresponding values.

• A class field labeled with List[Type] will create sub-Documents under root.chunks[0].chunks. For example,

  Field in List[Image]

  from typing import List
  
  from docarray import dataclass, Document
  from docarray.typing import Image
  
  
  @dataclass
  class MMDoc2:
      banners: List[Image]
  
  
  Document(MMDoc2(['test-1.jpeg', 'test-2.jpeg'])).summary()
  

  Document structure

  📄 Document: 52d9dcca4bc30cd0ef3b82917459cd32
  └── 💠 Chunks
      └── 📄 Document: 04edacf582c5aa7b0bcfcf3d3e0a57bf
          ╭──────────────────────┬───────────────────────────────────────────────────────╮
          │ Attribute            │ Value                                                 │
          ├──────────────────────┼───────────────────────────────────────────────────────┤
          │ parent_id            │ 52d9dcca4bc30cd0ef3b82917459cd32                      │
          │ granularity          │ 1                                                     │
          ╰──────────────────────┴───────────────────────────────────────────────────────╯
          └── 💠 Chunks
              ├── 📄 Document: f5e9f105162e26d1d42ef7e2d363095a
              │   ╭──────────────┬───────────────────────────────────────────────────────────────╮
              │   │ Attribute    │ Value                                                         │
              │   ├──────────────┼───────────────────────────────────────────────────────────────┤
              │   │ parent_id    │ 04edacf582c5aa7b0bcfcf3d3e0a57bf                              │
              │   │ granularity  │ 1                                                             │
              │   │ tensor       │ <class 'numpy.ndarray'> in shape (504, 504, 3), dtype: uint8  │
              │   │ mime_type    │ image/jpeg                                                    │
              │   │ uri          │ test-1.jpeg                                                   │
              │   │ modality     │ image                                                         │
              │   ╰──────────────┴───────────────────────────────────────────────────────────────╯
              └── 📄 Document: d7d0b506f690890113e6a601ef80f8c6
                  ╭──────────────┬───────────────────────────────────────────────────────────────╮
                  │ Attribute    │ Value                                                         │
                  ├──────────────┼───────────────────────────────────────────────────────────────┤
                  │ parent_id    │ 04edacf582c5aa7b0bcfcf3d3e0a57bf                              │
                  │ granularity  │ 1                                                             │
                  │ tensor       │ <class 'numpy.ndarray'> in shape (504, 504, 3), dtype: uint8  │
                  │ mime_type    │ image/jpeg                                                    │
                  │ uri          │ test-2.jpeg                                                   │
                  │ modality     │ image                                                         │
                  ╰──────────────┴───────────────────────────────────────────────────────────────╯
  

• A field annotated with another dataclass will create the full nested structure under the corresponding chunk:

  Field in another dataclass

  from docarray import dataclass, Document
  from docarray.typing import Image, Text
  
  
  @dataclass
  class BannerDoc:
      description: Text = 'this is a test empty image'
      banner: Image = 'test-1.jpeg'
  
  
  @dataclass
  class ColumnArticle:
      feature_image: BannerDoc
      description: Text = 'this is a column article'
      website: str = 'https://docs.docarray.org'
  
  
  Document(ColumnArticle(feature_image=BannerDoc())).summary()
  

  Document structure

  📄 Document: 75a3df4c26498d338589d2b2c20e156e
  ╭────────────────────┬─────────────────────────────────────────────────────────╮
  │ Attribute          │ Value                                                   │
  ├────────────────────┼─────────────────────────────────────────────────────────┤
  │ tags               │ {'website': 'https://docs.docarray.org'}                          │
  ╰────────────────────┴─────────────────────────────────────────────────────────╯
  └── 💠 Chunks
      ├── 📄 Document: cb1df29a384a6d39aa81e5af93316c4d
      │   ╭──────────────────────┬───────────────────────────────────────────────────────╮
      │   │ Attribute            │ Value                                                 │
      │   ├──────────────────────┼───────────────────────────────────────────────────────┤
      │   │ parent_id            │ 75a3df4c26498d338589d2b2c20e156e                      │
      │   │ granularity          │ 1                                                     │
      │   ╰──────────────────────┴───────────────────────────────────────────────────────╯
      │   └── 💠 Chunks
      │       ├── 📄 Document: 65cce8eb564f9ce136ff693b8ecb8f53
      │       │   ╭──────────────────────┬───────────────────────────────────────────────────────╮
      │       │   │ Attribute            │ Value                                                 │
      │       │   ├──────────────────────┼───────────────────────────────────────────────────────┤
      │       │   │ parent_id            │ cb1df29a384a6d39aa81e5af93316c4d                      │
      │       │   │ granularity          │ 1                                                     │
      │       │   │ text                 │ this is a test empty image                            │
      │       │   │ modality             │ text                                                  │
      │       │   ╰──────────────────────┴───────────────────────────────────────────────────────╯
      │       └── 📄 Document: 4dc4497d608b4f96094711e90cfb8078
      │           ╭──────────────┬───────────────────────────────────────────────────────────────╮
      │           │ Attribute    │ Value                                                         │
      │           ├──────────────┼───────────────────────────────────────────────────────────────┤
      │           │ parent_id    │ cb1df29a384a6d39aa81e5af93316c4d                              │
      │           │ granularity  │ 1                                                             │
      │           │ tensor       │ <class 'numpy.ndarray'> in shape (504, 504, 3), dtype: uint8  │
      │           │ mime_type    │ image/jpeg                                                    │
      │           │ uri          │ test-1.jpeg                                                   │
      │           │ modality     │ image                                                         │
      │           ╰──────────────┴───────────────────────────────────────────────────────────────╯
      └── 📄 Document: f7b3aaefeab73af18f8372a594405b46
          ╭──────────────────────┬───────────────────────────────────────────────────────╮
          │ Attribute            │ Value                                                 │
          ├──────────────────────┼───────────────────────────────────────────────────────┤
          │ parent_id            │ 75a3df4c26498d338589d2b2c20e156e                      │
          │ granularity          │ 1                                                     │
          │ text                 │ this is a column article                              │
          │ modality             │ text                                                  │
          ╰──────────────────────┴───────────────────────────────────────────────────────╯
  

• A dataclass that has only one field annotated with docarray.typing will still create a nested structure under root.chunks. In this case, len(root.chunks)=1 and your multimodal Document has basically a single modality, which may encourage you to consider if you really need to use a dataclass. After all, each Document represents a single modality, so in this case you could just use Document.

Construct from/to Document

It’s easy to convert a dataclass object from/to a Document object:

from docarray import dataclass, Document
from docarray.typing import Image


@dataclass
class MMDoc:
    banner: Image


m = MMDoc(banner='test-0.jpeg')
d = Document(m)  # to Document object
m_r = MMDoc(d)  # from Document object

assert m == m_r

Use field() for advanced configs

For common and simple use cases, no other functionality is required. There are, however, some dataclass features that require additional per-field information. For this, you can replace the default field value with a call to the provided field() function.

For example, a mutable object is not allowed as the default value of any dataclass field. You can solve it via:

from typing import List

from docarray import dataclass, field
from docarray.typing import Image


@dataclass
class MMDoc:
    banner: List[Image] = field(default_factory=lambda: ['test-1.jpeg', 'test-2.jpeg'])

Other parameters from the standard the Python field like init, compare, hash, repr are also supported. More details can be found here.

What’s next?

In this chapter, we’ve learned to use the @dataclass decorator and type annotation to build multimodal documents. The look and feel is exactly the same as Python’s builtin dataclass.

Leveraging the nested Document structure, DocArray’s dataclass offers great expressiveness for data scientists and machine learning engineers who work with multimodal data, allowing them to represent image, text, video, mesh, and tabular data in an intuitive way. Converting a multimodal dataclass object from/to a Document is straightforward.

In the next chapter, we’ll see how to select modality (aka sub-Document) via selector syntax.