Labeled Dataset Format Requirements for PAI TFRecord - Platform for AI - Alibaba Cloud - Platform For AI

Before converting a labeled dataset to a TFRecord file, export it from Machine Learning Platform for AI (PAI) as a CSV file. Each row in the CSV file represents one labeled entry with three fields.

Field	Data type	Description
Entry ID	INT	The unique ID of the entry.
Raw data	JSON	The URL of the source image.
Labeling result	JSON	The labeling result. Format varies by task type.

This topic describes the labeling result format for each supported task type:

Single-label image classification
Multi-label image classification
Object detection
Image segmentation
Text recognition
Text detection
End-to-end text recognition

Single-label image classification

Each entry assigns one label to the image.

CSV row format:

1,{"url":"http://a.jpg"},"{"option":"Passport"}
2,{"url":"http://b.jpg"},"{"option":"Passport"}

Labeling result schema:

{
    "option": "Passport"
}

Field	Type	Description
`option`	string	The label assigned to the image.

Multi-label image classification

Each entry assigns one or more labels to the image.

CSV row format:

1,{"url":"http://a.jpg"},{["option":"Passport", "option":"ID card"]}
2,{"url":"http://b.jpg"},{["option":"Passport", "option":"Exit-Entry Permit for Traveling to and from Hong Kong and Macau"]}

Labeling result schema:

{
    "option": ["Passport", "ID card"]
}

Field	Type	Description
`option`	array of strings	The labels assigned to the image.

Object detection

Each entry contains a list of detected objects. Each object has a category label and a bounding box.

CSV row format:

1,{"url": "http://b.jpg"},[{"text": "{\"class*\": \"Category 1\"}", "coord": ["306.73", "517.59", "324.42", "282.07", "347.69", "282.07", "333.73", "519.45"]}, {"text": "{\"class*\": \"Category 2\"}", "coord": ["342.11", "723.32", "349.56", "608.81", "366.31", "606.95", "360.73", "730.76"]}]
2,{"url": "http://a.jpg"},[{"text": "{\"class*\": \"Category 1\"}", "coord": ["338.35", "8.53", "700.16", "8.53", "700.16", "50.35", "338.35", "50.35"]}, {"text": "{\"class*\": \"Category 2\"}", "coord": ["26.88", "64.00", "218.03", "64.00", "218.03", "99.84", "26.88", "99.84"]}]

Labeling result schema:

The labeling result is an array. Each element represents one detected object.

[
    {
        "text": "{\"class*\": \"Category 1\"}",
        "coord": [
            "338.35",
            "8.53",
            "700.16",
            "8.53",
            "700.16",
            "50.35",
            "338.35",
            "50.35"
        ]
    },
    {
        "text": "{\"class*\": \"Category 2\"}",
        "coord": [
            "26.88",
            "64.00",
            "218.03",
            "64.00",
            "218.03",
            "99.84",
            "26.88",
            "99.84"
        ]
    }
]

Field	Type	Description
`text`	string	A JSON-encoded string containing the object's category. See the `text` field schema below.
`coord`	array of strings	Eight coordinate values defining the bounding box polygon: `[x1, y1, x2, y2, x3, y3, x4, y4]`.

`text` field schema:

The text field is a serialized JSON object with the following structure:

{"class*": "Category 1"}

Field	Type	Description
`class*`	string	The category of the detected object.

Image segmentation

Each entry references a mask image generated from the labeled segmentation.

Download a sample CSV file.

CSV row format:

1,{"http://a.jpg"},{"ossUrl":"http://ossgw.alicdn.com/a.png"}

Labeling result schema:

{
    "ossUrl": "http://ossgw.alicdn.com/a.png"
}

Field	Type	Description
`ossUrl`	string	The Object Storage Service (OSS) URL of the mask image generated for the entry. The mask is in PNG format.

Mask channel encoding:

The mask stores category information across its color channels. An image has red, blue, and green channels for storing information. The red channel usually stores category information. Channel IDs start at 0; valid values are 0 to 3. The value 0 indicates the background.

Text recognition

Each entry contains the text recognized from the image.

CSV row format:

1,{"url": "http://b.jpg"},{"text": "Text 1"}
2,{"url": "http://a.jpg"},{"text": "Text 2"}

Labeling result schema:

{
    "text": "Text 1"
}

Field	Type	Description
`text`	string	The recognized text content.

Text detection

Each entry contains a list of detected text lines. Each text line includes its position, orientation, and category. The entry also includes the overall image orientation.

CSV row format:

1,{"url": "http://b.jpg"},[[{"text": "{\"direction\": \"Bottom right\", \"class*\": \"Category 1\"}", "coord": ["306.73", "517.59", "324.42", "282.07", "347.69", "282.07", "333.73", "519.45"]}, {"text": "{\"direction\": \"Bottom right\", \"class*\": \"Category 2\"}", "coord": ["342.11", "723.32", "349.56", "608.81", "366.31", "606.95", "360.73", "730.76"]}], {"option": "Bottom right"}]
2,{"url": "http://a.jpg"},[[{"text": "{\"direction\": \"Bottom down\", \"class*\": \"Category 1\"}", "coord": ["338.35", "8.53", "700.16", "8.53", "700.16", "50.35", "338.35", "50.35"]}, {"text": "{\"direction\": \"Bottom down\", \"class*\": \"Category 2\"}", "coord": ["26.88", "64.00", "218.03", "64.00", "218.03", "99.84", "26.88", "99.84"]}], {"option": "Bottom down"}]

Labeling result schema:

The labeling result is a two-element array: a list of text line objects, followed by an image orientation object.

[
    [
        {
            "text": "{\"direction\": \"Bottom down\", \"class*\": \"Category 1\"}",
            "coord": [
                "338.35",
                "8.53",
                "700.16",
                "8.53",
                "700.16",
                "50.35",
                "338.35",
                "50.35"
            ]
        },
        {
            "text": "{\"direction\": \"Bottom down\", \"class*\": \"Category 2\"}",
            "coord": [
                "26.88",
                "64.00",
                "218.03",
                "64.00",
                "218.03",
                "99.84",
                "26.88",
                "99.84"
            ]
        }
    ],
    {
        "option": "Bottom down"
    }
]

Text line object fields:

Field	Type	Description
`text`	string	A JSON-encoded string containing the text line's orientation and category. See the `text` field schema below.
`coord`	array of strings	Eight coordinate values defining the bounding box polygon: `[x1, y1, x2, y2, x3, y3, x4, y4]`.

`text` field schema:

{"direction": "Bottom down", "class*": "Category 1"}

Field	Type	Description
`direction`	string	The orientation of the text line.
`class*`	string	The category of the text.

Image orientation object:

Field	Type	Description
`option`	string	The orientation of the entire image. Specifies whether the bottom of the image is downward, upward, leftward, or rightward.

End-to-end text recognition

End-to-end text recognition combines text detection with text recognition. Each entry contains detected text lines with their recognized text content, position, orientation, and category.

CSV row format:

1,{"url": "http://b.jpg"},[[{"text": "{\"text\": \"Text 1\", \"direction\": \"Bottom right\", \"class*\": \"Category 1\"}", "coord": ["306.73", "517.59", "324.42", "282.07", "347.69", "282.07", "333.73", "519.45"]}, {"text": "{\"text\": \"Text 2\", \"direction\": \"Bottom right\", \"class*\": \"Category 2\"}", "coord": ["342.11", "723.32", "349.56", "608.81", "366.31", "606.95", "360.73", "730.76"]}], {"option": "Bottom right"}]
2,{"url": "http://a.jpg"},[[{"text": "{\"text\": \"Text 3\", \"direction\": \"Bottom down\", \"class*\": \"Category 1\"}", "coord": ["338.35", "8.53", "700.16", "8.53", "700.16", "50.35", "338.35", "50.35"]}, {"text": "{\"text\": \"Text 4\", \"direction\": \"Bottom down\", \"class*\": \"Category 2\"}", "coord": ["26.88", "64.00", "218.03", "64.00", "218.03", "99.84", "26.88", "99.84"]}], {"option": "Bottom down"}]

Labeling result schema:

The structure is identical to text detection, with the addition of a text field inside each text line's embedded JSON string.

[
    [
        {
            "text": "{\"text\": \"Text 3\", \"direction\": \"Bottom down\", \"class*\": \"Category 1\"}",
            "coord": [
                "338.35",
                "8.53",
                "700.16",
                "8.53",
                "700.16",
                "50.35",
                "338.35",
                "50.35"
            ]
        },
        {
            "text": "{\"text\": \"Text 4\", \"direction\": \"Bottom down\", \"class*\": \"Category 2\"}",
            "coord": [
                "26.88",
                "64.00",
                "218.03",
                "64.00",
                "218.03",
                "99.84",
                "26.88",
                "99.84"
            ]
        }
    ],
    {
        "option": "Bottom down"
    }
]

Text line object fields:

Field	Type	Description
`text`	string	A JSON-encoded string containing the recognized text, line orientation, and category. See the `text` field schema below.
`coord`	array of strings	Eight coordinate values defining the bounding box polygon: `[x1, y1, x2, y2, x3, y3, x4, y4]`.

`text` field schema:

{"text": "Text 3", "direction": "Bottom down", "class*": "Category 1"}

Field	Type	Description
`text`	string	The recognized text content of the line.
`direction`	string	The orientation of the text line.
`class*`	string	The category of the text.

Image orientation object:

Field	Type	Description
`option`	string	The orientation of the entire image. Specifies whether the bottom of the image is downward, upward, leftward, or rightward.