Document management - Tablestore - Alibaba Cloud Documentation Center

Prerequisites

An AgentStorage instance has been created with its HTTPS endpoint and instance name obtained, and a knowledge base has been created. See Knowledge base management.
To upload local documents, your account must have an OSS bucket in the same region as the instance and OSS authorization must be completed. When initializing the SDK, you must provide additional OSS configuration (AccessKey and OSS endpoint/bucket).
Document metadata must match the metadata schema defined for the knowledge base. Otherwise, the document will be marked as failed.
If the knowledge base has subspace enabled, all document operations and retrieval must specify subspace.

Supported document formats

Type	Extension
PDF	`.pdf`
Word	`.doc`, `.docx`
Excel	`.xls`, `.xlsx`
PowerPoint	`.ppt`, `.pptx`
Plain text	`.txt`
Markdown	`.md`
Structured text	`.csv`, `.xml`, `.html`, `.json`
Image	`.jpeg`, `.png`, `.bmp`, `.webp`, `.gif`, `.tiff`

Importing images and multimodal retrieval require enabling multimodal when creating the knowledge base. Once enabled, this setting cannot be changed. Knowledge bases without multimodal enabled cannot perform image vectorization or retrieval on uploaded images.

A single request accepts up to 10 documents, with a maximum file size of 50 MB. To request higher limits, submit a ticket or join the Tablestore technical support DingTalk group 36165029092 and contact technical support.

Document status lifecycle

Documents go through the following statuses from upload to searchable:

Status	Description	Available operations
`Pending`	Upload task received, waiting for processing	Query status, delete
`Indexing`	Parsing, chunking, and vectorizing	Query status, delete
`Completed`	Indexing complete, searchable	Retrieve, update metadata, delete, view chunks
`Failed`	Indexing failed	View failure reason, delete, re-upload
`Deleting`	Document and its chunks are being deleted	Wait for deletion to complete

Documents in Pending or Indexing status cannot be retrieved. You must wait until the status changes to Completed before retrieval is possible.

Import documents

Import documents into a knowledge base. The system automatically performs parsing, chunking, Embedding vectorization, and index construction. Repeatedly uploading with the same ossKey overwrites the original document.

The SDK provides three import methods:

Method	SDK method	Description
Upload local files	`upload_documents()` / `uploadDocuments()`	Specify local file paths. The SDK automatically uploads them to OSS and then adds them to the knowledge base.
Add OSS files	`add_documents()` / `addDocuments()`	Specify existing OSS file paths.
Batch import from OSS directory	`add_documents()` / `addDocuments()`	Specify an OSS directory path. The system recursively scans all files in the directory.

For complete request fields and response details, see Import documents API.

Console

Open the target knowledge base card, go to Document management, and click Upload document.
Select an import method:
- Upload file: Select the upload location (OSS bucket), then select local files to upload.
- Upload folder: Batch upload by directory. You can configure whether to recurse and set included or excluded file extensions.
- Import from OSS: Select the data source bucket, then import existing files or directories from OSS.
Click OK to start the import.

If the knowledge base has a metadata schema defined, the upload panel displays Step 3 "Metadata (optional)" where you can fill in the corresponding fields for the document. Knowledge bases without a metadata schema do not have this step.

File type, single file size, and batch quantity limits are subject to the upload panel. After upload, documents must go through parsing and chunking. They can only be retrieved after the status changes to Completed. If an OSS authorization prompt appears, complete the authorization first before continuing.

SDK

Python

When using upload_documents, initializing AgentStorageClient requires both oss_endpoint and oss_bucket_name to be provided. Otherwise, a ValueError is raised.

from tablestore_agent_storage import AgentStorageClient

# To upload local files, configure OSS (the SDK first uploads files to OSS, then triggers chunking and vectorization)
client = AgentStorageClient(
    access_key_id="<AccessKey ID>",
    access_key_secret="<AccessKey Secret>",
    ots_endpoint="https://<instance>.<region>.ots.aliyuncs.com",
    ots_instance_name="<instance-name>",
    oss_endpoint="https://oss-<region>.aliyuncs.com",
    oss_bucket_name="<bucket-name>",
)

resp = client.upload_documents({
    "knowledgeBaseName": "product_docs_kb",
    "documents": [
        {
            "filePath": "/home/user/docs/product_manual.pdf",
            "metadata": {"author": "John", "category": "Product documentation"}
        },
        {
            "filePath": "/home/user/docs/faq.docx",
            "metadata": {"author": "Jane", "category": "FAQ"}
        }
    ]
})

for detail in resp["data"]["documentDetails"]:
    print(detail["ossKey"], detail["status"])

Add OSS files

When files are already on OSS, specify the ossKey directly. API key authentication is sufficient.

from tablestore_agent_storage import AgentStorageClient

client = AgentStorageClient(
    api_key="<your-api-key>",
    ots_endpoint="https://<instance>.<region>.ots.aliyuncs.com",
    ots_instance_name="<instance-name>",
)

resp = client.add_documents({
    "knowledgeBaseName": "product_docs_kb",
    "documents": [
        {
            "ossKey": "oss://example-bucket/docs/product_manual.pdf",
            "metadata": {"author": "John"}
        }
    ]
})

Batch import from OSS directory

Specify an OSS directory path. The system recursively scans all files in the directory. Use inclusionFilters and exclusionFilters to filter by file name patterns.

resp = client.add_documents({
    "knowledgeBaseName": "product_docs_kb",
    "documents": [
        {
            "ossKey": "oss://example-bucket/docs/",
            "inclusionFilters": ["*.pdf", "*.docx"],
            "exclusionFilters": ["*draft*"]
        }
    ]
})

TypeScript

To upload local files in TypeScript, use NodeAgentStorageClient and provide OSS configuration.

import { NodeAgentStorageClient } from '@tablestore/agent-storage/node';

const client = new NodeAgentStorageClient({
  accessKeyId: '<AccessKey ID>',
  accessKeySecret: '<AccessKey Secret>',
  endpoint: 'https://<instance>.<region>.ots.aliyuncs.com',
  instanceName: '<instance-name>',
  ossEndpoint: 'https://oss-<region>.aliyuncs.com',
  ossBucketName: '<bucket-name>',
  ossAccessKeyId: '<AccessKey ID>',
  ossAccessKeySecret: '<AccessKey Secret>',
});

const resp = await client.uploadDocuments({
  knowledgeBaseName: 'product_docs_kb',
  documents: [
    {
      filePath: '/home/user/docs/product_manual.pdf',
      metadata: { author: 'John', category: 'Product documentation' },
    },
  ],
});

for (const detail of resp.data.documentDetails) {
  console.log(detail.ossKey, detail.status);
}

Add OSS files

import { AgentStorageClient } from '@tablestore/agent-storage';

const client = new AgentStorageClient({
  apiKey: '<your-api-key>',
  endpoint: 'https://<instance>.<region>.ots.aliyuncs.com',
  instanceName: '<instance-name>',
});

const resp = await client.addDocuments({
  knowledgeBaseName: 'product_docs_kb',
  documents: [
    {
      ossKey: 'oss://example-bucket/docs/product_manual.pdf',
      metadata: { author: 'John' },
    },
  ],
});

Batch import from OSS directory

const resp = await client.addDocuments({
  knowledgeBaseName: 'product_docs_kb',
  documents: [
    {
      ossKey: 'oss://example-bucket/docs/',
      inclusionFilters: ['*.pdf', '*.docx'],
      exclusionFilters: ['*draft*'],
    },
  ],
});

API

Call AddDocuments to import documents. For complete request fields and response details, see Import documents API.

HTTP 200 with code: SUCCESS does not mean all documents were processed successfully. You must check the status field of each document in documentDetails individually.
status: "succeed" indicates the upload task was received, not that indexing is complete. You must wait until the document status changes to Completed before it can be retrieved.
If the knowledge base has subspace enabled, you must pass the subspace parameter. Otherwise, INVALID_PARAMETER is returned.
Metadata date fields must use a supported format (for example, yyyy-MM-dd HH:mm:ss). Unsupported formats will cause the document to be marked as failed.

Query indexing status

Documents must go through asynchronous processing after upload before they can be retrieved. To confirm whether indexing is complete, exponential backoff polling is recommended.

Console

In the Document management list, view the current status of each document in the Status column. You can also use the status filter above the list (All / Pending / Indexing / Completed / Failed / Deleting) to quickly locate documents. Click Details in the Actions column to view complete information including document ID, OSS path, status, chunk count, and more. A status of Completed means the document is searchable.

SDK

Python

import time

from tablestore_agent_storage import AgentStorageClient

client = AgentStorageClient(
    api_key="<your-api-key>",
    ots_endpoint="https://<instance>.<region>.ots.aliyuncs.com",
    ots_instance_name="<instance-name>",
)

def wait_for_document(client, kb_name, doc_id, max_interval=30):
    """Poll document status with exponential backoff, waiting for indexing to complete."""
    interval = 3
    while True:
        resp = client.get_document({
            "knowledgeBaseName": kb_name,
            "docId": doc_id
        })
        status = resp["data"][0]["status"]
        if status == "Completed":
            print(f"Indexing complete, chunk count: {resp['data'][0].get('chunkNum', 'N/A')}")
            return resp
        elif status == "Failed":
            raise Exception(f"Indexing failed: {resp['data'][0].get('failedDetails')}")
        print(f"Current status: {status}, retrying in {interval}s...")
        time.sleep(interval)
        interval = min(interval * 2, max_interval)

wait_for_document(client, "product_docs_kb", "fc6ed97f-...")

Document processing time depends on file size, type, and quantity. Small files typically complete in a few seconds, while large files or batch imports may take several minutes.

TypeScript

import { AgentStorageClient } from '@tablestore/agent-storage';

const client = new AgentStorageClient({
  apiKey: '<your-api-key>',
  endpoint: 'https://<instance>.<region>.ots.aliyuncs.com',
  instanceName: '<instance-name>',
});

async function waitForDocument(kbName: string, docId: string, maxInterval = 30) {
  let interval = 3;
  while (true) {
    const resp = await client.getDocument({
      knowledgeBaseName: kbName,
      docId,
    });
    const doc = resp.data[0];
    if (doc.status === 'Completed') {
      console.log(`Indexing complete, chunk count: ${doc.chunkNum ?? 'N/A'}`);
      return resp;
    } else if (doc.status === 'Failed') {
      throw new Error(`Indexing failed: ${doc.failedDetails}`);
    }
    console.log(`Current status: ${doc.status}, retrying in ${interval}s...`);
    await new Promise((r) => setTimeout(r, interval * 1000));
    interval = Math.min(interval * 2, maxInterval);
  }
}

await waitForDocument('product_docs_kb', 'fc6ed97f-...');

API

Call GetDocument to query document status. For complete field details, see Get document API.

Query document

Query detailed information for a specified document, including processing status, chunk count, and metadata. Provide either docId or ossKey.

Console

In the Document management list, click Details in the Actions column of the target document to view the document ID, name, OSS path, status, chunk count, subspace, ETag, creation time, and update time.

SDK

Python

from tablestore_agent_storage import AgentStorageClient

client = AgentStorageClient(
    api_key="<your-api-key>",
    ots_endpoint="https://<instance>.<region>.ots.aliyuncs.com",
    ots_instance_name="<instance-name>",
)

resp = client.get_document({
    "knowledgeBaseName": "product_docs_kb",
    "docId": "fc6ed97f-..."
})

doc = resp["data"][0]
print(f"Status: {doc['status']}, chunk count: {doc.get('chunkNum', 'N/A')}")

TypeScript

import { AgentStorageClient } from '@tablestore/agent-storage';

const client = new AgentStorageClient({
  apiKey: '<your-api-key>',
  endpoint: 'https://<instance>.<region>.ots.aliyuncs.com',
  instanceName: '<instance-name>',
});

const resp = await client.getDocument({
  knowledgeBaseName: 'product_docs_kb',
  docId: 'fc6ed97f-...',
});

const doc = resp.data[0];
console.log(`Status: ${doc.status}, chunk count: ${doc.chunkNum ?? 'N/A'}`);

API

Call GetDocument to query document details. For complete field details, see Get document API.

When the same ossKey goes through create, delete, then create again, get_document may return multiple records (including historical ones). Use the status field to identify the valid document and select the record with Completed status.

List documents

Paginate through the document list in a knowledge base.

Console

Go to the Document management page of the knowledge base. The list displays document name, creation time, update time, chunk count, and status. You can use the status filter above the list (All / Pending / Indexing / Completed / Failed / Deleting) to quickly locate documents.

SDK

Python

from tablestore_agent_storage import AgentStorageClient

client = AgentStorageClient(
    api_key="<your-api-key>",
    ots_endpoint="https://<instance>.<region>.ots.aliyuncs.com",
    ots_instance_name="<instance-name>",
)

# Get the first page
resp = client.list_documents({
    "knowledgeBaseName": "product_docs_kb",
    "maxResults": 20
})
for doc in resp["data"]["documentDetails"]:
    print(f"[{doc['status']}] {doc['ossKey']} (chunks: {doc.get('chunkNum', '-')})")

# Paginate
next_token = resp["data"].get("nextToken")
if next_token:
    resp = client.list_documents({
        "knowledgeBaseName": "product_docs_kb",
        "maxResults": 20,
        "nextToken": next_token
    })

TypeScript

import { AgentStorageClient } from '@tablestore/agent-storage';

const client = new AgentStorageClient({
  apiKey: '<your-api-key>',
  endpoint: 'https://<instance>.<region>.ots.aliyuncs.com',
  instanceName: '<instance-name>',
});

const resp = await client.listDocuments({
  knowledgeBaseName: 'product_docs_kb',
  maxResults: 20,
});
for (const doc of resp.data.documentDetails) {
  console.log(`[${doc.status}] ${doc.ossKey} (chunks: ${doc.chunkNum ?? '-'})`);
}

API

Call ListDocuments to list documents. For complete field details, see List documents API.

maxResults defaults to 10, with a maximum of 1000.
The subspace parameter accepts a multi-value list of up to 10 entries. Exceeding this limit returns an error. It is required when subspace is enabled.

Update document metadata

Update the metadata of a specified document. Only documents in Completed status can have their metadata updated. Calling this on documents in other statuses returns an error.

Console

The console does not support editing document metadata. Use the SDK or API to perform the update.

SDK

Python

from tablestore_agent_storage import AgentStorageClient

client = AgentStorageClient(
    api_key="<your-api-key>",
    ots_endpoint="https://<instance>.<region>.ots.aliyuncs.com",
    ots_instance_name="<instance-name>",
)

resp = client.update_document({
    "knowledgeBaseName": "product_docs_kb",
    "docId": "fc6ed97f-...",
    "metadata": {"author": "John", "category": "Technical documentation", "version": 2}
})

print(f"Update status: {resp['data']['updateStatus']}")  # UPDATED or NO_OP

TypeScript

import { AgentStorageClient } from '@tablestore/agent-storage';

const client = new AgentStorageClient({
  apiKey: '<your-api-key>',
  endpoint: 'https://<instance>.<region>.ots.aliyuncs.com',
  instanceName: '<instance-name>',
});

const resp = await client.updateDocument({
  knowledgeBaseName: 'product_docs_kb',
  docId: 'fc6ed97f-...',
  metadata: { author: 'John', category: 'Technical documentation', version: 2 },
});

console.log(`Update status: ${resp.data.updateStatus}`); // UPDATED or NO_OP

API

Call UpdateDocument to update document metadata. For complete field details, see Update document API.

Metadata uses overwrite-style updates. The values you pass will fully replace the existing metadata. If you want to update only one field, you must include all fields.
Passing "metadata": null clears all metadata. Omitting the metadata field preserves the original values.
The total size of metadata (key + value) is limited to 4 KB, with a maximum of 200 fields.

Delete documents

Important: Deleting a document also deletes all of its chunk data. This operation is irreversible.

Console

In the Document management list, select one or more documents and click Batch delete, or click Delete in the Actions column of a single row. A confirmation dialog appears: "Are you sure you want to delete the document? This action cannot be undone." Click OK to confirm the deletion.

SDK

Python

from tablestore_agent_storage import AgentStorageClient

client = AgentStorageClient(
    api_key="<your-api-key>",
    ots_endpoint="https://<instance>.<region>.ots.aliyuncs.com",
    ots_instance_name="<instance-name>",
)

resp = client.delete_documents({
    "knowledgeBaseName": "product_docs_kb",
    "documents": [
        {"docId": "fc6ed97f-..."},
        {"ossKey": "oss://example-bucket/docs/faq.docx"}
    ]
})

# Check each deletion result
for detail in resp["data"]["documentDetails"]:
    print(f"{detail['ossKey']}: {detail['status']}")

TypeScript

import { AgentStorageClient } from '@tablestore/agent-storage';

const client = new AgentStorageClient({
  apiKey: '<your-api-key>',
  endpoint: 'https://<instance>.<region>.ots.aliyuncs.com',
  instanceName: '<instance-name>',
});

const resp = await client.deleteDocuments({
  knowledgeBaseName: 'product_docs_kb',
  documents: [
    { docId: 'fc6ed97f-...' },
    { ossKey: 'oss://example-bucket/docs/faq.docx' },
  ],
});

// Check each deletion result
for (const detail of resp.data.documentDetails) {
  console.log(`${detail.ossKey}: ${detail.status}`);
}

API

Call DeleteDocuments to delete documents. For complete field details, see Delete documents API.

As with AddDocuments, deletion results must also be checked individually by examining the status of each document in documentDetails.

Prerequisites

Supported document formats

Document status lifecycle

Import documents

Console

SDK

Python

Add OSS files

Batch import from OSS directory

TypeScript

Add OSS files

Batch import from OSS directory

API

Query indexing status

Console

SDK

Python

TypeScript

API

Query document

Console

SDK

Python

TypeScript

API

List documents

Console

SDK

Python

TypeScript

API

Update document metadata

Console

SDK

Python

TypeScript

API

Delete documents

Console

SDK

Python

TypeScript

API

Related documents