DataWorks:Snowflake

Configure an offline sync task for a single table

• For the step-by-step procedure, see Configure a sync task in the codeless UI and Configure a sync task in the code editor.

• For script parameters and sample scripts, see the Appendix: Script demos and parameter descriptions section.

Out-of-memory (OOM) errors

Symptom: The sync task fails with an out-of-memory (OOM) error.

Causes and fixes:

• fetchSize exceeds 2048. Reduce fetchSize to 1024 (the default) or lower.

• batchSize is set too high for the available memory. Reduce batchSize to 1024 (the default) or lower.

• splitFactor is set too high. Keep splitFactor in the range 1–100. A high value creates too many data chunks and increases memory pressure.

splitPk has no effect

Symptom: Data is not split across concurrent threads despite setting splitPk.

Cause: splitPk supports only integer columns. If you specify a string, floating-point, or date column, Snowflake Reader reports an error.

Fix: Set splitPk to an integer primary key column. If no integer column is available, increase concurrent instead to improve throughput.

Configure an offline task script

Use the code editor to write the sync task script in the required format. For the general format requirements, see Configure a sync task in the code editor. The following sections provide script samples and parameter descriptions for Snowflake Reader and Snowflake Writer.

Reader script demo

{
    "transform": false,
    "type": "job",
    "version": "2.0",
    "steps": [
        {
            "stepType": "snowflake",
            "parameter": {
                "schema": "PUBLIC",
                "envType": 0,
                "datasource": "snowflake_datasource",
                "column": [
                    "ID",
                    "NAME"
                ],
                "where": "",
                "table": "table"
            },
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "stream",
            "parameter": {
                "print": true
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "throttle": false,
            "concurrent": 2
        }
    },
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    }
}

Reader script parameters

Writer script demo

{
    "transform": false,
    "type": "job",
    "version": "2.0",
    "steps": [
        {
            "stepType": "stream",
            "parameter": {},
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "snowflake",
            "parameter": {
                "schema": "PUBLIC",
                "envType": 0,
                "datasource": "snowflake_datasource",
                "column": [
                    "ID",
                    "NAME"
                ],
                "table": "TABLE"
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "executeMode": null,
        "errorLimit": {
            "record": "0"
        },
        "speed": {
            "concurrent": 2,
            "throttle": false
        }
    },
    "order": {
        "hops": []
    }
}

Writer script parameters