Training datasets often contain images that are too small (low resolution or corrupt) or too large (inflated file size that slows training). The LVM-Image-Size Filter (DLC) component removes these outliers by filtering image data outside a size range you define, helping you maintain training data quality for image generation models.
How it works
The component calculates the file size of each image and compares it against the minimum and maximum thresholds you set. Images outside the specified range are excluded from the output, which is written to an Object Storage Service (OSS) directory you specify.
On the first run, the component generates a metadata file (meta.jsonl) in the parent directory of your image data path. On subsequent runs, point to this file directly instead of re-scanning the full directory.
Supported computing resources
Configure the component
Add the LVM-Image-Size Filter (DLC) component to your pipeline in Machine Learning Designer, connect an input source, and set the parameters below.
Connect an input source
Connect one of the following to the component's input port:
-
A Read File Data component that reads the OSS path where your training data is stored
-
An Image Data OSS Path pointing to the OSS directory or the
meta.jsonlmetadata file from a previous run -
Any upstream image preprocessing component
Set parameters
| Tab | Parameter | Required | Description | Default | |
|---|---|---|---|---|---|
| Field Settings | Image Data OSS Path | No | The OSS directory containing your image data, or the meta.jsonl metadata file generated by a previous run. If no upstream component is connected on the first run, specify this path manually. The component generates meta.jsonl in the parent directory of this path. |
— | |
| Output File OSS Path | Yes | The OSS directory where filtering results are stored. | — | ||
| Output Filename | Yes | The base name for the output file. | result.jsonl |
||
| Parameter Settings | Minimum Size | Yes | Images smaller than this value are filtered out. Accepted units: KB, MB, GB, TB, PB, EB, ZB, YB. Example: 30 KB. |
0 |
|
| Maximum Size | Yes | Images larger than this value are filtered out. Accepted units: KB, MB, GB, TB, PB, EB, ZB, YB. Example: 10 MB. |
1GB |
||
| Execution Tuning | Number of processes | Yes | The number of parallel processes for filtering. | 4 |
|
| Select Resource Group | Public resource group | No | The CPU or GPU instance type and VPC to use. We recommend that you use the CPU instance type to save costs. | — | |
| Dedicated resource group | No | The number of vCPUs, memory, shared memory, and GPUs to allocate. | — | ||
| Maximum running duration (seconds) | No | The time limit for the component run. The job stops if this duration is exceeded. | — | ||
Review the output
The component writes the following files to the Output File OSS Path:
| File | Description | When to use |
|---|---|---|
{name}.jsonl |
The filtered image list. The file name is set by Output Filename. | Use this as input for downstream training or preprocessing steps. |
{name}_stats.jsonl |
The state file. | Review this file to understand how many images were filtered and why. |
dj_run_yaml.yaml |
The parameter configuration used during the run. | Use this to reproduce or audit the filter settings. |