Detect language and filter text records by confidence score using fastText - Platform For AI

Detect language and filter text records by confidence score using fastText. Supports 176 languages.

How it works

The component performs three operations on each input record:

Language identification: fastText analyzes text and returns an ISO 639 code (such as en, zh, or fr).
Confidence scoring: fastText assigns a score from 0 to 1 indicating prediction reliability.
Filtering: Records with languages not in the target list or confidence scores below the minimum threshold are removed.

In a typical LLM data preprocessing pipeline, place this component after converting source data to JSON Lines format and before running quality scoring or deduplication.

Text extraction  -->  Language identification & filtering  -->  Quality scoring  -->  Deduplication

Input requirements

Input data files must be stored in Object Storage Service (OSS) and meet the following requirements:

Requirement	Details
Format	JSON Lines (`.jsonl`). Each line is a standalone JSON object. The file itself is not a valid JSON object.
Encoding	UTF-8. The fastText model was trained on UTF-8 data and requires UTF-8 input.
Field	Each JSON object must contain a text field. Specify the field name in the Target Process Field parameter.

Example input

{"content": "Alibaba Cloud provides scalable cloud computing services.", "id": "doc_001"}
{"content": "Le cloud computing offre des services évolutifs.", "id": "doc_002"}

For a complete sample file, see the example data.

Example output

After processing with Language ID Name set to en and Minimum Score set to 0.65, only English records with confidence scores of 0.65 or higher are retained. In the preceding example, only the first record (doc_001) passes because doc_002 is Chinese and doc_003 is French.

Supported computing resources

This component runs on Deep Learning Containers (DLC).

Supported languages

The component recognizes 176 languages. Specify languages in the Language ID Name parameter using the ISO codes listed below. Separate multiple codes with commas (for example, en,zh,fr).

af, als, am, an, ar, arz, as, ast, av, az, azb, ba, bar, bcl, be, bg, bh, bn, bo, bpy, br, bs, bxr, ca, cbk, ce, ceb, ckb, co, cs, cv, cy, da, de, diq, dsb, dty, dv, el, eml, en, eo, es, et, eu, fa, fi, fr, frr, fy, ga, gd, gl, gn, gom, gu, gv, he, hi, hif, hr, hsb, ht, hu, hy, ia, id, ie, ilo, io, is, it, ja, jbo, jv, ka, kk, km, kn, ko, krc, ku, kv, kw, ky, la, lb, lez, li, lmo, lo, lrc, lt, lv, mai, mg, mhr, min, mk, ml, mn, mr, mrj, ms, mt, mwl, my, myv, mzn, nah, nap, nds, ne, new, nl, nn, no, oc, or, os, pa, pam, pfl, pl, pms, pnb, ps, pt, qu, rm, ro, ru, rue, sa, sah, sc, scn, sco, sd, sh, si, sk, sl, so, sq, sr, su, sv, sw, ta, te, tg, th, tk, tl, tr, tt, tyv, ug, uk, ur, uz, vec, vep, vi, vls, vo, wa, war, wuu, xal, xmf, yi, yo, yue, zh

Configure the component

In Machine Learning Designer, drag the LLM-Quality Predict and Language Recognition-FastText (DLC) component onto the canvas and configure the following parameters.

Section	Parameter		Required	Description	Default
Field settings	Target Process Field		Yes	JSON field that contains the text to process.	No default
	Language ID Name		Yes	Target language codes for filtering. Separate multiple codes with commas (`,`). Example: `en,zh`. For the list of supported codes, see the Supported languages section.	No default
	Minimum Score		Yes	Confidence score threshold. Records scoring below this value are filtered out. Higher thresholds produce stricter filtering.	No default
	OSS Directory for Saving OutputData		No	OSS path for the output data. If left blank, the default workspace path is used.	Default workspace path
Performance tuning	Number of Processes		No	Number of parallel processes. Increase for larger datasets.	8
	Select Resource Group	Public Resource Group	No	Instance type (CPU or GPU), number of instances, and VPC configuration.	No default
	Select Resource Group	Dedicated resource group	No	Number of vCPUs, memory, shared memory, GPU count, and instances.	No default
	Maximum Running Duration		No	Maximum time the component is allowed to run. The DLC job terminates if this duration is exceeded.	No default

Best practices

Set Minimum Score based on your precision-recall trade-off. A threshold of 0.65 is a common starting point. Lower values retain more data but include more misidentified records; higher values produce cleaner output but may discard valid records.
Short texts (under 20 characters) tend to have lower confidence scores. Consider removing very short records before running this component.
The fastText model assigns a single language label per record. For mixed-language texts, the model returns the dominant language.