The Multiclass Classification Evaluation algorithm is used to evaluate the classification efficiency of a model that is used to process classification problems involving more than two classes. The algorithm provides multiple metrics such as accuracy, recall rate, F1 score, and confusion matrix to help quantify the classification accuracy for each class. The confusion matrix displays the relationship between the classes predicted by the model and the actual classes. Other metrics provide information to help you determine whether the classification result for each class is correct. These metrics help you understand how the model performs in each class and can be used for subsequent model optimization.
Configure the component
Method 1: Configure the component on the pipeline page
Configure the parameters of the Multiclass Classification Evaluation component on the pipeline page of Machine Learning Designer in the Platform for AI (PAI) console. The following table describes the parameters.
Tab | Parameter | Description |
Fields Setting | Original Classification Result Column | The original label column can be selected for this parameter. A maximum of 1,000 classes are supported. |
Predicted Classification Result Column | The column of the predicted classification result. In most cases, this parameter is set to prediction_result. | |
Advanced Options | If you select Advanced Options, the Predicted Classification Result Column parameter is valid. | |
Prediction Result Probability Column | The column that is used to calculate the log loss of the model. In most cases, this parameter is set to prediction_detail. This parameter is valid only for the random forest model. If you configure this parameter for other models, an error may be reported. | |
Tuning | Cores | The number of cores. By default, the system determines the value. This parameter must be used with the Memory Size per Core parameter. |
Memory Size per Core | The memory size of each core. Unit: MB. By default, the system determines the value. |
Method 2: Use PAI commands
Configure the parameters of the Multiclass Classification Evaluation component by using PAI commands. You can use the SQL Script component to call PAI commands. For more information, see Scenario 4: Execute PAI commands within the SQL script component.
PAI -name MultiClassEvaluation -project algo_public
-DinputTableName="test_input"
-DoutputTableName="test_output"
-DlabelColName="label"
-DpredictionColName="prediction_result"
-Dlifecycle=30;Parameter | Required | Default value | Description |
inputTableName | Yes | No default value | The name of the input table. |
inputTablePartitions | No | Full table | The partitions that are selected from the input table for training. |
outputTableName | Yes | No default value | The name of the output table. |
labelColName | Yes | No default value | The name of the original label column in the input table. |
predictionColName | Yes | No default value | The name of the label column of the prediction result. |
predictionDetailColName | No | No default value | The name of the probability column of the prediction result. Example: |
lifecycle | No | No default value | The lifecycle of the output table. |
coreNum | No | Determined by the system | The number of cores. |
memSizePerCore | No | Determined by the system | The memory size of each core. |
Example
Add the SQL Script component as a node to the canvas and execute the following SQL statements to generate training data.
drop table if exists multi_esti_test; create table multi_esti_test as select * from ( select '0' as id,'A' as label,'A' as prediction,'{"A": 0.6, "B": 0.4}' as detail union all select '1' as id,'A' as label,'B' as prediction,'{"A": 0.45, "B": 0.55}' as detail union all select '2' as id,'A' as label,'A' as prediction,'{"A": 0.7, "B": 0.3}' as detail union all select '3' as id,'A' as label,'A' as prediction,'{"A": 0.9, "B": 0.1}' as detail union all select '4' as id,'B' as label,'B' as prediction,'{"A": 0.2, "B": 0.8}' as detail union all select '5' as id,'B' as label,'B' as prediction,'{"A": 0.1, "B": 0.9}' as detail union all select '6' as id,'B' as label,'A' as prediction,'{"A": 0.52, "B": 0.48}' as detail union all select '7' as id,'B' as label,'B' as prediction,'{"A": 0.4, "B": 0.6}' as detail union all select '8' as id,'B' as label,'A' as prediction,'{"A": 0.6, "B": 0.4}' as detail union all select '9' as id,'A' as label,'A' as prediction,'{"A": 0.75, "B": 0.25}' as detail )tmp;Add the SQL Script component as a node to the canvas and run the following PAI commands to train the model.
drop table if exists ${o1}; PAI -name MultiClassEvaluation -project algo_public -DinputTableName="multi_esti_test" -DoutputTableName=${o1} -DlabelColName="label" -DpredictionColName="prediction" -Dlifecycle=30;Right-click the SQL Script component and choose View Data > SQL Script Output to view the training results.
| result | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | { "ActualLabelFrequencyList": [5, 5], "ActualLabelProportionList": [0.5, 0.5], "ConfusionMatrix": [[4, 1], [2, 3]], "LabelList": ["A", "B"], "LabelMeasureList": [{ "Accuracy": 0.7, "F1": 0.7272727272727273, "FalseDiscoveryRate": 0.3333333333333333, "FalseNegative": 1, "FalseNegativeRate": 0.2, "FalsePositive": 2, "FalsePositiveRate": 0.4, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.75, "Precision": 0.6666666666666666, "Sensitivity": 0.8, "Specificity": 0.6, "TrueNegative": 3, "TruePositive": 4}, { "Accuracy": 0.7, "F1": 0.6666666666666666, "FalseDiscoveryRate": 0.25, "FalseNegative": 2, "FalseNegativeRate": 0.4, "FalsePositive": 1, "FalsePositiveRate": 0.2, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.6666666666666666, "Precision": 0.75, "Sensitivity": 0.6, "Specificity": 0.8, "TrueNegative": 4, "TruePositive": 3}], "LabelNumber": 2, "OverallMeasures": { "Accuracy": 0.7, "Kappa": 0.3999999999999999, "LabelFrequencyBasedMicro": { "Accuracy": 0.7, "F1": 0.696969696969697, "FalseDiscoveryRate": 0.2916666666666666, "FalseNegative": 1.5, "FalseNegativeRate": 0.3, "FalsePositive": 1.5, "FalsePositiveRate": 0.3, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.7083333333333333, "Precision": 0.7083333333333333, "Sensitivity": 0.7, "Specificity": 0.7, "TrueNegative": 3.5, "TruePositive": 3.5}, "MacroAveraged": { "Accuracy": 0.7, "F1": 0.696969696969697, "FalseDiscoveryRate": 0.2916666666666666, "FalseNegative": 1.5, "FalseNegativeRate": 0.3, "FalsePositive": 1.5, "FalsePositiveRate": 0.3, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.7083333333333333, "Precision": 0.7083333333333333, "Sensitivity": 0.7, "Specificity": 0.7, "TrueNegative": 3.5, "TruePositive": 3.5}, "MicroAveraged": { "Accuracy": 0.7, "F1": 0.7, "FalseDiscoveryRate": 0.3, "FalseNegative": 3, "FalseNegativeRate": 0.3, "FalsePositive": 3, "FalsePositiveRate": 0.3, "Kappa": 0.3999999999999999, "NegativePredictiveValue": 0.7, "Precision": 0.7, "Sensitivity": 0.7, "Specificity": 0.7, "TrueNegative": 7, "TruePositive": 7}}, "PredictedLabelFrequencyList": [6, 4], "PredictedLabelProportionList": [0.6, 0.4], "ProportionMatrix": [[0.8, 0.2], [0.4, 0.6]]} |
Appendix
If you run the Multiclass Classification Evaluation component on the pipeline page, you can right-click the Multiclass Classification Evaluation component and select Visual Analysis to view the results.
