Elasticsearch:Use Elasticsearch Machine Learning to perform intelligent detection and prediction on business data

Train a flight delay prediction task

In this example, a supervised machine learning task for flight delay prediction is trained by using regression algorithms based on specific historical data in a sample dataset that is provided by Kibana. The sample dataset is named Sample flight data and provides simulated flight data. The task can provide important reference information for airways and passengers and help passengers better schedule itineraries and flights.

The following code shows a data record in the Sample flight data dataset:

{
  "_index": "kibana_sample_data_flights",
  "_type": "_doc",
  "_id": "7b0aeogBmNQSVxOwslB_",
  "_version": 1,
  "_score": null,
  "_source": {
    "FlightNum": "QYX9S3I",
    "DestCountry": "CH",
    "OriginWeather": "Cloudy",
    "OriginCityName": "Chicago",
    "AvgTicketPrice": 824.8516378170061,
    "DistanceMiles": 4442.909325899777,
    "FlightDelay": false,
    "DestWeather": "Thunder & Lightning",
    "Dest": "Zurich Airport",
    "FlightDelayType": "No Delay",
    "OriginCountry": "US",
    "dayOfWeek": 4,
    "DistanceKilometers": 7150.1694661808515,
    "timestamp": "2023-06-02T07:28:15",
    "DestLocation": {
      "lat": "47.464699",
      "lon": "8.54917"
    },
    "DestAirportID": "ZRH",
    "Carrier": "Logstash Airways",
    "Cancelled": false,
    "FlightTimeMin": 420.59820389299125,
    "Origin": "Chicago O'Hare International Airport",
    "OriginLocation": {
      "lat": "41.97859955",
      "lon": "-87.90480042"
    },
    "DestRegion": "CH-ZH",
    "OriginAirportID": "ORD",
    "OriginRegion": "US-IL",
    "DestCityName": "Zurich",
    "FlightTimeHour": 7.009970064883188,
    "FlightDelayMin": 0
  }
}

1. Click the icon in the upper-left corner of the Kibana console. In the left-side navigation pane, choose Analytics > Machine Learning.

2. In the left-side navigation pane of the page that appears, choose Data Frame Analytics > Jobs.

3. On the Data Frame Analytics Jobs page, click Create job.

4. On the New analytics job / Choose a source data view page, select the kibana_sample_data_logs index.

5. In the Configuration step of the Create job page, configure basic information for the job.

   1. Select Regression.

      

   2. Select FlightDelayMin from the Dependent variable drop-down list. The dependent variable is the variable on which a prediction needs to be performed.

   3. In the included fields section, clear Cancelled, FlightDelay, and FlightDelayType in the Field name column.

      The Cancelled, FlightDelay, and FlightDelayType fields are useless for the prediction of flight delays. To ensure that the task is not affected by unnecessary factors and improve prediction accuracy, we recommend that you exclude these fields.

   4. In the Training percent section, slide the knob to change the percentage of data that is used for training.

      You can change the percentage based on your business requirements. In this example, the percentage is set to 90%.

      Note

      If a large amount of data is used for training, you need to take note of the time that is required to train the task. A larger amount of data requires a longer training period. If you have a large amount of data, we recommend that you specify a small percentage, such as 50% or less. Then, you can change the percentage based on your business requirements to obtain an appropriate accuracy.

6. Click Continue. Configure settings in the Additional options step.

   1. Set Feature importance values to 5. This way, the top five features that most significantly affect the prediction result are determined. This helps you identify which features have the most impact on the prediction result, and help you select the most appropriate features and optimize the task.

   2. Set Prediction field name to FlightDelayMin_prediction. This parameter specifies the name of the variable on which you want to perform a prediction.

   3. Disable Use estimated model memory limit and set Model memory limit to 500 MB. The Model memory limit parameter specifies the upper limit for the size of the memory that can be used by the machine learning task. If the data size of your dataset is large and your machine learning task is complex, a large amount of memory may be consumed. If the size of the memory that is used exceeds the upper limit, the training of the machine learning task may fail, or the performance of the task degrades. You must specify an appropriate upper limit based on the data size of your dataset and the complexity of your machine learning task.

   4. Set Maximum number of threads to 1. This parameter specifies the maximum number of threads that can be used to train the machine learning task. If you specify a large number of threads, the memory may be insufficient, or the system may stop responding.

   5. Click Continue, and configure parameters in the Job details step. Set Job ID to flightdelaymin_job.

   6. Click Continue. The system validates the task.

   7. If the validation is passed, click Continue.

   8. In the Create step, click Create.

      The system requires a specific amount of time to generate the job. The actual time that is consumed is determined by the amount of data used for the training.

7. After the job is created, click View Results to view the running result of the job.

   In the Model evaluation section of the page that appears, view the metrics that indicate the reliability of the task.

   • Generalization error: measures the performance of the task on new data, which reflects the generalization capability of the task. A smaller generalization error indicates a better generalization capability. A task with a better generalization capability can perform more accurate prediction on new data.

   • Training error: reflects the performance of the task on dataset-based training and the error generated during the learning process of the task. A smaller value indicates better performance of the task on dataset-based training.

   • Description of evaluation metrics:

     • Mean squared error: an important metric that measures the performance of a regression task. A smaller value indicates a more accurate prediction result. The value is obtained by calculating the mean square of the difference between an actual value and a prediction value that is provided by a regression task.

     • R squared: an important metric that measures the performance of a regression task. A value that is closer to 1 indicates that the task better fits actual values. A value greater than 0.8 indicates that the task has a high fitting degree.

     • Mean squared logarithmic error: A smaller value indicates a better prediction effect. The value is obtained by calculating the average value of the sum of squared errors after the logarithms of prediction values and actual values are returned.

     Note

     When you compare multiple regression tasks, you must take note of the value of the Mean squared error metric and the value of the R squared metric. This can help you determine the optimal task or the task that is suitable for specific datasets.

     • In most cases, the value of the Mean squared error metric is impossible to be 0, and the value of the R squared metric is impossible to be 1. The reason is that the prediction result of a task is affected by multiple factors. You may not be able to take into account all these factors and eliminate errors.

     • The value NaN of the Mean squared logarithmic error metric indicates that the prediction result or the actual value is a non-positive number or 0.

     For more information, see Evaluating data frame analytics.

Use the flight delay prediction task

You can use the flight delay prediction task based on the inference processor provided by Kibana.

1. Click the icon in the upper-left corner of the Kibana console. In the left-side navigation pane, choose Management > Dev Tools.

2. On the Console tab, run the following command to view and record the value of the model_id parameter:

   GET _ml/inference/flightdelaymin_job*?human=true

   This command is used to query all inference and analysis results of the job whose ID is flightdelaymin_job and return the results in a human-readable format. flightdelaymin_job in the command specifies the ID of the job that is created for training the regression and inference task.

3. Create a pipeline based on the regression and inference task and the inference processor.

   Note

   You must replace the value of the model_id parameter in the following command with the value that you obtained in the preceding step.

   PUT _ingest/pipeline/flight_flightDelayMin_predict
   {
     "description": "Predict the number of minutes of delay for each flight",
     "processors": [
       {
         "inference": {
           "model_id": "flightDelayMin_job-168609891****",
           "inference_config": {
             "regression": {}
           },
           "field_map": {},
           "tag": "flightDelayMin_prediction"
         }
       }
     ]
   }

4. Perform data analysis and prediction on the FlightDelayMin variable based on data in the kibana_sample_data_flights index.

   POST _ingest/pipeline/flight_flightDelayMin_predict/_simulate
   {
     "docs": [
       {
         "_source": {
           "FlightNum": "EDGSV3T",
           "DestCountry": "CN",
           "OriginWeather": "Damaging Wind",
           "OriginCityName": "Durban",
           "AvgTicketPrice": 1065.7037805199147,
           "DistanceMiles": 7273.460817641552,
           "FlightDelay": true,
           "DestWeather": "Rain",
           "Dest": "Shanghai Pudong International Airport",
           "FlightDelayType": "Carrier Delay",
           "OriginCountry": "ZA",
           "dayOfWeek": 5,
           "DistanceKilometers": 11705.500526106527,
           "timestamp": "2023-06-03T09:34:00",
           "DestLocation": {
             "lat": "31.14340019",
             "lon": "121.8050003"
           },
           "DestAirportID": "PVG",
           "Carrier": "Kibana Airlines",
           "Cancelled": false,
           "FlightTimeMin": 881.1071804361806,
           "Origin": "King Shaka International Airport",
           "OriginLocation": {
             "lat": "-29.61444444",
             "lon": "31.11972222"
           },
           "DestRegion": "SE-BD",
           "OriginAirportID": "DUR",
           "OriginRegion": "SE-BD",
           "DestCityName": "Shanghai",
           "FlightTimeHour": 14.685119673936343,
           "FlightDelayMin": 45
         }
       }
     ]
   }

   The following figure shows the result.

   

   The actual value of the FlightDelayMin variable is 45 minutes. The value of the FlightDelayMin_prediction variable obtained in the prediction is 50.28 minutes, which is close to the actual value.

   The feature_importance parameter returns the top five factors that most significantly affect the prediction result of flight delays: FlightTimeMin, DistanceKilometers, DistanceMiles, OriginAirportID, and FlightTimeHour. You can change the values of these factors to obtain different prediction results. This way, the task can predict the delay time of each flight in a more accurate manner.