Elasticsearch:Detect and predict data with Elasticsearch machine learning

Train a flight delay prediction model

In this section, you train a supervised machine learning model using regression. This prediction helps airlines and passengers better plan flights and schedules.

The following example shows a data record from the Sample flight data dataset.

{
  "_index": "kibana_sample_data_flights",
  "_type": "_doc",
  "_id": "7b0aeogBmNQSVxOwslB_",
  "_version": 1,
  "_score": null,
  "_source": {
    "FlightNum": "QYX9S3I",
    "DestCountry": "CH",
    "OriginWeather": "Cloudy",
    "OriginCityName": "Chicago",
    "AvgTicketPrice": 824.8516378170061,
    "DistanceMiles": 4442.909325899777,
    "FlightDelay": false,
    "DestWeather": "Thunder & Lightning",
    "Dest": "Zurich Airport",
    "FlightDelayType": "No Delay",
    "OriginCountry": "US",
    "dayOfWeek": 4,
    "DistanceKilometers": 7150.1694661808515,
    "timestamp": "2023-06-02T07:28:15",
    "DestLocation": {
      "lat": "47.464699",
      "lon": "8.54917"
    },
    "DestAirportID": "ZRH",
    "Carrier": "Logstash Airways",
    "Cancelled": false,
    "FlightTimeMin": 420.59820389299125,
    "Origin": "Chicago O'Hare International Airport",
    "OriginLocation": {
      "lat": "41.97859955",
      "lon": "-87.90480042"
    },
    "DestRegion": "CH-ZH",
    "OriginAirportID": "ORD",
    "OriginRegion": "US-IL",
    "DestCityName": "Zurich",
    "FlightTimeHour": 7.009970064883188,
    "FlightDelayMin": 0
  }
}

1. In the Kibana left navigation pane, click the menu icon and then click Kibana>Machine learning.

2. In the left navigation pane, click Data frame analytics > Job.

3. On the Data frame analytics jobs page, click Create job.

4. Select the kibana_sample_data_flights data view.

5. In the Configure section of the Create job page, configure the job parameters.

   1. Select regression as the analytics type.

   2. For the dependent variable, select FlightDelayMin. This is the field the model will learn to predict.

   3. In the included fields, exclude the Cancelled, FlightDelay, and FlightDelayType fields.

      These fields are directly related to the outcome. Including them would cause the model to overfit, which reduces its prediction accuracy on new data.

   4. Adjust the training percentage.

      This setting splits your data into a training set and a testing set. This tutorial uses a training percentage of 90%.

      Note

      Training time increases with the amount of data. For very large datasets, start with a smaller training percentage, such as 50%, and adjust it as needed to balance training time and model accuracy.

6. Click Continue and configure Additional options.

   1. Set feature importance values to 5. This calculates the top five features that contribute most to the predictions, which helps with feature selection and job optimization.

   2. Set Prediction field name to FlightDelayMin_prediction. This specifies the name of the field that will store the prediction results.

   3. Clear the Use estimated model memory limit checkbox and set the Model memory limit to 500MB. This specifies the maximum memory the job can use. Complex jobs on large datasets can consume significant memory. If the job exceeds this limit, it may fail or perform poorly.

   4. Set The maximum number of threads to 1. This parameter specifies the maximum number of threads for the training job. Using too many threads can lead to memory issues or system instability.

   5. Click Continue and set the job ID to flightdelaymin_job.

   6. Click Continue to validate the job configuration.

   7. After the validation passes, click Continue.

   8. In the Create section, click Create to start the job.

      The time required to create the job depends on the amount of training data.

7. After the job is complete, click View Results to see the analysis.

   In the model evaluation section, you can review the model's performance.

   • Generalization error: Measures the model's performance on new, unseen data. A lower generalization error indicates that the model is more accurate on unknown data.

   • Training error: Measures the model's performance on the training dataset. A lower training error indicates that the model fits the training data well.

   • Evaluation metrics:

     • mean squared error: An important metric for evaluating regression performance. It is the average of the squared differences between the actual and predicted values. A smaller value indicates more accurate predictions.

     • R squared: A metric that indicates how well the model fits the data. A value closer to 1 indicates a better fit. A value over 0.8 is generally considered a good fit.

     • mean squared logarithmic error: The mean of the squared differences between the logarithm of the actual and predicted values. A smaller value indicates better prediction performance.

     Note

     When you compare multiple regression models, consider both the mean squared error and R squared values to find the model that best fits your dataset.

     • A mean squared error of 0 or an R squared value of 1 is usually impossible, as model predictions are influenced by many factors that cannot be fully accounted for.

     • A mean squared logarithmic error of NaN (Not a Number) indicates that the actual or predicted value was a non-positive number or zero.

     For more information, see Evaluating data frame analytics.

Use the trained model for inference

Use an inference processor in an ingest pipeline to make predictions with your trained flight delay model.

1. In the Kibana left navigation pane, click the menu icon and then click Management > Developer Tools.

2. In the Console, run the following command to find and record the model_id.

   GET _ml/inference/flightdelaymin_job*?human=true

   This command queries all inference analysis results for jobs that match flightdelaymin_job and returns the output in a human-readable format. The flightdelaymin_job is the job ID you defined when creating the regression job.

3. Create an ingest pipeline that uses the inference processor and your trained model.

   Note

   Replace the model_id in the command with the value you obtained in the previous step.

   PUT _ingest/pipeline/flight_flightDelayMin_predict
   {
     "description": "Predict the number of minutes of delay for each flight",
     "processors": [
       {
         "inference": {
           "model_id": "flightDelayMin_job-168609891****",
           "inference_config": {
             "regression": {}
           },
           "field_map": {},
           "tag": "flightDelayMin_prediction"
         }
       }
     ]
   }

4. Simulate the pipeline to make a prediction on new data, using FlightDelayMin as the target variable.

   POST _ingest/pipeline/flight_flightDelayMin_predict/_simulate
   {
     "docs": [
       {
         "_source": {
           "FlightNum": "EDGSV3T",
           "DestCountry": "CN",
           "OriginWeather": "Damaging Wind",
           "OriginCityName": "Durban",
           "AvgTicketPrice": 1065.7037805199147,
           "DistanceMiles": 7273.460817641552,
           "FlightDelay": true,
           "DestWeather": "Rain",
           "Dest": "Shanghai Pudong International Airport",
           "FlightDelayType": "Carrier Delay",
           "OriginCountry": "ZA",
           "dayOfWeek": 5,
           "DistanceKilometers": 11705.500526106527,
           "timestamp": "2023-06-03T09:34:00",
           "DestLocation": {
             "lat": "31.14340019",
             "lon": "121.8050003"
           },
           "DestAirportID": "PVG",
           "Carrier": "Kibana Airlines",
           "Cancelled": false,
           "FlightTimeMin": 881.1071804361806,
           "Origin": "King Shaka International Airport",
           "OriginLocation": {
             "lat": "-29.61444444",
             "lon": "31.11972222"
           },
           "DestRegion": "SE-BD",
           "OriginAirportID": "DUR",
           "OriginRegion": "SE-BD",
           "DestCityName": "Shanghai",
           "FlightTimeHour": 14.685119673936343,
           "FlightDelayMin": 45
         }
       }
     ]
   }

   The result is as follows.

   "flightDelayMin_prediction" : {
     "FlightDelayMin_prediction" : 50.278760382477095,
     "feature_importance" : [
       {
         "feature_name" : "FlightTimeMin",
         "importance" : 43.5994522867254
       },
       {
         "feature_name" : "DistanceKilometers",
         "importance" : -16.095227076624866
       },
       {
         "feature_name" : "DistanceMiles",
         "importance" : -15.496369672570939
       },
       {
         "feature_name" : "OriginAirportID",
         "importance" : -5.120423036009302
       },
       {
         "feature_name" : "FlightTimeHour",
         "importance" : 3.944654321505046
       }
     ],
     "model_id" : "flightdelaymin_job3-1686721092815"
   }

   The actual flight delay (FlightDelayMin) was 45 minutes, and the predicted delay (FlightDelayMin_prediction) is 50.28 minutes, which is a close estimate.

   The feature_importance object shows the top five factors that contributed most to the prediction: FlightTimeMin, DistanceKilometers, DistanceMiles, OriginAirportID, and FlightTimeHour. You can use this information to better understand the factors that influence flight delays and potentially refine your model.