Cloud Monitor:Entity data model and ingestion

System fields

Field descriptions

Custom fields

You can add any custom fields defined in the corresponding EntitySet.

{
  "__domain__": "apm",
  "__entity_type__": "apm.service",
  "__entity_id__": "391274C9A1D369FB5222E273D02B81B8",
  "__method__": "Update",
  "__last_observed_time__": 1700000001,
  "__keep_alive_seconds__": 3600,
  "service_name": "user-service",
  "version": "v1.2.3",
  "cluster": "production",
  "labels": {
    "env": "prod",
    "team": "backend"
  }
}

Time range concepts

Each entity has a visibility range. The query engine returns an entity only when the query window intersects its visibility range.

Time intersection logic

The query engine returns an entity when its visibility range overlaps with the query time window.

An entity is returned if: QueryTimeWindow ∩ EntityVisibleTimeRange is not empty.

The intersection logic is as follows:

if (QueryEndTime >= EntityFirstObservedTime && 
    QueryStartTime < EntityVisibleEndTime) {
    // Return the entity
} else {
    // Do not return the entity
}

Create

Use case: Use only when you are certain a new entity is being created, such as a Pod creation event.

Behavior:

• If the entity exists, no action is taken.

• If the entity does not exist, creates it with the provided fields.

Example:

{
  "__domain__": "k8s",
  "__entity_type__": "k8s.pod",
  "__entity_id__": "abc123...",
  "__method__": "Create",
  "__first_observed_time__": 1700000000,
  "__last_observed_time__": 1700000000,
  "pod_name": "web-app-123",
  "namespace": "default"
}

Update

Use case: Suitable for periodic full syncs and incremental changes.

Behavior:

• Overwrites all fields except __first_observed_time__.

• Handling of __first_observed_time__:

  • If the entity exists, preserves its __first_observed_time__.

  • If the entity does not exist and the field is provided, uses the given value.

  • If the entity does not exist and the field is omitted, defaults to __last_observed_time__.

Example:

{
  "__domain__": "apm",
  "__entity_type__": "apm.service",
  "__entity_id__": "def456...",
  "__method__": "Update",
  "__last_observed_time__": 1700000100,
  "service_name": "payment-service",
  "status": "running",
  "instance_count": 3
}

Expire

Use case: Logically delete an entity while preserving its historical record, such as on a Pod deletion event.

Behavior:

• Sets an internal __deleted__ flag to true.

• Updates __last_observed_time__.

• KeepAliveSeconds is no longer counted in queries.

Time range change:

• Before expiration: [FirstObservedTime, LastObservedTime + KeepAliveSeconds).

• After expiration: [FirstObservedTime, LastObservedTime)

Example:

{
  "__domain__": "k8s",
  "__entity_type__": "k8s.pod",
  "__entity_id__": "abc123...",
  "__method__": "Expire",
  "__last_observed_time__": 1700000200
}

Delete (use with extreme caution)

Use case: Make an entity invisible across its entire history, such as when a feature is disabled.

Behavior:

• Physically purges the entity record from the database.

• Entity becomes unqueryable across all time ranges.

⚠️ Warning: This operation is irreversible. Use it only when absolutely necessary.

Revise (use with extreme caution)

Use case: Forcibly correct incorrectly written data. Often used with Delete.

Behavior:

• Full overwrite of all fields, including __first_observed_time__.

• Ignores the entity's previous state entirely.

Recommended: Incremental events + periodic full syncs

Combine real-time incremental events with periodic full syncs for data reconciliation and fault tolerance.

Configuration recommendations:

• Incremental events: Write in real time or near real time.

• Periodic full sync: Run every hour or longer.

• Keep-alive time: Typically 10 minutes to 1 hour, based on business needs.

• Note: Set __keep_alive_seconds__ longer than your reporting interval. Use a reporting interval of 5 minutes or longer to avoid performance degradation.

Practical scenarios

// Cloud resource creation event
{
  "__domain__": "acs",
  "__entity_type__": "acs.ecs.instance",
  "__entity_id__": "i-bp1234567890",
  "__method__": "Create",
  "__first_observed_time__": 1700000000,
  "__last_observed_time__": 1700000000,
  "__keep_alive_seconds__": 86400,
  "instance_id": "i-bp1234567890",
  "instance_name": "web-server-01",
  "instance_type": "ecs.c6.large",
  "status": "Running"
}

// K8s Pod full collection data
{
  "__domain__": "k8s",
  "__entity_type__": "k8s.pod",
  "__entity_id__": "pod-web-app-123",
  "__method__": "Update",
  "__last_observed_time__": 1700000400,
  "__keep_alive_seconds__": 3600,
  "pod_name": "web-app-123",
  "namespace": "production",
  "node_name": "worker-01",
  "pod_ip": "10.0.1.100",
  "labels": {
    "app": "web-app",
    "version": "v1.0"
  }
}

Discouraged ingestion patterns