Cloud Monitor:Best practices for importing CMDB data into Cloud Monitor 2.0

Core concept mapping

Core transformation principles

• Model definition layer: Configuration Item Type → EntitySet

• Instance data layer: Configuration Item Instance → Entity

• Relationship definition layer: Configuration Item Type Relationship → EntitySetLink

• Relationship instance layer: Configuration Item Type Relationship → Topo

Entity type mapping table

Based on the example scenario, the nine Configuration Item Types are mapped to UModel as follows:

2.1 Configure mapping rules

Edit the config.yaml file to configure the mapping from Configuration Item Type to EntitySet:

ci_type_mapping:
  business_unit:
    entity_set: "cmdb.business_unit"
    domain: "cmdb"
    primary_keys: ["bu_name"]
    name_fields: ["bu_name"]
    
  product:
    entity_set: "cmdb.product"
    domain: "cmdb"
    primary_keys: ["product_name"]
    name_fields: ["product_name"]
    
  application:
    entity_set: "cmdb.application"
    domain: "cmdb"
    primary_keys: ["project_name"]
    name_fields: ["project_name"]
    
  server:
    entity_set: "cmdb.server"
    domain: "cmdb"
    primary_keys: ["server_name"]
    name_fields: ["server_name", "private_ip"]

Configuration description:

• entity_set: The UModel entity set name.

• primary_keys: The field that uniquely identifies the entity.

• name_fields: The fields used for display.

2.2 Run the generation tool

python3 generate_umodel_config.py

1. Log on to the Cloud Monitor 2.0 console.

2. Go to the target workspace. In the navigation pane on the left, select UModel Explorer.

3. On the UModel Explorer page, click the icon in the upper-right corner. In the dialog box that appears, click Upload UModel YAML/JSON.

4. In the Batch Upload UModel dialog box, click the icon to select files, or drag the files to the upload box. Upload all YAML files from the umodel_config/ folder.

   

5. After the files are uploaded, click Import.

6. After the import is successful, click Submit in the upper-right corner.

7. On the submission preview page, verify the content and then click Confirm that the content is as expected and execute the change. In the Change Status dialog box, click OK.

8. Verification: In the entity set list, you should see nine new cmdb.* entity sets.

Entity data format transformation

The VeOps API uses /api/v0.1/ci to retrieve all entity data in the following format:

GET /api/v0.1/ci?ci_type=server&page=1&count=100

Sample response data:

{
    "_id": 1234,
    "ci_id": "272",
    "ci_type_id": "4",
    "bu": "Cloud Computing",
    "buy_date": "2023-01-08",
    "ci_type": "server",
    "ci_type_alias": "Physical Server",
    "cmc_ip": "28.191.122.80",
    "cnc_ip": "224.127.127.194",
    "cpu": "AMD EPYC 7542",
    "cpu_count": "2",
    "ctc_ip": "75.76.9.36",
    "device_spec": "Huawei RH2288H V5",
    "env": "test",
    "idc": "Alibaba Cloud-Heyuan",
    "ilo_ip": "10.161.85.215",
    "ilo_mac": "05:0f:a9:63:91:ad",
    "kernel_version": "3.10.0-1160.el7.x86_64",
    "logic_cpu_count": "40",
    "manufacturer": "Dell",
    "oneagent_id": "XD1VLEEIR15O",
    "op_duty": "[\"Zhang Lihua\"]",
    "os_version": "CentOS 7.9",
    "perm": "{\"execute\": false, \"groups\": [\"viewers\", \"admins\", \"guests\"], \"owner\": \"Zhang Zhiqiang\", \"read\": true, \"write\": true}",
    "private_ip": "10.122.144.73",
    "rack": "59",
    "raid": "RAID 10",
    "ram": "64GB DDR4",
    "ram_size": "64",
    "rd_duty": "[\"Qi Jing\"]",
    "server_name": "phy-server-011",
    "server_room": "Alibaba Cloud-Heyuan Data Center",
    "sn": "C2LVEAZRIVH1",
    "ssh_port": "22",
    "status": "Online",
    "unique": "sn",
    "unique_alias": "Server Serial Number",
    "vnc_port": "5910",
    "__pack_meta__": "0|MTc2MjgzMzA5NTg5OTYzMjg1OQ==|32|3",
    "__topic__": "",
    "__source__": "30.221.145.27",
    "__tag__:__receive_time__": "1763629311",
    "__time__": "1763629311",
    "__time_ns_part__": "127734016"
}

After transformation, the data is written to ${workspace}__entity. The system field mappings are as follows:

Complete example:

{
    "__domain__": "cmdb",
    "__entity_type__": "cmdb.server",
    "__entity_id__": "b4da1c11d1a62257087aab071d27c1bc",
    "__method__": "Update",
    "__last_observed_time__": "1763629311",
    "__keep_alive_seconds__": "3600",
    "ci_id": "272",
    "ci_type_id": "4",
    "bu": "Cloud Computing",
    "buy_date": "2023-01-08",
    "ci_type": "server",
    "ci_type_alias": "Physical Server",
    "cmc_ip": "28.191.122.80",
    "cnc_ip": "224.127.127.194",
    "cpu": "AMD EPYC 7542",
    "cpu_count": "2",
    "ctc_ip": "75.76.9.36",
    "device_spec": "Huawei RH2288H V5",
    "env": "test",
    "idc": "Alibaba Cloud-Heyuan",
    "ilo_ip": "10.161.85.215",
    "ilo_mac": "05:0f:a9:63:91:ad",
    "kernel_version": "3.10.0-1160.el7.x86_64",
    "logic_cpu_count": "40",
    "manufacturer": "Dell",
    "oneagent_id": "XD1VLEEIR15O",
    "op_duty": "[\"Zhang Lihua\"]",
    "os_version": "CentOS 7.9",
    "perm": "{\"execute\": false, \"groups\": [\"viewers\", \"admins\", \"guests\"], \"owner\": \"Zhang Zhiqiang\", \"read\": true, \"write\": true}",
    "private_ip": "10.122.144.73",
    "rack": "59",
    "raid": "RAID 10",
    "ram": "64GB DDR4",
    "ram_size": "64",
    "rd_duty": "[\"Qi Jing\"]",
    "server_name": "phy-server-011",
    "server_room": "Alibaba Cloud-Heyuan Data Center",
    "sn": "C2LVEAZRIVH1",
    "ssh_port": "22",
    "status": "Online",
    "unique": "sn",
    "unique_alias": "Server Serial Number",
    "vnc_port": "5910",
    "__pack_meta__": "0|MTc2MjgzMzA5NTg5OTYzMjg1OQ==|32|3",
    "__topic__": "",
    "__source__": "30.221.145.27",
    "__tag__:__receive_time__": "1763629311",
    "__time__": "1763629311",
    "__time_ns_part__": "127734016"
}

Relationship data format transformation

VeOps uses /api/v0.1/relation_ci to retrieve configuration item relationship data. The API call is as follows:

GET /api/v0.1/relation_ci?parent_type=product&child_type=application&level=1

Sample response data:

{
  "parent_id": 100,
  "child_id": 200,
  "relation_type": "contains"
}

{
    "__src_domain__": "cmdb",
    "__src_entity_type__": "cmdb.server",
    "__src_entity_id__": "cb18feff0a2faa1b007000773c717942",
    "__dest_domain__": "cmdb",
    "__dest_entity_type__": "cmdb.ram",
    "__dest_entity_id__": "b4da1c11d1a62257087aab071d27c1bc",
    "__relation_type__": "contains",
    "__method__": "Update",
    "__last_observed_time__": "1763703248"
}

2.1 Configure the synchronization connection

Edit the config.yaml file to configure the CMDB and Simple Log Service connection information:

# CMDB configuration
cmdb:
  endpoint: "http://cmdb.example.com"
  api_key: "your-api-key"
  secret: "your-api-secret"

# SLS configuration
sls:
  endpoint: "cn-hangzhou.log.aliyuncs.com"
  access_key_id: "YOUR_ACCESS_KEY_ID"
  access_key_secret: "YOUR_ACCESS_KEY_SECRET"
  project: "cms-workspace-project"
  entity_logstore: "workspace__entity"
  topo_logstore: "workspace__topo"

2.2 Perform one-click synchronization

Run the full synchronization command to synchronize all entity and relationship data:

python3 main.py --mode full --all

2.3 Set up scheduled synchronization

UModel entities require regular updates to the last_observed_time field to maintain their active status (KeepAlive mechanism). For actual deployments, a combination of full and incremental synchronization is recommended:

Full synchronization

• Scans all configuration item instances and relationships in the CMDB and rewrites them to the EntityStore.

• Ensures data integrity and fixes any data that might have been missed during incremental synchronization.

• Use cases: Initial import, data repair, and periodic data validation (such as running daily at midnight).

Incremental synchronization

• Synchronizes only the data that has changed since the last synchronization (relies on the updated_at field in the CMDB).

• Fast synchronization with low resource consumption, suitable for frequent execution.

• Use cases: Continuous daily synchronization (such as running every 5 to 15 minutes).

This example uses a crontab scheduled task to perform full synchronization:

# Edit crontab
crontab -e

# Add a scheduled task (runs once per hour)
0 * * * * cd /path/to/cmdb_sync && python3 main.py --mode full --all >> logs/cron.log 2>&1

1. Entity list

You can view all entities synchronized from the CMDB.

2. Entity relationship topology

This topology visually displays the complete relationship network between entities.

3. Individual entity topology

View the relationships of a specific entity.

Trace data query

.topo | graph-call cypher(`
        MATCH (src {__entity_type__:"cmdb.business_unit"})-[e1]-(n1)-[e2]-(n2)
        RETURN src, e1, n1, e2, n2
      `)

Multi-hop query

.topo | graph-call cypher(`
          MATCH (src {__entity_type__:"cmdb.business_unit"})-[e*3..4]->(dest)
          WHERE dest.__domain__ = 'cmdb'
          RETURN src, dest, dest.__entity_type__
      `)

Get available methods for a single entity

.entity_set with(domain='cmdb', name='cmdb.business_unit', ids=['cc723a11313b5b16a20806a64b89a212'])  | entity-call __list_method__()

Get neighbor nodes

.entity_set with(domain='cmdb', name='cmdb.business_unit', ids=['cc723a11313b5b16a20806a64b89a212'])  | entity-call get_neighbor_entities()

Use high-order operators for advanced capabilities

When an entity is connected to other observable data, you can use high-order Structured Process Language (SPL) operators for advanced capabilities. The following example shows how to perform anomaly detection on the request latency metric of an application:

.entity_set with(domain='apm', name='apm.service', ids=['e22adf09a11550f4d7eae98d8a11a1e9']) 
| entity-call get_metric('apm', 'apm.metric.apm.service', 'avg_request_latency_seconds', aggregate=false) 
| extend r = series_decompose_anomalies(__value__) 
| extend anomaly_b =r.anomalies_score_series , anomaly_type = r.anomalies_type_series , __anomaly_msg__ = r.error_msg  
| extend x = zip(anomaly_b, __ts__, anomaly_type, __value__) 
| extend __anomaly_rst__ = filter(x, x-> x.field0 > 0) 
| project __entity_id__, __labels__, __anomaly_rst__, __anomaly_msg__

The __anomaly_rst__ field returns the time of the anomaly, the type of anomaly, and the value at that time.