Monitor Cassandra with Prometheus JMX agent - Managed Service for Prometheus

Prerequisites

A Prometheus instance for ECS is created. For more information, see Monitor an ECS instance.

Limits

You can install the component only for Prometheus instances for ECS.

Step 1: Deploy a Cassandra JMX agent

Download the appropriate Cassandra JMX agent to the Elastic Compute Service (ECS) instance where Cassandra runs.
- Cassandra 4.1.x and later
- Cassandra 4.0.x and earlier
Extract the package to MCAC_ROOT and add the following to the cassandra-env.sh file:
```
MCAC_ROOT=/path/to/directory
JVM_OPTS="$JVM_OPTS -javaagent:${MCAC_ROOT}/lib/datastax-mcac-agent.jar"
```
Important
The default port exposed to Managed Service for Prometheus is 9103. To change it, edit the ${MCAC_ROOT}/config/collectd.conf.tmpl file.
Restart Cassandra and run curl localhost:{jmx port}/metrics on the ECS instance. If data is returned, the agent is installed.

Step 2: Integrate Cassandra

Procedure

Option 1: From the Prometheus instance integration center

Log on to the Cloud Monitor console.
In the navigation pane on the left, choose Managed Service for Prometheus > Instances to open the instance list for Managed Service for Prometheus.
Click the name of the Managed Service for Prometheus instance instance that you want to manage to go to the Integration Center page.

Option 2: From the ARMS console

Log on to the Application Real-Time Monitoring Service (ARMS) console.
In the left-side navigation pane, click Integration Center. In the Components section, find Cassandra and click Add. In the panel that appears, integrate Cassandra as prompted.

Integrate Cassandra

The following steps use the Prometheus instance integration center.

Install or add the Cassandra component.
- To install Cassandra for the first time:
  
  In the Not Installed section of the Integration Center page, find Cassandra and click Install.
  
  Note
  Click the card to preview metrics and dashboard thumbnails. Listed metrics are for reference only. After installation, view actual metrics collected by Managed Service for Prometheus. For more information, see Key metrics.
- To add another Cassandra instance:
  
  In the Installed section of the Integration Center page, find Cassandra and click Add.

On the Settings tab in STEP2, configure the following parameters and click OK.

Parameter	Description
Instance name	The name of the exporter. Allows only lowercase letters, digits, and hyphens (-). Cannot start or end with a hyphen. The name must be unique.
ECS Label Key (service discovery)	ECS tag for exporter service discovery. Valid values: acs:emr:nodeGroupType and acs:emr:hostGroupType.
ECS Label value	ECS tag values. Default: CORE,MASTER. Separate multiple values with commas.
JMX Agent listening port	Port from which Managed Service for Prometheus scrapes metrics. Default value: 9103.
Metrics path	HTTP path for metric collection. Default value: /metrics.
Metrics scrape interval (seconds)	Managed Service for Prometheus scrape interval in seconds. Default value: 30.

Note

View available metrics on the Metrics tab in STEP2.

The installed component is displayed on the Integration Management page in the Managed Service for Prometheus console. The Integration Management page consists of the Integrated Environments, Integrated Addons, and Query Dashboards tabs, where you can view information such as targets, metrics, dashboards, and alerts.

View exporter status on the Targets tab.

Step 3: View Cassandra dashboards

The Dashboards tab shows metrics including availability, client read/write latency, throughput, and node resource usage.

On Integration Center, find Cassandra in Installed and click the Dashboards tab. Click a dashboard link to open the Grafana page. The following sections describe key dashboard metrics.

Cluster/Node Information section
Client Read Latency, Write Delay, and Throughput section
Exceptions and Errors section
Caching and Bloom Filters section
Hardware resource usage section
Storage occupancy details section
Thread Pool Status section
JVM and Garbage Collection section

Step 4: Configure alerting

On Integration Center, find Cassandra in Installed and click the Alerts tab to view configured alert rules.

Managed Service for Prometheus provides built-in dashboards and alert rules for Cassandra exporters, managed through the ARMS console.

Managed Service for Prometheus provides default alert rules for key Cassandra metrics. The following table lists these rules.

Category	Metric	Description
Node status	Proportion of inactive nodes in the cluster	If the value is greater than 10, one or more nodes in the cluster are down.
Resource usage	CPU utilization	Triggers when CPU utilization exceeds 85% over 5 minutes.
	Memory usage	Triggers when memory usage exceeds 85%.
	Hard disk usage	Triggers when disk usage exceeds 85%.
Read and write latency and throughput	Read latency	Triggers when read latency exceeds 200 ms over 1 minute.
	Write latency	Triggers when write latency exceeds 200 ms over 1 minute.
	Read throughput	Triggers when read operations exceed 1,000 per minute.
	Write throughput	Triggers when write operations exceed 1,000 per minute.
Exceptions and errors	Timed out requests	Triggers when timed-out requests exceed 10 per minute.
	Failed requests	Triggers when failed requests exceed 10 per minute.
	Dropped messages	Triggers when dropped messages exceed 10 per minute.
JVM	GC time ratio	Triggers when GC time exceeds 1% over 5 minutes.

You can also create alert rules based on your business requirements. For more information, see Create alert rules for a Prometheus instance.

Key metrics

Cluster and node information

Metric	Level	Description	Remarks
mcac_client_connected_native_clients	Major	Number of CQL connections	High values consume system resources and increase client latency.
mcac_table_live_disk_space_used_total	Major	Space occupied by Cassandra	High values may exhaust storage and increase access latency.
mcac_table_snapshots_size	Recommand	Cassandra snapshot file size	Used for data restoration. Large snapshots may exhaust storage.
collectd_uptime	Major	Node startup time	Long uptimes without restarts may expose security risks.

Key performance metrics

Metric	Level	Description	Remarks
mcac_table_read_latency	Critical	Client read latency	High latency slows application reads and degrades user experience.
mcac_table_write_latency	Critical	Client write latency	High latency slows application writes and degrades user experience.

Exceptions and errors

Metric	Level	Description	Remarks
mcac_client_request_timeouts_total	Critical	Timed out client requests	High values indicate system overload and severely affect user experience.
mcac_client_request_failures_total	Critical	Abnormal client requests	High values indicate system overload and severely affect user experience.
mcac_dropped_message_dropped_total	Critical	Dropped messages	High values indicate system overload and severely affect user experience.

Caching and Bloom filters

Metric	Level	Description	Remarks
mcac_table_key_cache_hit_rate	Major	Hit rate of key_cache	Low hit rates may slow reads and affect user experience.
mcac_table_row_cache_hit_total	Major	Number of hits of row_cache	Low hit rates may slow reads and affect user experience.
mcac_table_row_cache_miss_total	Recommand	Number of missed hits of row_cache	High values may slow reads and affect user experience.
mcac_table_row_cache_hit_out_of_range_total	Recommand	Number of times that row_cache hits but still accesses the disk	If the value is too large, the read speed of the application may be slow, affecting user experience.
mcac_table_bloom_filter_false_ratio	Major	False-positive rate of the Bloom filter	High false-positive rates cause non-existent elements to be misjudged as existent, wasting query time and resources, and degrading performance.

Usage trends in CPU, memory, and disks

Metric	Level	Description	Remarks
collectd_cpu_total	Critical	CPU utilization	High CPU utilization prolongs client request latency.
collectd_memory	Critical	Memory usage	High memory usage prolongs client request latency.
collectd_df_df_complex	Critical	Hard disk usage	Insufficient disk space prevents data persistence and may cause system crashes.

SSTable compression

Metric	Level	Description	Remarks
mcac_table_pending_compactions	Major	SSTable compression task in progress	High values indicate overload and increase client latency. Consider adjusting the SSTable compaction interval.
mcac_table_compaction_bytes_written_total	Major	SSTable compression speed	Low throughput causes task backlog. Consider upgrading node hardware.
mcac_table_compression_ratio	Major	SSTable compression ratio	High ratios indicate ineffective compaction.

Disk file

Metric	Level	Description	Remarks
mcac_table_live_ss_table_count	Major	Number of SSTables	High SSTable counts increase disk usage and read/write latency. Consider adjusting the compaction policy.
mcac_table_live_disk_space_used_total	Major	Hard disk space occupied by SSTable	High disk usage increases read/write latency. Consider adjusting the compaction policy.
mcac_table_ss_tables_per_read_histogram	Major	Number of SSTables for each read operation	If the value is too large, the client read latency is high.
mcac_commit_log_total_commit_log_size	Major	Hard disk space occupied by Commit Log	Large commit logs may exhaust disk space, degrade I/O performance, and extend recovery time.
mcac_table_memtable_live_data_size	Major	Space occupied by the MemTable	High values degrade write performance and node stability.
mcac_table_waiting_on_free_memtable_space	Major	Time spent waiting for the MemTable to be released	High values degrade write performance and node stability.

Thread pool status

Metric	Level	Description	Remarks
mcac_thread_pools_active_tasks	Critical	Number of active tasks in the thread pool	Excessive active tasks consume resources and may slow responses or cause crashes.
mcac_thread_pools_total_blocked_tasks_total	Critical	Number of blocked tasks in the thread pool	Excessive blocked tasks consume resources and may slow responses or cause crashes.
mcac_thread_pools_pending_tasks	Critical	Number of pending tasks in the thread pool	High pending counts consume resources. Timed-out pending tasks risk system crashes.
mcac_thread_pools_completed_tasks	Major	Number of completed tasks in the thread pool	Higher values indicate better system throughput.

JVM

Metric	Level	Description	Remarks
mcac_jvm_memory_used	Critical	Size of the used JVM heap memory	High heap usage triggers frequent GC and reduces application throughput.
mcac_jvm_gc_time	Critical	Time spent by the application in GC	Excessive GC time reduces capacity for user tasks and may cause request timeouts or crashes.