Monitor Cassandra with Prometheus - Managed Service for Prometheus

Prerequisites

You have created a Prometheus instance. For more information, see:

Step 1: Deploy the Cassandra JMX agent

Download the Cassandra JMX agent that matches your Cassandra version to the ECS instance running Cassandra.
- Cassandra 4.1.x and later
- Cassandra 4.0.x and earlier
Decompress the downloaded agent package to the MCAC_ROOT directory and append the following content to the cassandra-env.sh file.
```
MCAC_ROOT=/path/to/directory
JVM_OPTS="$JVM_OPTS -javaagent:${MCAC_ROOT}/lib/datastax-mcac-agent.jar"
```
Important
The Cassandra JMX agent exposes port 9103 for Prometheus to scrape data. To change this port, modify the value in the ${MCAC_ROOT}/config/collectd.conf.tmpl file to your desired port.
```
LoadPlugin write_prometheus
<Plugin write_prometheus>
  Port "9103"
</Plugin>
```

After completing the configuration, restart the Cassandra application. On the ECS server, run the curl localhost:{jmx_port}/metrics command. A successful installation returns data.

# HELP collectd_collectd_cache_size write_prometheus plugin: 'collectd' Type: 'cache_size', Dstype: 'gauge', Dsname: 'value'
# TYPE collectd_collectd_cache_size gauge
collectd_collectd_cache_size{collectd="cache",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1"} 6985 16
# HELP collectd_collectd_derive_total write_prometheus plugin: 'collectd' Type: 'derive', Dstype: 'derive', Dsname: 'value'
# TYPE collectd_collectd_derive_total counter
collectd_collectd_derive_total{collectd="write_queue",type="dropped",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1"} 0 168768
# HELP collectd_collectd_queue_length write_prometheus plugin: 'collectd' Type: 'queue_length', Dstype: 'gauge', Dsname: 'value'
# TYPE collectd_collectd_queue_length gauge
collectd_collectd_queue_length{collectd="write_queue",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1"} 144 168
# HELP collectd_contextswitch_total write_prometheus plugin: 'contextswitch' Type: 'contextswitch', Dstype: 'derive', Dsname: 'value'
# TYPE collectd_contextswitch_total counter
collectd_contextswitch_total{instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1"} 3... 168768
# HELP collectd_cpu_total write_prometheus plugin: 'cpu' Type: 'cpu', Dstype: 'derive', Dsname: 'value'
# TYPE collectd_cpu_total counter
collectd_cpu_total{cpu="0",type="idle",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 50160255 16876
collectd_cpu_total{cpu="0",type="interrupt",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 0 1687686
collectd_cpu_total{cpu="0",type="nice",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 226 1687686335
collectd_cpu_total{cpu="0",type="softirq",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 13644 16876
collectd_cpu_total{cpu="0",type="steal",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 0 16876863350
collectd_cpu_total{cpu="0",type="system",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 231356 16876
collectd_cpu_total{cpu="0",type="user",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 1519742 168768
collectd_cpu_total{cpu="0",type="wait",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 585 1687686335
collectd_cpu_total{cpu="1",type="idle",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 50596353 16876
collectd_cpu_total{cpu="1",type="interrupt",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 0 1687686
collectd_cpu_total{cpu="1",type="nice",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 219 1687686335
collectd_cpu_total{cpu="1",type="softirq",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 9320 168768
collectd_cpu_total{cpu="1",type="steal",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 0 16876863350
collectd_cpu_total{cpu="1",type="system",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 129114 16876
collectd_cpu_total{cpu="1",type="user",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 1504382 168768
collectd_cpu_total{cpu="1",type="wait",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 50115892 16876
collectd_cpu_total{cpu="2",type="softirq",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 4906 168768
collectd_cpu_total{cpu="2",type="steal",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 0 16876863350
collectd_cpu_total{cpu="2",type="system",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 230256 16876
collectd_cpu_total{cpu="2",type="user",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 1562142 168768
collectd_cpu_total{cpu="2",type="wait",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 8267 16876863
collectd_cpu_total{cpu="3",type="idle",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 50612210 16876
collectd_cpu_total{cpu="3",type="interrupt",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 0 1687686
collectd_cpu_total{cpu="3",type="nice",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 218 1687686335
collectd_cpu_total{cpu="3",type="softirq",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 2815 168768
collectd_cpu_total{cpu="3",type="steal",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 0 16876863350
collectd_cpu_total{cpu="3",type="system",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 129656 16876
collectd_cpu_total{cpu="3",type="user",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 1506299 168768
collectd_cpu_total{cpu="3",type="wait",instance="172.16.x.x",cluster="Test Cluster",dc="datacenter1",rack="rack1",insight_series="1"} 183 1687686335

Step 2: Integrate Cassandra

Log on to the Cloud Monitor console.
In the left-side navigation pane, click Access Center.
On the Access Center page, in the Database section, click Cassandra.

On the Start Integration tab of the Cassandra panel, configure the parameters and then click OK.

Parameter	Description
Select environment type	Select the environment type: Container Service environment ECS (Virtual Private Cloud (VPC))
Select cluster	Select the target cluster.
Pod label	The label of the pod that runs the Cassandra service. A unique label is recommended.
Service port	The port used by the Cassandra JMX agent installed in Step 1. The default port is automatically entered.
Metrics path	The metrics endpoint of the Cassandra JMX agent installed in Step 1. The default path is automatically entered.
Scrape interval (seconds)	The interval, in seconds, for scraping metrics. The default is 30.

Step 3: View Cassandra dashboards

Log on to the Cloud Monitor console.
In the left-side navigation pane, click Data Import Management.
On the Data Import Management page, click the Installed tab. Find the desired environment and click its name to open the details page.
On the Component Management tab, click Cassandra in the Component Type section, and then click the Dashboards tab to view all dashboard names.

The page displays two dashboards: Cassandra Detail and Cassandra Summary.
Click a dashboard name to view the Grafana dashboard.

Step 4: Configure Cassandra alerts

On the Component Management tab, click Cassandra in the Component Type section, and then click the Alert Rules tab to view the default alert rules.

The default alert rules include:
- Request timeouts
- Failed requests
- Total messages
- Disk usage
- Memory usage
- CPU usage
- Client connections
You can also create custom alert rules. For more information, see Create a Prometheus alert rule.

Key metrics

Cluster and node information

Metric	Level	Description	Explanation
mcac_client_connected_native_clients	Major	Number of CQL connections	An excessive number of connections consumes system resources and increases client latency.
mcac_table_live_disk_space_used_total	Major	Disk space used by Cassandra	A high value can cause insufficient disk space and increase data access latency.
mcac_table_snapshots_size	Recommended	Cassandra snapshot file size	Snapshots are used for data recovery. A high value can lead to insufficient disk space, preventing the storage of complete snapshots.
collectd_uptime	Major	Node uptime	A high value indicates that the system has not been restarted for a long time. If the system has known vulnerabilities, this may increase security risks.

Key performance metrics

Metric	Level	Description	Explanation
mcac_table_read_latency	Critical	Client read latency	A high value slows application reads and impacts user experience.
mcac_table_write_latency	Critical	Client write latency	A high value slows application writes and impacts user experience.

Exceptions and errors

Metric	Level	Description	Explanation
mcac_client_request_timeouts_total	Critical	Timed-out client requests	A high value indicates a heavy system load, which severely affects user experience.
mcac_client_request_failures_total	Critical	Client requests with exceptions	A high value indicates a heavy system load, which severely affects user experience.
mcac_dropped_message_dropped_total	Critical	Dropped messages	A high value indicates a heavy system load, which severely affects user experience.

Cache and Bloom filters

Metric	Level	Description	Explanation
mcac_table_key_cache_hit_rate	Major	Key cache hit rate	A low value can decrease application read performance and affect user experience.
mcac_table_row_cache_hit_total	Major	Number of row cache hits	A low value can decrease application read performance and affect user experience.
mcac_table_row_cache_miss_total	Recommended	Number of row cache misses	A high value can decrease application read performance and affect user experience.
mcac_table_row_cache_hit_out_of_range_total	Recommended	Number of row cache hits that still required disk access.	A high value can decrease application read performance and affect user experience.
mcac_table_bloom_filter_false_ratio	Major	Bloom filter false positive ratio	A high false positive ratio for the Bloom filter causes many non-existent elements to be incorrectly identified as existing. This wastes query time and resources, which degrades query performance and increases costs.

CPU, memory, and disk usage

Metric	Level	Description	Explanation
collectd_cpu_total	Critical	CPU utilization	A high value indicates a high system load, which increases client request latency and severely affects user experience.
collectd_memory	Critical	Memory usage	A high value indicates a high system load, which increases client request latency and severely affects user experience.
collectd_df_df_complex	Critical	Disk usage	A high value indicates insufficient available disk space, which can prevent data persistence and risk system downtime.

SSTable compaction

Metric	Level	Description	Explanation
mcac_table_pending_compactions	Major	SSTable compaction tasks in progress	A high value indicates a high system load, which increases client request latency. Configure a reasonable compaction interval for SSTables.
mcac_table_compaction_bytes_written_total	Major	SSTable compaction rate	A low value indicates a slow compaction rate, which can cause task buildup. Consider upgrading node specifications.
mcac_table_compression_ratio	Major	SSTable compression ratio	A high value indicates that the compressed files are still too large and compression is ineffective.

Disk files

Metric	Level	Description	Explanation
mcac_table_live_ss_table_count	Major	Number of SSTables	A high value increases disk usage and read/write latency. Configure a proper SSTable compaction strategy.
mcac_table_live_disk_space_used_total	Major	Disk space used by SSTables	A high value increases disk usage and read/write latency. Configure a proper SSTable compaction strategy.
mcac_table_ss_tables_per_read_histogram	Major	Number of SSTables read per read operation	A high value increases client read latency.
mcac_commit_log_total_commit_log_size	Major	Disk space used by the commit log	A high value can cause insufficient disk space, degraded read/write performance, and longer data recovery time.
mcac_table_memtable_live_data_size	Major	Space used by the MemTable	A high value can degrade data write performance and node stability.
mcac_table_waiting_on_free_memtable_space	Major	Time spent waiting for MemTable space to be freed	A high value can degrade data write performance and node stability.

Thread pool status

Metric	Level	Description	Explanation
mcac_thread_pools_active_tasks	Critical	Number of active tasks in the thread pool	Too many blocked tasks consume system resources, slow down responses, and may even crash the system.
mcac_thread_pools_total_blocked_tasks_total	Critical	Number of blocked tasks in the thread pool	Too many blocked tasks consume system resources, slow down responses, and may even crash the system.
mcac_thread_pools_pending_tasks	Critical	Number of pending tasks in the thread pool	Too many pending tasks consume excessive system resources, cause corresponding requests to time out, and can even cause the system to crash.
mcac_thread_pools_completed_tasks	Major	Number of completed tasks in the thread pool	This metric reflects the system's throughput. A higher value indicates better system performance.

JVM metrics

Metric	Level	Description	Explanation
mcac_jvm_memory_used	Critical	Used JVM heap memory	A high value can cause insufficient memory, which triggers frequent garbage collection and reduces application throughput.
mcac_jvm_gc_time	Critical	Time spent on garbage collection (GC)	A high value indicates frequent GC. The system has less time to run user workloads, which may cause request timeouts or even system crashes.