How to set up monitoring

Charmed Apache Cassandra Snap come with the JMX exporter. The metrics can be queried by accessing the http://<cassandra-unit-ip>:7071/metrics endpoint.

Additionally, the charm provides integration with the Canonical Observability Stack.

Enable monitoring

Deploy the cos-lite bundle in a Kubernetes environment. This can be done by following the deployment tutorial. Since the Charmed Apache Cassandra is deployed directly on a cloud infrastructure environment, it is needed to offer the endpoints of the COS relations. The offers-overlay can be used, and this step is shown in the COS tutorial.

Offer interfaces via the COS controller

Switch to COS K8s environment and offer COS interfaces to be cross-model related with Charmed Apache Cassandra VM model:

juju switch <k8s_controller>:<cos_model_name>

juju offer grafana:grafana-dashboard grafana-dashboards
juju offer loki:logging loki-logging
juju offer prometheus:receive-remote-write prometheus-receive-remote-write

Consume offers via the Apache Cassandra model

Switch back to the Charmed Apache Cassandra model, find offers and relate with them:

juju switch <machine_controller_name>:<cassandra_model_name>

juju find-offers <k8s_controller>:

A similar output should appear, if k8s is the K8s controller name and COS the model where cos-lite has been deployed:

Store      URL                                        Access  Interfaces
k8s        admin/cos.grafana-dashboards               admin   grafana_dashboard:grafana-dashboard
k8s        admin/cos.loki-logging                     admin   loki_push_api:logging
k8s        admin/cos.prometheus-receive-remote-write  admin   prometheus-receive-remote-write:receive-remote-write
...

Consume offers to be reachable in the current model:

juju consume <k8s_controller>:admin/<cos_model_name>.prometheus-receive-remote-write
juju consume <k8s_controller>:admin/<cos_model_name>.loki-logging
juju consume <k8s_controller>:admin/<cos_model_name>.grafana-dashboards

Now, deploy grafana-agent (subordinate charm) and relate it with Charmed Apache Cassandra:

juju deploy grafana-agent
juju integrate cassandra:cos-agent grafana-agent

Finally, relate grafana-agent with consumed COS offers:

juju integrate grafana-agent grafana-dashboards
juju integrate grafana-agent loki-logging
juju integrate grafana-agent prometheus-receive-remote-write

Wait for all components to settle down on a active/idle state on both models, e.g. <cassandra_model_name> and <cos_model_name>.

After this is complete, the monitoring COS stack should be up and running and ready to be used.

Connect Grafana web interface

To connect to the Grafana web interface, follow the Browse dashboards section of the MicroK8s “Getting started” guide.

juju run grafana/leader get-admin-password --model <k8s_cos_controller>:<cos_model_name>

Alerts and dashboards

This guide shows you how to integrate an existing set of rules and/or dashboards to your Charmed Apache Cassandra deployment to be consumed with the Canonical Observability Stack (COS). To do so, we will sync resources stored in a git repository to COS Lite.

Prerequisites

Deploy the cos-lite bundle in a Kubernetes environment and integrate Charmed Apache Cassandra to the COS offers, as shown in the How to Enable Monitoring guide. This guide will refer to the models that charms are deployed into as:

  • <cos-model> for the model containing observability charms (and deployed on K8s)

  • <apps-model> for the model containing Charmed Apache Cassandra

  • <apps-model> for other optional charms (e.g. TLS-certificates operators, grafana-agent, data-integrator, etc.).

Create a repository with a custom monitoring setup

Create an empty git repository, or in an existing one, save your alert rules and dashboard models under the <path_to_prom_rules>, <path_to_loki_rules> and <path_to_models> folders.

If you want a primer to rule writing, refer to the Prometheus documentation.

Then, push your changes to the remote repository.

Deploy the COS configuration charm

Deploy the COS configuration charm in the <cos-model> model:

juju deploy cos-configuration-k8s cos-config \
  --config git_repo=<repository_url> \
  --config git_branch=<branch> \

The COS configuration charm keeps the monitoring stack in sync with our repository, by forwarding resources to Prometheus, Loki and Grafana. Refer to the documentation for all configuration options, including how to access a private repository.
Adding, updating or deleting an alert rule or a dashboard in the repository will be reflected in the monitoring stack.

Note

You need to manually refresh cos-config’s local repository with the sync-now action if you do not want to wait for the next update-status event to pull the latest changes.

Forward the rules and dashboards

The path to the resource folders can be set after deployment:

juju config cos-config \
  --config prometheus_alert_rules_path=<path_to_prom_rules> \
  --config loki_alert_rules_path=<path_to_loki_rules> \
  --config grafana_dashboards_path=<path_to_models>

Then, integrate the charm with the COS operators to forward the rules and dashboards:

juju integrate cos-config prometheus
juju integrate cos-config grafana
juju integrate cos-config loki

After this is complete, the monitoring COS stack should be up and ready to fire alerts based on our rules. As for the dashboards, they should be available in the Grafana interface.

Pre‑built dashboards and alerting rules

This repository already contains a set of ready‑to‑use monitoring assets:

  • Grafana dashboard: src/grafana_dashboards/grafana-dashboard.json

  • Loki alerting rules: src/alert_rules/loki/loki_alerts.yaml

  • Prometheus alerting rules: src/alert_rules/prometheus/prometheus_alerts.yaml

Point the corresponding *_path configuration options to these files (or their parent directories) if you would like COS to pick them up automatically.

Conclusion

In this guide, we enabled monitoring on a Charmed Apache Cassandra deployment and integrated alert rules and dashboards by syncing a git repository to the COS stack.