> ## Documentation Index
> Fetch the complete documentation index at: https://risingwavelabs-wyx-add-timestamp-dedicated-page.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitor a RisingWave cluster

> If you run RisingWave in a Kubernetes cluster, you can monitor its performance, including the usage of resources like CPU, memory, and network, and the status of different nodes.

## Install the monitoring stack

To monitor a RisingWave cluster, you can use the monitoring stack in the `RisingWave-Operator` repository. The stack includes Prometheus for collecting data, Grafana for visualization and alerting, and other necessary resources.

Clone or download the `RisingWave-Operator` repository to your environment.

```bash theme={null}
git clone https://github.com/risingwavelabs/risingwave-operator.git
```

Run the script to install the monitoring stack. It will create the `monitoring` namespace and deploy all monitoring services in the namespace.

```bash theme={null}
./monitoring/install.sh
```

## Start monitoring

The RisingWave operator is now integrated with the Prometheus operator, which creates a service monitor for RisingWave and ensures that data is collected automatically.

1. Check if Prometheus is collecting data from RisingWave:

```bash theme={null}
kubectl get servicemonitors -A -l risingwave/name
```

The expected output is like this:

```bash theme={null}
NAMESPACE    NAME                         AGE
monitoring   risingwave-service-monitor   119m
```

2. Forward the web port of Grafana to localhost.

```bash theme={null}
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:http-web
```

The command above allows you to access Grafana from the local machine only. If you need to access Grafana from a different host, append `--address 0.0.0.0` to the command to allow Grafana to accept requests from external hosts. The complete command is:

```bash theme={null}
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:http-web --address 0.0.0.0
```

3. Go to [http://localhost:3000](http://localhost:3000) if you access Grafana from a local machine, or `http://<client_address>:3000` if you access Grafana from a different host, where `<client_address>` is the IP address of the machine running the Grafana service. When prompted, enter the default credentials (username: `admin`; password: `prom-operator`).
4. From **Dashboards**, select **RisingWave** -> **RisingWave Dashboard**. You can edit metrics in the dashboard and add new metrics.

<Frame>
  <img src="https://mintcdn.com/risingwavelabs-wyx-add-timestamp-dedicated-page/5vQODE3orY0cH0d2/images/current/monitor-risingwave-cluster/risingwave-grafana-k8s.png?fit=max&auto=format&n=5vQODE3orY0cH0d2&q=85&s=af278cf6863090ea71fe6372404dde28" width="3840" height="1952" data-path="images/current/monitor-risingwave-cluster/risingwave-grafana-k8s.png" />
</Frame>

## Generate a Grafana dashboard for multiple RisingWave clusters

This section explains how to generate a Grafana dashboard for monitoring multiple RisingWave clusters deployed within a Kubernetes environment. These steps only generate the dashboard's JSON model and can be performed on any machine with the necessary tools.

1. Clone the [risingwave](https://github.com/risingwavelabs/risingwave) repository:

```bash theme={null}
git clone https://github.com/risingwavelabs/risingwave.git
```

2. Follow the instructions in the [RisingWave Grafana directory README](https://github.com/risingwavelabs/risingwave/blob/main/grafana/README.md) to configure the required toolchain.

3. Generate the Grafana dashboard JSON model.
   Execute the following command to create the Grafana dashboard JSON model, enabling multi-cluster support:

```bash theme={null}
DASHBOARD_NAMESPACE_FILTER_ENABLED=true \
DASHBOARD_RISINGWAVE_NAME_FILTER_ENABLED=true \
DASHBOARD_DYNAMIC_SOURCE=true \
DASHBOARD_UID=risingwave_dashboard \
./generate.sh
```

This command sets several environment variables to configure the dashboard generation:

* `DASHBOARD_NAMESPACE_FILTER_ENABLED=true`: Enables filtering by Kubernetes namespace.

* `DASHBOARD_RISINGWAVE_NAME_FILTER_ENABLED=true`: Enables filtering by RisingWave cluster name.

* `DASHBOARD_DYNAMIC_SOURCE=true`: Enables dynamic selection of the data source in Grafana.

* `DASHBOARD_UID=risingwave_dashboard`: Sets the unique identifier (UID) for the dashboard.

4. Import the generated JSON models (with the extension `.gen.json`) into your Grafana instance.

### Troubleshooting

#### Namespace filter not working

If the namespace filter in your Grafana dashboard is not functioning correctly, the issue may stem from the absence of the `risingwave_name` label in your collected metrics. To resolve this, you can use the metricRelabelings and relabelings fields in the [Prometheus Operator endpoint spec](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api-reference/api.md#endpoint) to append this label to your metrics.