# Set up Slack alerts for a Kubernetes cluster

## **Introduction**

Alerting is essential for monitoring modern systems, ensuring issues are detected and communicated in real-time. In Kubernetes, tools like Prometheus and Alertmanager handle metrics monitoring and alert routing to channels like Slack or email. Effective alerting minimizes downtime, improves response times, and maintains system performance.

## Prerequisites

* A Kubernetes cluster with Prometheus and Alertmanager installed. It comes by default when installing cattle-monitoring-system.
* A Slack workspace with a channel for receiving alerts.
* Permissions to create and manage Slack webhooks.

## Procedure

1. **Create a Slack Incoming Webhook.**
   1. Go to the Slack [API Webhooks page](https://api.slack.com/messaging/webhooks).
   2. Create a new Slack App (if you don’t have one already).
   3. Add an **Incoming Webhook** to the app.
   4. Select the Slack channel where you want alerts to appear.
   5. Copy the generated Webhook URL (e.g., `https://hooks.slack.com/services/...`).
2. **Deploy alerting configuration to kuberenets cluster.**
   1. Clone the [repository](https://github.com/OpenG2P/openg2p-deployment/tree/main/alerting) to you local and update the required changes mentioned below.
   2. Update **slack\_api\_url** and **channel** in **alertmanager.yaml**
   3. Modify the `routes` and `receivers` in the **alertmanager.yaml** configuration to customize which alerts are sent to Slack., as shown below.

      ```yaml
      routes:
      - match:
          alertname: Watchdog
        receiver: 'null'
      ```
   4. The monitoring package provided by Rancher includes various default alerting rules, which are often sufficient for most use cases. Sample custom alerts are available under the `custom-alerts` directory. Modify these as needed.
   5. Add cluster name in, `patch-cluster-name.yaml`
   6. Run **install.sh** to apply the configuration for alertmanager.
   7. And restart the alert-manager service on k8s-cluster.
   8. Verify whether the alerts are firing from the Prometheus UI and check if Slack notifications are being received.<br>

      <figure><img src="/files/BL5Sh5aeGvfmOX0S2wrN" alt=""><figcaption><p><em><strong>Prometheus dashboard</strong></em><br></p></figcaption></figure>

      <figure><img src="/files/H58zG9q1hYCuiXKTmfQd" alt=""><figcaption><p><em><strong>slack channel</strong></em></p></figcaption></figure>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.openg2p.org/operations/deployment/deployment-guide/set-up-slack-alerts-for-a-kubernetes-cluster.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.