> For the complete documentation index, see [llms.txt](https://docs.openg2p.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openg2p.org/operations/deployment/deployment-guide/set-up-slack-alerts-for-a-kubernetes-cluster.md).

# Set up Slack alerts for a Kubernetes cluster

This guide explains how to configure Slack alerts for a Kubernetes cluster using Prometheus Alertmanager.

{% hint style="info" %}
**In production this is already wired up — you just supply the channel.** The infrastructure automation installs Prometheus + Alertmanager (metrics) and Grafana Loki (logs) with alert rules preconfigured. You enable notification channels by setting the corresponding keys in `prod-config.yaml` and re-running the compute stage — no manual `alertmanager.yaml` editing:

* **Slack** — `alert_slack_webhook_url`, `alert_slack_channel`
* **Email (SMTP)** — `alert_smtp_smarthost`, `alert_smtp_from`, `alert_smtp_username`, `alert_smtp_password`, `alert_smtp_to`
* **Telegram** — `alert_telegram_bot_token`, `alert_telegram_chat_id`

Any channel you leave blank stays inactive. Log-based alerts (Loki LogQL rules) route to the same Alertmanager. The manual procedure below is the **legacy / self-managed** path — use it only if you are not deploying with the automation.
{% endhint %}

## **Introduction**

Alerting is essential for monitoring modern systems, ensuring issues are detected and communicated in real-time. In Kubernetes, tools like Prometheus and Alertmanager handle metrics monitoring and alert routing to channels like Slack or email. Effective alerting minimizes downtime, improves response times, and maintains system performance.

## Prerequisites

* A Kubernetes cluster with Prometheus and Alertmanager installed. It comes by default when installing cattle-monitoring-system.
* A Slack workspace with a channel for receiving alerts.
* Permissions to create and manage Slack webhooks.

## Procedure

1. **Create a Slack Incoming Webhook.**
   1. Go to the Slack [API Webhooks page](https://api.slack.com/messaging/webhooks).
   2. Create a new Slack App (if you don’t have one already).
   3. Add an **Incoming Webhook** to the app.
   4. Select the Slack channel where you want alerts to appear.
   5. Copy the generated Webhook URL (e.g., `https://hooks.slack.com/services/...`).
2. **Deploy alerting configuration to kuberenets cluster.**
   1. Clone the [repository](https://github.com/OpenG2P/openg2p-deployment/tree/main/alerting) to you local and update the required changes mentioned below.
   2. Update **slack\_api\_url** and **channel** in **alertmanager.yaml**
   3. Modify the `routes` and `receivers` in the **alertmanager.yaml** configuration to customize which alerts are sent to Slack., as shown below.

      ```yaml
      routes:
      - match:
          alertname: Watchdog
        receiver: 'null'
      ```
   4. The monitoring package provided by Rancher includes various default alerting rules, which are often sufficient for most use cases. Sample custom alerts are available under the `custom-alerts` directory. Modify these as needed.
   5. Add cluster name in, `patch-cluster-name.yaml`
   6. Run **install.sh** to apply the configuration for alertmanager.
   7. And restart the alert-manager service on k8s-cluster.
   8. Verify whether the alerts are firing from the Prometheus UI and check if Slack notifications are being received.<br>

      <figure><img src="https://3034178245-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJZcdob2emEcLMvLyIxqT%2Fuploads%2FrYYNBjGqwsuK7StM92Bn%2Fimage.png?alt=media&amp;token=e1621d13-482b-4051-9c7a-27e6b7a1c404" alt=""><figcaption><p><em><strong>Prometheus dashboard</strong></em><br></p></figcaption></figure>

      <figure><img src="https://3034178245-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJZcdob2emEcLMvLyIxqT%2Fuploads%2FNnybzWOMIfCoJutDUWUD%2Fimage.png?alt=media&amp;token=cd6277c2-1c25-43bc-8c30-d25465081d14" alt=""><figcaption><p><em><strong>slack channel</strong></em></p></figcaption></figure>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openg2p.org/operations/deployment/deployment-guide/set-up-slack-alerts-for-a-kubernetes-cluster.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.