Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions documentation/docs/alert/alert_status.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# View alert status by node or service
When alerts fire, you need to know which nodes or services are affected and what state each alert is in. Use the **Alert Status** page to get that picture at a glance, triage by state, and silence alerts without switching pages.


## Check alerts for a specific node

To see all alerts affecting a single node:
{.power-number}

1. Go to **Alerts > Status**.
2. Toggle **Group by node** in the toolbar.
3. Find your node and expand it to see its individual alerts and their states.

## Filter alerts by state

Use the **State** dropdown in the toolbar to show only alerts in a specific state: **Normal**, **Pending**, **Firing**, **Recovering**, **No Data**, **Error** or **Silenced**.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## Check alerts for a specific service

To filter the alert list by service:
{.power-number}

1. Go to **Alerts > Status**.
2. Click **Show/Hide filters** in the toolbar to reveal the column filters.
3. In the **Service** column filter, enter the service name.

The table updates to show only alerts associated with that service.

## Get details on an alert

Click any alert row to open the details pane. From here you can:

- **See what triggered the alert**: the **Details** tab shows the summary, description, state and duration, node, service, severity, triggered at timestamp, and the MetricsQL expression. Check **Rule configuration** to see evaluation settings, template name, folder, and rule health.
- **Debug custom templates or verify label values**: switch to **Raw data** to inspect the full label set and JSON payload.
- **Move between alerts**: use the arrow buttons in the pane header to go to the next or previous alert without closing the pane.

## Silence an alert

You need **Editor** role or higher to silence alerts.
{.power-number}

1. Click the actions menu on the alert row.
2. Click **Silence**. PMM opens the **Silences** page with the alert labels pre-filled.
3. Set the duration and confirm.

Silenced alerts stay visible in the table with a **Silenced** badge. To unsilence, open the actions menu and click **Unsilence**.

## Make custom alerts appear in this view

Built-in PMM alert templates automatically include the labels that this page uses to group alerts by node and service.

The following templates show data even when no alert is firing, so you can see how close a node or service is to triggering an alert before it does:

- **Node high CPU load**
- **MongoDB DBPath disk space utilization**
- **PostgreSQL high rate of transaction rollbacks**
- **PostgreSQL connections in use**

Support for more templates to show data when no alert is firing will be added in future releases.

If your custom templates do not show up correctly, add these labels to the template definition:

- `node_name`: identifies the monitored node.
- `service_name`: identifies the monitored service.

For instructions, see [Alert rules and alert templates](../alert/alert_rules.md).
2 changes: 1 addition & 1 deletion documentation/docs/alert/silence_alerts.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ You can also schedule a silence for a future date and time. This is referred to

During a silence, PMM continues to track metrics but does not trigger alerts or send notifications to any specified contact points. Once the silence expires alerts and notifications will resume.

Silenced alerts are still recorded under **Alerting > Fired Alerts** so that you can review them later. Silenced alerts show up as **Suppressed** and are disabled for as long as it's specified in the **Silence Duration**, or until you remove a silence.
Silenced alerts are still recorded under **Alerting > Fired Alerts** so that you can review them later. Silenced alerts show up as **Silenced** and are disabled for as long as it's specified in the **Silence Duration**, or until you remove a silence.

## Using silences

Expand Down
8 changes: 4 additions & 4 deletions documentation/docs/alert/templates_list.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The table below lists all the alert templates available in Percona Monitoring an

| Area | Template name | Description | Database technology |
| :----|:------------- | :---------- | :------------------ |
| OS | **Node high CPU load** | Monitors node CPU usage and alerts when it surpasses 80% (default threshold). Provides details about specific nodes experiencing high CPU load, indicating potential performance issues or scaling needs. | MySQL, MongoDB, PostgreSQL |
| OS | **Node high CPU load** | Monitors node CPU usage and alerts when it surpasses 80% (default threshold). Provides details about specific nodes experiencing high CPU load, indicating potential performance issues or scaling needs. Visible in the **Alert Status** page even when CPU usage is within the threshold, so you can monitor load before it becomes a problem. | MySQL, MongoDB, PostgreSQL |
| OS | **Memory available less than a threshold** | Tracks available memory on nodes and alerts when free memory drops below 20% (default threshold). Helps prevent system instability due to memory constraints. | MySQL, MongoDB, PostgreSQL |
| OS | **Node high swap filling up** | Monitors node swap usage and alerts when it exceeds 80% (default threshold). Indicates potential memory pressure and performance degradation, allowing for timely intervention. | MySQL, MongoDB, PostgreSQL |

Expand All @@ -36,7 +36,7 @@ The table below lists all the alert templates available in Percona Monitoring an
| :----|:------------- | :---------- | :------------------ |
| MongoDB | **MongoDB down** | Detects when a MongoDB instance becomes unavailable, enabling rapid response to maintain database accessibility. When monitoring MongoDB sharded clusters, alerts using this template detect outages in any cluster component (configuration servers, Mongos routers, data-bearing nodes, and arbiters). | MongoDB |
| MongoDB | **MongoDB restarted** | Detects recent MongoDB restarts, alerting if an instance has been restarted within the last 5 minutes (default threshold). Facilitates investigation of unexpected downtime and potential issues. | MongoDB |
| MongoDB | **MongoDB DBPath disk space utilization** | Monitors disk space usage in MongoDB's data directory and alerts when it exceeds set thresholds. Helps prevent storage-related issues and ensures adequate space for database operations. | MongoDB |
| MongoDB | **MongoDB DBPath disk space utilization** | Monitors disk space usage in MongoDB's data directory and alerts when it exceeds set thresholds. Helps prevent storage-related issues and ensures adequate space for database operations. Visible in the Alert Status page even when disk usage is within the threshold, so you can monitor usage before it becomes a problem. | MongoDB |
| MongoDB | **MongoDB host SSL certificate expiry** | Tracks SSL certificate expiration dates for MongoDB hosts and alerts when certificates are approaching expiry. Enables timely certificate renewal to maintain secure connections. | MongoDB |
| MongoDB | **MongoDB oplog window** | Monitors the oplog window size and alerts when it falls below the recommended threshold (typically 24-48 hours). Ensures sufficient time for secondary nodes to replicate data and maintain cluster consistency. | MongoDB |
| MongoDB | **MongoDB read tickets** | Tracks read ticket availability in the WiredTiger storage engine and alerts when it falls below set thresholds. Helps optimize read performance and identify potential bottlenecks. | MongoDB |
Expand Down Expand Up @@ -75,12 +75,12 @@ The table below lists all the alert templates available in Percona Monitoring an
| :----|:------------- | :---------- | :------------------ |
| PostgreSQL | **PostgreSQL down** | Detects when PostgreSQL instances become unavailable, enabling quick response to maintain database services. Provides details about affected services and nodes. | PostgreSQL |
| PostgreSQL | **PostgreSQL restarted** | Identifies recent PostgreSQL restarts, alerting if an instance has been restarted within the last 5 minutes (default threshold). Aids in investigating unexpected downtime and potential issues. | PostgreSQL |
| PostgreSQL | **PostgreSQL connections in use** | Monitors PostgreSQL connection usage and alerts when the percentage of active connections exceeds 80% of the maximum allowed (default threshold). Helps prevent performance degradation due to excessive connections. | PostgreSQL |
| PostgreSQL | **PostgreSQL connections in use** | Monitors PostgreSQL connection usage and alerts when the percentage of active connections exceeds 80% of the maximum allowed (default threshold). Helps prevent performance degradation due to excessive connections. Visible in the Alert Status page even when connection usage is within the threshold, so you can monitor connections before it becomes a problem. | PostgreSQL |
| PostgreSQL | **PostgreSQL index bloat is high** | Detects excessive index bloat and alerts users. Helps identify performance degradation due to bloated indexes, enabling timely maintenance to improve query performance. | PostgreSQL |
| PostgreSQL | **PostgreSQL high number of dead tuples** | Monitors the accumulation of dead tuples in relations and alerts when they exceed set thresholds. Indicates potential issues with vacuum settings and helps optimize storage and query performance. | PostgreSQL |
| PostgreSQL | **PostgreSQL has a high number of statement timeouts** | Tracks and alerts on frequent query cancellations due to statement timeouts. Helps identify various issues such as high load, poorly written queries, or inadequate resource allocation. | PostgreSQL |
| PostgreSQL | **PostgreSQL table bloat is high** | Detects excessive table bloat and alerts users. Indicates a need to adjust vacuum settings for specific relations or globally, helping to maintain optimal query performance and storage efficiency. | PostgreSQL |
| PostgreSQL | **PostgreSQL high rate of transaction rollbacks** | Monitors the ratio of transaction rollbacks to commits and alerts on high rates. Helps identify potential application or database issues leading to frequent transaction failures. | PostgreSQL |
| PostgreSQL | **PostgreSQL high rate of transaction rollbacks** | Monitors the ratio of transaction rollbacks to commits and alerts on high rates. Helps identify potential application or database issues leading to frequent transaction failures. Visible in the Alert Status page even when the rollback rate is within the threshold, so you can monitor transaction health before it becomes a problem. | PostgreSQL |
| PostgreSQL | **PostgreSQL tables not auto analyzed** | Identifies tables that are not being auto-analyzed and alerts users. Crucial for maintaining accurate statistics and generating proper query execution plans. | PostgreSQL |
| PostgreSQL | **PostgreSQL tables not auto vacuumed** | Detects tables that are not being auto-vacuumed and alerts users. Essential for managing bloat, optimizing storage, and maintaining overall database health. | PostgreSQL |
| PostgreSQL | **PostgreSQL unused replication slot** | Identifies and alerts on unused replication slots. Helps prevent excessive WAL retention and potential disk space issues, especially when replicas are offline. | PostgreSQL |
Expand Down
15 changes: 12 additions & 3 deletions documentation/docs/release-notes/3.9.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,20 @@ PMM 3.9.0 introduces the **Alert Status** page for viewing all active alerts by

This release also removes the provisioned PostgreSQL datasource as a security measure, adds Ubuntu 26.04 LTS support for PMM Client, and includes a large number of dashboard, UI, and stability fixes.

### Alert Status page: view alerts by node or service
## Alert Status page: view alerts by node or service

You can now check the full alert picture for any node or service in a single view. Go to **Alerts > Status** and filter by service or group by node to see every associated alert rule and its current state, so you don't have to piece together the situation from alerts grouped by alert name.

All built-in PMM alert templates work with this view automatically. If you use custom templates, you can add `node_name` and `service_name` labels to make your alerts appear here too. To learn how, see [View alert status by node or service](../alert/alert_status.md).
All built-in PMM alert templates appear in this view when alerts are firing. Four templates also show data even when no alert is firing, so you can see how close a node or service is to triggering an alert before it does.

- **Node high CPU load**
- **MongoDB DBPath disk space utilization**
- **PostgreSQL high rate of transaction rollbacks**
- **PostgreSQL connections in use**

Support for continuous reporting will be extended to more templates in future releases.

If you use custom templates, you can add `node_name` and `service_name` labels to make your alerts appear here too. To learn how, see [View alert status by node or service](../alert/alert_status.md).

![Alert status page](../images/Alert_status.png)

Expand Down Expand Up @@ -259,7 +268,7 @@ To lower your exposure in the meantime:
- [PMM-15114](https://perconadev.atlassian.net/browse/PMM-15114): Fixed the **Scanned and Moved Objects** panel in the **MongoDB Instance Summary** dashboard showing blank labels in the legend instead of the operation type (`scanned`, `scanned_objects`).


## Known issues
## 🔶 Known issues

### UI upgrade progress bar stops when upgrading to PMM 3.9.0 ([PMM-15266](https://perconadev.atlassian.net/browse/PMM-15266))

Expand Down
2 changes: 2 additions & 0 deletions documentation/mkdocs-base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -395,10 +395,12 @@ nav:
- Percona Alerting:
- alert/index.md
- alert/alert_rules.md
- alert/alert_status.md
- alert/contact_points.md
- alert/templates_list.md
- alert/silence_alerts.md
- alert/disable_alerts.md

- Back up and restore:
- backup/index.md
- backup/prepare_storage_location.md
Expand Down