fix(alerting): evaluate the replay-paused branch of DbReplicationLag, add ApiMemoryPressure; retarget grey-zone scenarios
DbReplicationLag の式は PromQL の or が左辺のサンプル(常に 0)を残すため pg_is_wal_replay_paused の分岐が評価されず、第 1・2 回とも発火しなかった。2 つの bool を加算して clamp する式に直す。api コンテナのメモリが limit の 80% を超えて 2 分続いたときの ApiMemoryPressure(第 2 層)を追加する。C-G2 の StressChaos は size "80%"(ノード基準)で即 OOM になっていたため 200MB に、V-G1 の対象は トラフィックを受けているインスタンス legacy:busy に変更する。slo.md に新ルールを追記。
This commit is contained in:
1 parent
b2e10cf300
commit
1e0e17b33c
4 files changed
+50
-19
No files matched your search
@@ -57,8 +57,7 @@ groups:
|
||||
model:
|
||||
refId: A
|
||||
instant: true
|
||||
expr: histogram_quantile(0.95, sum by (le) (rate(demo_http_request_duration_seconds_bucket{route=~"orders|price"}[5m])))
|
||||
> bool 0.5
|
||||
expr: histogram_quantile(0.95, sum by (le) (rate(demo_http_request_duration_seconds_bucket{route=~"orders|price"}[5m]))) > bool 0.5
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
@@ -89,8 +88,7 @@ groups:
|
||||
model:
|
||||
refId: A
|
||||
instant: true
|
||||
expr: (sum(rate(demo_http_requests_total{status=~"5.."}[5m])) / sum(rate(demo_http_requests_total[5m])))
|
||||
> bool 0.01
|
||||
expr: (sum(rate(demo_http_requests_total{status=~"5.."}[5m])) / sum(rate(demo_http_requests_total[5m]))) > bool 0.01
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
@@ -112,8 +110,7 @@ groups:
|
||||
layer: '1'
|
||||
signature: pod_crashloop
|
||||
annotations:
|
||||
summary: container restarted 3+ times in 5m (kubelet is handling; escalate after
|
||||
5m)
|
||||
summary: container restarted 3+ times in 5m (kubelet is handling; escalate after 5m)
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
@@ -123,8 +120,7 @@ groups:
|
||||
model:
|
||||
refId: A
|
||||
instant: true
|
||||
expr: max by (pod) (increase(kube_pod_container_status_restarts_total{namespace="demo"}[5m]))
|
||||
>= bool 3
|
||||
expr: max by (pod) (increase(kube_pod_container_status_restarts_total{namespace="demo"}[5m])) >= bool 3
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
@@ -146,8 +142,7 @@ groups:
|
||||
layer: '1'
|
||||
signature: legacy_unhealthy
|
||||
annotations:
|
||||
summary: fewer than 2 healthy legacy-api instances (MIG autohealing is handling;
|
||||
escalate after 10m)
|
||||
summary: fewer than 2 healthy legacy-api instances (MIG autohealing is handling; escalate after 10m)
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
@@ -158,7 +153,6 @@ groups:
|
||||
refId: A
|
||||
instant: true
|
||||
expr: count(up{job="legacy-api"} == 1) < bool 2
|
||||
# no series until WP-15 ships Alloy on the legacy VMs: absent -> NoData -> OK (the `or vector(0)` form fired continuously)
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
@@ -189,8 +183,7 @@ groups:
|
||||
model:
|
||||
refId: A
|
||||
instant: true
|
||||
expr: (max(pg_replication_lag_seconds) > bool 60) or (max(pg_is_wal_replay_paused)
|
||||
== bool 1)
|
||||
expr: clamp_max((max(pg_replication_lag_seconds) > bool 60) + (max(pg_is_wal_replay_paused) == bool 1), 1)
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
@@ -265,3 +258,35 @@ groups:
|
||||
type: gt
|
||||
params:
|
||||
- 0
|
||||
- uid: api-memory-pressure
|
||||
title: ApiMemoryPressure
|
||||
condition: C
|
||||
for: 2m
|
||||
noDataState: OK
|
||||
execErrState: Error
|
||||
labels:
|
||||
layer: '2'
|
||||
annotations:
|
||||
summary: an api container holds more than 80% of its memory limit for 2 minutes (grey zone; rollout restart after approval)
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: mimir
|
||||
model:
|
||||
refId: A
|
||||
instant: true
|
||||
expr: max by (pod) (container_memory_working_set_bytes{namespace="demo",container="api"}) / on (pod) max by (pod) (kube_pod_container_resource_limits{namespace="demo",container="api",resource="memory"})
|
||||
> bool 0.8
|
||||
- refId: C
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: A
|
||||
conditions:
|
||||
- evaluator:
|
||||
type: gt
|
||||
params:
|
||||
- 0
|
||||
Reference in new issue
Block a user