fix(alerting): evaluate the replay-paused branch of DbReplicationLag, add ApiMemoryPressure; retarget grey-zone scenarios

DbReplicationLag の式は PromQL の or が左辺のサンプル(常に 0)を残すため
pg_is_wal_replay_paused の分岐が評価されず、第 1・2 回とも発火しなかった。2 つの
bool を加算して clamp する式に直す。api コンテナのメモリが limit の 80% を超えて
2 分続いたときの ApiMemoryPressure(第 2 層)を追加する。C-G2 の StressChaos は
size "80%"(ノード基準)で即 OOM になっていたため 200MB に、V-G1 の対象は
トラフィックを受けているインスタンス legacy:busy に変更する。slo.md に新ルールを追記。
This commit is contained in:
seon committed 2026-10-08 19:20:50 +09:00
1 parent b2e10cf300
commit 1e0e17b33c
4 files changed
+50 -19

No files matched your search

@@ -57,8 +57,7 @@ groups:
model:
refId: A
instant: true
expr: histogram_quantile(0.95, sum by (le) (rate(demo_http_request_duration_seconds_bucket{route=~"orders|price"}[5m])))
> bool 0.5
expr: histogram_quantile(0.95, sum by (le) (rate(demo_http_request_duration_seconds_bucket{route=~"orders|price"}[5m]))) > bool 0.5
- refId: C
datasourceUid: __expr__
model:
@@ -89,8 +88,7 @@ groups:
model:
refId: A
instant: true
expr: (sum(rate(demo_http_requests_total{status=~"5.."}[5m])) / sum(rate(demo_http_requests_total[5m])))
> bool 0.01
expr: (sum(rate(demo_http_requests_total{status=~"5.."}[5m])) / sum(rate(demo_http_requests_total[5m]))) > bool 0.01
- refId: C
datasourceUid: __expr__
model:
@@ -112,8 +110,7 @@ groups:
layer: '1'
signature: pod_crashloop
annotations:
summary: container restarted 3+ times in 5m (kubelet is handling; escalate after
5m)
summary: container restarted 3+ times in 5m (kubelet is handling; escalate after 5m)
data:
- refId: A
relativeTimeRange:
@@ -123,8 +120,7 @@ groups:
model:
refId: A
instant: true
expr: max by (pod) (increase(kube_pod_container_status_restarts_total{namespace="demo"}[5m]))
>= bool 3
expr: max by (pod) (increase(kube_pod_container_status_restarts_total{namespace="demo"}[5m])) >= bool 3
- refId: C
datasourceUid: __expr__
model:
@@ -146,8 +142,7 @@ groups:
layer: '1'
signature: legacy_unhealthy
annotations:
summary: fewer than 2 healthy legacy-api instances (MIG autohealing is handling;
escalate after 10m)
summary: fewer than 2 healthy legacy-api instances (MIG autohealing is handling; escalate after 10m)
data:
- refId: A
relativeTimeRange:
@@ -158,7 +153,6 @@ groups:
refId: A
instant: true
expr: count(up{job="legacy-api"} == 1) < bool 2
# no series until WP-15 ships Alloy on the legacy VMs: absent -> NoData -> OK (the `or vector(0)` form fired continuously)
- refId: C
datasourceUid: __expr__
model:
@@ -189,8 +183,7 @@ groups:
model:
refId: A
instant: true
expr: (max(pg_replication_lag_seconds) > bool 60) or (max(pg_is_wal_replay_paused)
== bool 1)
expr: clamp_max((max(pg_replication_lag_seconds) > bool 60) + (max(pg_is_wal_replay_paused) == bool 1), 1)
- refId: C
datasourceUid: __expr__
model:
@@ -265,3 +258,35 @@ groups:
type: gt
params:
- 0
- uid: api-memory-pressure
title: ApiMemoryPressure
condition: C
for: 2m
noDataState: OK
execErrState: Error
labels:
layer: '2'
annotations:
summary: an api container holds more than 80% of its memory limit for 2 minutes (grey zone; rollout restart after approval)
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: mimir
model:
refId: A
instant: true
expr: max by (pod) (container_memory_working_set_bytes{namespace="demo",container="api"}) / on (pod) max by (pod) (kube_pod_container_resource_limits{namespace="demo",container="api",resource="memory"})
> bool 0.8
- refId: C
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: A
conditions:
- evaluator:
type: gt
params:
- 0