fix(alerting): evaluate the replay-paused branch of DbReplicationLag, add ApiMemoryPressure; retarget grey-zone scenarios

DbReplicationLag の式は PromQL の or が左辺のサンプル(常に 0)を残すため
pg_is_wal_replay_paused の分岐が評価されず、第 1・2 回とも発火しなかった。2 つの
bool を加算して clamp する式に直す。api コンテナのメモリが limit の 80% を超えて
2 分続いたときの ApiMemoryPressure(第 2 層)を追加する。C-G2 の StressChaos は
size "80%"(ノード基準)で即 OOM になっていたため 200MB に、V-G1 の対象は
トラフィックを受けているインスタンス legacy:busy に変更する。slo.md に新ルールを追記。
This commit is contained in:
seon committed 2026-10-08 19:20:50 +09:00
1 parent b2e10cf300
commit 1e0e17b33c
4 files changed
+50 -19

No files matched your search

+5 -2
View File
@@ -5,7 +5,7 @@ asset: container
expected_layer: 2
ground_truth_label: memory_pressure_api
ground_truth_action: k8s.rollout_restart
description: StressChaos memory workers 2 size 80%, app=api, duration 10m
description: StressChaos memory workers 2 size 200MB (api limit 256Mi), app=api, duration 10m
notes:
open_item: "design 11.3: whether the stress re-attaches to the restarted pods for the remaining duration"
inject:
@@ -24,7 +24,10 @@ inject:
stressors:
memory:
workers: 2
size: "80%"
# 2026-10-08: "80%" is a share of the node memory (memStress reached 4.4 GB), so the cgroup OOM killer took memStress
# and the api process within a second: one restart per pod, no sustained pressure, no alert. 200MB stays under the
# 256Mi limit and keeps the working set near 90% for the whole duration (ApiMemoryPressure fires after 2m).
size: "200MB"
duration: "10m"
revert:
method: chaos_delete