fix(alerting): evaluate the replay-paused branch of DbReplicationLag, add ApiMemoryPressure; retarget grey-zone scenarios

DbReplicationLag の式は PromQL の or が左辺のサンプル(常に 0)を残すため
pg_is_wal_replay_paused の分岐が評価されず、第 1・2 回とも発火しなかった。2 つの
bool を加算して clamp する式に直す。api コンテナのメモリが limit の 80% を超えて
2 分続いたときの ApiMemoryPressure(第 2 層)を追加する。C-G2 の StressChaos は
size "80%"(ノード基準)で即 OOM になっていたため 200MB に、V-G1 の対象は
トラフィックを受けているインスタンス legacy:busy に変更する。slo.md に新ルールを追記。
This commit is contained in:
seon committed 2026-10-08 19:20:50 +09:00
1 parent b2e10cf300
commit 1e0e17b33c
4 files changed
+50 -19

No files matched your search

+5 -2
View File
@@ -5,7 +5,7 @@ asset: container
expected_layer: 2
ground_truth_label: memory_pressure_api
ground_truth_action: k8s.rollout_restart
description: StressChaos memory workers 2 size 80%, app=api, duration 10m
description: StressChaos memory workers 2 size 200MB (api limit 256Mi), app=api, duration 10m
notes:
open_item: "design 11.3: whether the stress re-attaches to the restarted pods for the remaining duration"
inject:
@@ -24,7 +24,10 @@ inject:
stressors:
memory:
workers: 2
size: "80%"
# 2026-10-08: "80%" is a share of the node memory (memStress reached 4.4 GB), so the cgroup OOM killer took memStress
# and the api process within a second: one restart per pod, no sustained pressure, no alert. 200MB stays under the
# 256Mi limit and keeps the working set near 90% for the whole duration (ApiMemoryPressure fires after 2m).
size: "200MB"
duration: "10m"
revert:
method: chaos_delete
+6 -3
View File
@@ -5,12 +5,15 @@ asset: vm
expected_layer: 2
ground_truth_label: network_latency_legacy
ground_truth_action: mig.recreate_instance
description: tc qdisc add dev ens4 root netem delay 400ms on legacy instance 0
description: tc qdisc add dev ens4 root netem delay 400ms on the legacy instance that currently serves the api traffic (legacy:busy)
# 2026-10-08: the ILB hashes the api pods' few keep-alive connections onto one instance; in campaign 2 legacy:0 served 0 rps,
# so the delay was never on the request path. legacy:busy = the instance with the most legacy-api requests over the last
# 2 minutes (Mimir); the harness caches the choice per scenario so the revert hits the same instance.
inject:
method: ssh
host: legacy:0
host: legacy:busy
command: sudo tc qdisc add dev ens4 root netem delay 400ms
revert:
method: ssh
host: legacy:0
host: legacy:busy
command: sudo tc qdisc del dev ens4 root || true