seon
1e0e17b33c
DbReplicationLag の式は PromQL の or が左辺のサンプル（常に 0）を残すため pg_is_wal_replay_paused の分岐が評価されず、第 1・2 回とも発火しなかった。2 つの bool を加算して clamp する式に直す。api コンテナのメモリが limit の 80% を超えて 2 分続いたときの ApiMemoryPressure（第 2 層）を追加する。C-G2 の StressChaos は size "80%"（ノード基準）で即 OOM になっていたため 200MB に、V-G1 の対象は トラフィックを受けているインスタンス legacy:busy に変更する。slo.md に新ルールを追記。
34 lines
1.2 KiB
YAML
34 lines
1.2 KiB
YAML
# design 7 C-G2: grey zone, container. StressChaos memory on api pods; latency rises, intermittent 5xx. Correct: k8s.rollout_restart after approval.
|
|
id: C-G2
|
|
path: gray
|
|
asset: container
|
|
expected_layer: 2
|
|
ground_truth_label: memory_pressure_api
|
|
ground_truth_action: k8s.rollout_restart
|
|
description: StressChaos memory workers 2 size 200MB (api limit 256Mi), app=api, duration 10m
|
|
notes:
|
|
open_item: "design 11.3: whether the stress re-attaches to the restarted pods for the remaining duration"
|
|
inject:
|
|
method: chaos
|
|
manifest: |
|
|
apiVersion: chaos-mesh.org/v1alpha1
|
|
kind: StressChaos
|
|
metadata:
|
|
namespace: chaos-mesh
|
|
spec:
|
|
mode: all
|
|
selector:
|
|
namespaces: [demo]
|
|
labelSelectors:
|
|
app: api
|
|
stressors:
|
|
memory:
|
|
workers: 2
|
|
# 2026-10-08: "80%" is a share of the node memory (memStress reached 4.4 GB), so the cgroup OOM killer took memStress
|
|
# and the api process within a second: one restart per pod, no sustained pressure, no alert. 200MB stays under the
|
|
# 256Mi limit and keeps the working set near 90% for the whole duration (ApiMemoryPressure fires after 2m).
|
|
size: "200MB"
|
|
duration: "10m"
|
|
revert:
|
|
method: chaos_delete
|