feat(ai-env,alerting): reference queries per alert in slo.md, rule expression as alert annotation

第 4 回 V-G1 で LLM が route フィルタのない 5 分窓の p95(0.017 秒)を根拠に
「遅延なし」と結論した。slo.md にアラートごとの基準クエリ(規則と同じ式、
legacy のインスタンス別遅延と要求率を含む)を書き、確かめ直すときは同じ式を
使わせる。各規則の annotations.condition に式をそのまま入れ、AI 側が
「アラート条件が解消したか」を計算できるようにする(Jev の recovering 再定義)。
This commit is contained in:
seon committed 2026-10-09 08:33:22 +09:00
1 parent bfd5a710f7
commit fd08c32501
2 files changed
+22 -1

No files matched your search

@@ -17,6 +17,7 @@ groups:
layer: '2'
annotations:
summary: ALB synthetic probe failing (symptom alert, cause unknown)
condition: max(probe_success{job="blackbox",target="alb"}) == bool 0
data:
- refId: A
relativeTimeRange:
@@ -48,6 +49,7 @@ groups:
layer: '2'
annotations:
summary: api p95 over the last minute above 500 ms for 2 minutes
condition: histogram_quantile(0.95, sum by (le) (rate(demo_http_request_duration_seconds_bucket{route=~"orders|price"}[1m]))) > bool 0.5
data:
- refId: A
relativeTimeRange:
@@ -79,6 +81,7 @@ groups:
layer: '2'
annotations:
summary: demo api 5xx ratio above 1% (grey zone)
condition: (sum(rate(demo_http_requests_total{status=~"5.."}[5m])) / sum(rate(demo_http_requests_total[5m]))) > bool 0.01
data:
- refId: A
relativeTimeRange:
@@ -111,6 +114,7 @@ groups:
signature: pod_crashloop
annotations:
summary: container restarted 3+ times in 5m (kubelet is handling; escalate after 5m)
condition: max by (pod) (increase(kube_pod_container_status_restarts_total{namespace="demo"}[5m])) >= bool 3
data:
- refId: A
relativeTimeRange:
@@ -143,6 +147,7 @@ groups:
signature: legacy_unhealthy
annotations:
summary: fewer than 2 healthy legacy-api instances (MIG autohealing is handling; escalate after 10m)
condition: count(up{job="legacy-api"} == 1) < bool 2
data:
- refId: A
relativeTimeRange:
@@ -174,6 +179,7 @@ groups:
layer: '2'
annotations:
summary: replica lag above 60s or WAL replay paused (grey zone, small SLO impact)
condition: clamp_max((max(pg_replication_lag_seconds) > bool 60) + (max(pg_is_wal_replay_paused) == bool 1), 1)
data:
- refId: A
relativeTimeRange:
@@ -205,6 +211,7 @@ groups:
layer: '2'
annotations:
summary: db-primary 5432 unreachable (layer 2 diagnosis, layer 3 expected)
condition: max(probe_success{job="blackbox",target="db-primary"}) == bool 0
data:
- refId: A
relativeTimeRange:
@@ -237,6 +244,7 @@ groups:
signature: gitops_drift
annotations:
summary: Argo CD application OutOfSync for 5m (selfHeal is handling; recorded)
condition: max(argocd_app_info{sync_status!="Synced"}) == bool 1
data:
- refId: A
relativeTimeRange:
@@ -268,6 +276,8 @@ groups:
layer: '2'
annotations:
summary: an api container holds more than 80% of its memory limit for 1 minute (grey zone; rollout restart after approval)
condition: max by (pod) (container_memory_working_set_bytes{namespace="demo",container="api"}) / on (pod) max by (pod) (kube_pod_container_resource_limits{namespace="demo",container="api",resource="memory"})
> bool 0.8
data:
- refId: A
relativeTimeRange: