seon
7ce2dc68fe
WP-12(실험 설계서 6장) vm/ai-env: apply.sh(시크릿 4종을 Secret Manager에서 파일로, 미등록이면
성능 저하 모드로 기동), compose(agent 8080·hitl 8081·rag-db pgvector 0.8.7-pg16), ai.env(Discord
서버 Selfheal Lab 1557424985793101824, approvals 1557425140302872698, escalation 1557425282363691048,
OPENAI_MODEL gpt-6.1-sol 후보), 프롬프트 3종, 러너북 11종(조치별·토폴로지·3층 절차).
WP-15(실험 설계서 4.1·4.3): apps/alloy.yaml(Alloy Helm 1.13.0 DaemonSet, 클러스터링으로 KSM·Argo CD·
cadvisor 스크랩 분산, demo 등 파드 로그 → Loki), apps/kube-state-metrics.yaml(Helm 8.6.0).
legacy-api compose에 Alloy 사이드카(up{job="legacy-api"}, 컨테이너 로그). DB VM은
vm/postgres-common/alloy.sh로 apt 설치 + prometheus.exporter.postgres(pg_replication_lag_seconds)
+ custom query pg_is_wal_replay_paused + PostgreSQL 로그.
vm/harness/apply.sh: 준비 대기 루프의 curl에 || true(러너 재기동 순간 curl 56으로 중단되던 문제,
2026-10-08 harness-sync 실측).
5 lines
1.2 KiB
Markdown
5 lines
1.2 KiB
Markdown
Availability: avg_over_time(probe_success[5m]) >= 99.5% over the campaign. Latency: api p95 (histogram_quantile(0.95, sum by (le) (rate(demo_http_request_duration_seconds_bucket[5m])))) < 0.5 s. Error ratio: 5xx / all over 5 minutes < 1%.
|
|
Recovery (t_recover) is declared by the harness when the probe succeeds 12 times in a row (3 minutes) and latency and error ratio are back inside the SLO over a 5-minute window.
|
|
Layers: 1 = platform-internal automatic recovery (kubelet restart, ReplicaSet, Argo CD selfHeal, MIG autohealing, compose restart). 2 = your proposal + human approval in Discord, executed by the harness. 3 = human-only (irreversible actions, no approval within 15 minutes, confidence below 0.5).
|
|
Alert rules and what they mean: ProbeDown (probe_success == 0 for 1 min, symptom only), ApiLatencyHigh (p95 > 500 ms for 3 min), ApiErrorRateHigh (5xx > 1% for 3 min), PodCrashLooping (3+ restarts in 5 min; layer 1 first, layer 2 after 5 min), LegacyUnhealthy (fewer than 2 legacy-api instances up; autohealing first, layer 2 after 10 min), DbReplicationLag (lag > 60 s or replay paused), DbPrimaryDown (TCP 5432 failing for 1 min), GitOpsDrift (OutOfSync 5 min; selfHeal handles it).
|