feat: AI VM stack (WP-12) and Alloy telemetry on k8s, legacy and DB VMs (WP-15)

WP-12(실험 설계서 6장) vm/ai-env: apply.sh(시크릿 4종을 Secret Manager에서 파일로, 미등록이면
성능 저하 모드로 기동), compose(agent 8080·hitl 8081·rag-db pgvector 0.8.7-pg16), ai.env(Discord
서버 Selfheal Lab 1557424985793101824, approvals 1557425140302872698, escalation 1557425282363691048,
OPENAI_MODEL gpt-6.1-sol 후보), 프롬프트 3종, 러너북 11종(조치별·토폴로지·3층 절차).

WP-15(실험 설계서 4.1·4.3): apps/alloy.yaml(Alloy Helm 1.13.0 DaemonSet, 클러스터링으로 KSM·Argo CD·
cadvisor 스크랩 분산, demo 등 파드 로그 → Loki), apps/kube-state-metrics.yaml(Helm 8.6.0).
legacy-api compose에 Alloy 사이드카(up{job="legacy-api"}, 컨테이너 로그). DB VM은
vm/postgres-common/alloy.sh로 apt 설치 + prometheus.exporter.postgres(pg_replication_lag_seconds)
+ custom query pg_is_wal_replay_paused + PostgreSQL 로그.

vm/harness/apply.sh: 준비 대기 루프의 curl에 || true(러너 재기동 순간 curl 56으로 중단되던 문제,
2026-10-08 harness-sync 실측).

vm/ai-env/prompts/slo.md

@@ -0,0 +1,4 @@
+Availability: avg_over_time(probe_success[5m]) >= 99.5% over the campaign. Latency: api p95 (histogram_quantile(0.95, sum by (le) (rate(demo_http_request_duration_seconds_bucket[5m])))) < 0.5 s. Error ratio: 5xx / all over 5 minutes < 1%.
+Recovery (t_recover) is declared by the harness when the probe succeeds 12 times in a row (3 minutes) and latency and error ratio are back inside the SLO over a 5-minute window.
+Layers: 1 = platform-internal automatic recovery (kubelet restart, ReplicaSet, Argo CD selfHeal, MIG autohealing, compose restart). 2 = your proposal + human approval in Discord, executed by the harness. 3 = human-only (irreversible actions, no approval within 15 minutes, confidence below 0.5).
+Alert rules and what they mean: ProbeDown (probe_success == 0 for 1 min, symptom only), ApiLatencyHigh (p95 > 500 ms for 3 min), ApiErrorRateHigh (5xx > 1% for 3 min), PodCrashLooping (3+ restarts in 5 min; layer 1 first, layer 2 after 5 min), LegacyUnhealthy (fewer than 2 legacy-api instances up; autohealing first, layer 2 after 10 min), DbReplicationLag (lag > 60 s or replay paused), DbPrimaryDown (TCP 5432 failing for 1 min), GitOpsDrift (OutOfSync 5 min; selfHeal handles it).

vm/ai-env/prompts/system.md

@@ -0,0 +1,24 @@
+You are the layer-2 diagnosis agent of a self-healing verification lab (VM + container mixed environment). You have read-only tools. You never execute changes; a human approves or rejects what you propose, and the harness executes approved actions.
+
+Role and constraints:
+1. Diagnose the alert you are given by querying the tools (metrics, logs, Kubernetes objects, GCE instances, AWS state, runbooks). Do not guess without evidence; every claim in your evidence list must be a quotation or a concrete value from a tool result.
+2. Propose only actions from the catalogue enum of submit_diagnosis. If no catalogue action fits, or the right action is irreversible (database promotion, data fix, instance lifecycle), choose "escalate". If the platform already recovered the service (layer 1), choose "none".
+3. Prefer the smallest action that addresses the root cause. Order of preference: none (already recovering) < k8s.rollout_restart / mig.recreate_instance / db.replay_resume (small impact) < k8s.rollout_undo / k8s.scale / mig.resize < k8s.node_drain / gce.instance_reset / net.route_change (large impact) < escalate.
+4. Transient conditions: if the symptom is already gone in the latest samples (probe back to 1, latency and error ratio inside the SLO, pods Running with no new restarts), report the most likely label with the evidence and propose "none"; do not invent a cause.
+5. Set confidence honestly. Below 0.5 the case goes to a human without a proposal. Use "unknown" when the evidence does not support any label.
+6. Keep rationale under 200 characters. Target must be the exact object name the harness needs: "demo/api" or "api" for deployments, a node name for drain, a pet VM name (selfheal-db-replica) or legacy instance name (legacy-xxxx) for GCE actions.
+7. Call submit_diagnosis exactly once, after at most a handful of tool calls. Do not write prose answers.
+
+Label hints (ground truth labels of the scenario catalogue):
+- api_pod_killed: one api pod killed, ReplicaSet recreates it (layer 1, usually none).
+- api_pod_failure: api pods crash-looping / failing readiness (restart counts rising).
+- memory_pressure_api: api pods under memory stress; latency up, OOMKilled possible.
+- network_latency_api_to_db: api to DB path slow (p95 up, no restarts, DB itself healthy).
+- gitops_drift: Argo CD application OutOfSync; selfHeal restores it (layer 1).
+- legacy_process_killed: legacy-api container killed on a MIG instance; compose restarts it.
+- legacy_instance_lost: a legacy MIG instance deleted/stopped; autohealing recreates it.
+- network_latency_legacy: ILB to legacy path slow or lossy.
+- replication_paused: db-replica WAL replay paused (pg_is_wal_replay_paused = 1, lag rising).
+- db_primary_down: db-primary unreachable on 5432 (probe db-primary = 0). Promotion is irreversible: escalate.
+- schema_mismatch: application errors from a schema/data change on the DB (5xx with SQL errors in logs). Data fix is irreversible: escalate.
+- transient_partition / transient_loss_legacy: short network partition or packet loss that already cleared; propose none.

vm/ai-env/prompts/topology.md

@@ -0,0 +1,4 @@
+Container layer (AWS Tokyo, kubeadm v1.37, flannel): control plane ip-10-10-0-10 (10.10.0.10, API 6443), workers ip-10-10-10-11, ip-10-10-10-12, ip-10-10-11-11. Namespace demo: Deployment front (2 replicas, nginx) and api (3 replicas, Go) behind ingress-nginx; ALB forwards to NodePort 30080 on the workers; path /api/* -> api, / -> front. Argo CD (namespace argocd) syncs everything from the public gitops repository with selfHeal and prune. Chaos Mesh in namespace chaos-mesh injects faults during campaigns.
+VM layer (GCP Tokyo, subnet 10.20.0.0/20): observability VM selfheal-obs 10.20.0.10 (Mimir 9009, Loki 3100, Tempo 4317, Grafana 3000); harness VM selfheal-harness 10.20.0.11 (runner 8080, blackbox probes, k6 load); AI VM selfheal-ai 10.20.0.12 (this agent); PostgreSQL primary selfheal-db-primary 10.20.0.20 and streaming replica selfheal-db-replica 10.20.0.21 (role app, database app, table app.price); legacy-api runs in docker compose on the MIG selfheal-legacy-mig (2 instances named legacy-xxxx, port 8080) behind the internal load balancer 10.20.0.100:8080 with autohealing.
+Request path: client -> ALB -> ingress-nginx -> api pod -> (VPN) -> ILB 10.20.0.100 -> legacy-api -> db-primary. The two clouds are joined by an HA VPN (4 tunnels, BGP); AWS 10.10.0.0/16 <-> GCP 10.20.0.0/20.
+Synthetic probes (blackbox on the harness, every 15 s): target=alb (GET http://<ALB>/api/orders), target=db-primary (TCP 10.20.0.20:5432).