feat: AI VM stack (WP-12) and Alloy telemetry on k8s, legacy and DB VMs (WP-15)

WP-12(실험 설계서 6장) vm/ai-env: apply.sh(시크릿 4종을 Secret Manager에서 파일로, 미등록이면
성능 저하 모드로 기동), compose(agent 8080·hitl 8081·rag-db pgvector 0.8.7-pg16), ai.env(Discord
서버 Selfheal Lab 1557424985793101824, approvals 1557425140302872698, escalation 1557425282363691048,
OPENAI_MODEL gpt-6.1-sol 후보), 프롬프트 3종, 러너북 11종(조치별·토폴로지·3층 절차).

WP-15(실험 설계서 4.1·4.3): apps/alloy.yaml(Alloy Helm 1.13.0 DaemonSet, 클러스터링으로 KSM·Argo CD·
cadvisor 스크랩 분산, demo 등 파드 로그 → Loki), apps/kube-state-metrics.yaml(Helm 8.6.0).
legacy-api compose에 Alloy 사이드카(up{job="legacy-api"}, 컨테이너 로그). DB VM은
vm/postgres-common/alloy.sh로 apt 설치 + prometheus.exporter.postgres(pg_replication_lag_seconds)
+ custom query pg_is_wal_replay_paused + PostgreSQL 로그.

vm/harness/apply.sh: 준비 대기 루프의 curl에 || true(러너 재기동 순간 curl 56으로 중단되던 문제,
2026-10-08 harness-sync 실측).

vm/ai-env/runbooks/db-primary-down.md

@@ -0,0 +1,5 @@
+# db_primary_down: primary unreachable (layer 3)
+
+Symptoms: probe_success{target="db-primary"} = 0, api 5xx on /api/orders and /api/price, legacy-api logs show connection refused or timeout to 10.20.0.20:5432.
+
+Decision tree for the human: 1) Is the VM RUNNING and only PostgreSQL down? Restart PostgreSQL on the primary (service restart; layer 1 systemd handles crashes). 2) Is the VM stopped or lost? gce.instance_reset or recreate through Terraform. 3) Only if the primary cannot come back: promote the replica (`SELECT pg_promote()` on selfheal-db-replica) and switch the application DSN to 10.20.0.21; this is irreversible (timeline fork), the old primary must be re-cloned. The agent must escalate, never propose promotion.

vm/ai-env/runbooks/db-replay-resume.md

@@ -0,0 +1,5 @@
+# db.replay_resume (layer 2, small impact, reversible)
+
+When: DbReplicationLag fires with pg_is_wal_replay_paused = 1 on selfheal-db-replica (replay was paused by an experiment or an operator). Lag alone with replay not paused points to network or load; do not propose this action then.
+
+Procedure (harness action db.replay_resume, target {host: db-replica}): the harness runs `SELECT pg_wal_replay_resume()` as role harness over TLS. Verify pg_replication_lag_seconds falling to near 0 within a minute. Undo: `pg_wal_replay_pause()` (only for experiments).

vm/ai-env/runbooks/escalation.md

@@ -0,0 +1,5 @@
+# Layer-3 queue: how a human acts
+
+Every escalation lands in the Discord escalation channel with the diagnosis and the matching runbook in a thread. The human either follows the runbook directly or calls the harness with the operator token (Secret Manager selfheal-harness-token): `POST http://10.20.0.11:8080/actions/execute {"action_id": "...", "target": {...}, "token": "<operator token>", "note": "..."}`. Both paths are recorded in the harness events as actor=human, which feeds the human-path MTTR.
+
+Proposals that receive no decision within 15 minutes are moved here automatically; the original message is marked expired.

vm/ai-env/runbooks/gce-instance-reset.md

@@ -0,0 +1,5 @@
+# gce.instance_reset (layer 2, large impact, reversible)
+
+When: a pet VM (selfheal-obs, selfheal-harness, selfheal-db-replica) is hung: no SSH, no metrics, docker not responding, while the instance status is RUNNING. Do not reset selfheal-db-primary on a suspicion; a primary outage is a layer-3 decision (db-primary-down.md).
+
+Procedure (harness action gce.instance_reset, target {instance: selfheal-db-replica}): `gcloud compute instances reset`; the startup script re-runs vm/<role>/apply.sh and services come back in 2 to 4 minutes. Undo: none; the disk is kept.

vm/ai-env/runbooks/k8s-node-drain.md

@@ -0,0 +1,5 @@
+# k8s.node_drain and k8s.node_uncordon (layer 2, large impact, reversible)
+
+When: one worker is unhealthy (NotReady, kubelet errors, pods on that node failing while other nodes are fine). Draining moves the pods to the other two workers.
+
+Procedure (harness action k8s.node_drain, target {node: ip-10-10-10-11}): cordon + drain with eviction (PDB respected, may take minutes). Afterwards investigate the node. Undo: k8s.node_uncordon with the same target. Only drain one worker at a time; the api PDB needs two replicas available.

vm/ai-env/runbooks/k8s-rollout-restart.md

@@ -0,0 +1,5 @@
+# k8s.rollout_restart (layer 1 with signature, layer 2 in the gray zone)
+
+When: api or front pods are degraded without a configuration change (readiness failing, stale connections after a network incident, memory stress that cleared). Impact small, reversible.
+
+Procedure (harness action k8s.rollout_restart, target {namespace: demo, deployment: api}): the harness runs the equivalent of `kubectl -n demo rollout restart deployment/api`; pods are replaced one by one under the PodDisruptionBudget. Verify with `kubectl -n demo rollout status deployment/api` and probe_success{target="alb"} back to 1 within 2 minutes. Undo: none needed.

vm/ai-env/runbooks/k8s-rollout-undo.md

@@ -0,0 +1,5 @@
+# k8s.rollout_undo (layer 2, large impact, reversible)
+
+When: a new revision of api or front started failing (ReplicaSet of the latest revision has ready < desired while the previous revision was healthy). Check `get_k8s replicasets` revisions and events before proposing.
+
+Procedure (harness action k8s.rollout_undo, target {namespace: demo, deployment: api}): the harness runs `kubectl rollout undo deployment/<name>`. Note that Argo CD selfHeal will re-apply the Git revision; a lasting fix needs a Git commit. Undo: run rollout undo again.

vm/ai-env/runbooks/k8s-scale.md

@@ -0,0 +1,5 @@
+# k8s.scale (layer 2, small impact, reversible, costs resources)
+
+When: latency or error ratio is caused by capacity (all api pods Running, CPU or memory saturated, no crashes) and more replicas would absorb the load. Not for crashes or DB problems.
+
+Procedure (harness action k8s.scale, target {namespace: demo, deployment: api, replicas: N}): the harness sets spec.replicas. Argo CD selfHeal restores the Git value (3) at the next sync unless the change is committed, so this is a temporary relief. Undo: scale back to the original replicas.

vm/ai-env/runbooks/mig-recreate-instance.md

@@ -0,0 +1,5 @@
+# mig.recreate_instance and mig.resize (layer 2, small impact, reversible)
+
+When: one legacy-api instance serves errors or is unreachable but the MIG health check has not (yet) replaced it; or the MIG target size is below 2 after an experiment.
+
+Procedure (harness action mig.recreate_instance, target {instance: legacy-xxxx} or {instance: "legacy:0"}): the instance is deleted and recreated from the template; the new instance clones the gitops repository and starts compose. 2 to 3 minutes. mig.resize target {mig: selfheal-legacy-mig, size: 2} restores the standard size. Undo: resize back to 2 before the end of the session (ownership rule).

vm/ai-env/runbooks/net-route-change.md

@@ -0,0 +1,5 @@
+# net.route_change (layer 2, large impact, reversible, Terraform path)
+
+When: the cross-cloud path (api pods -> VPN -> ILB -> legacy) is broken by a route or firewall problem that is not transient (BGP sessions down, routes missing on one side). Transient partitions and packet loss that already cleared need no action.
+
+Procedure: the harness records the proposal; the change itself is applied by an operator through Terraform (stacks/10-network). Undo: Terraform apply of the previous revision.

vm/ai-env/runbooks/schema-mismatch.md

@@ -0,0 +1,5 @@
+# schema_mismatch: application errors after a schema or data change (layer 3)
+
+Symptoms: api or legacy-api 5xx with SQL errors in the logs (column does not exist, relation missing, constraint violation) while the database is reachable and replication is fine.
+
+Decision for the human: identify the change (ask who ran it, check recent DDL), then either roll the application forward to a matching version through Git or repair the schema/data with SQL (db.data_fix, irreversible). Restarting pods does not help. The agent must escalate.

vm/ai-env/runbooks/topology.md

@@ -0,0 +1,7 @@
+# Topology and ownership
+
+Container layer on AWS (kubeadm cluster: control plane 10.10.0.10, three workers 10.10.10.11, 10.10.10.12, 10.10.11.11). Workloads in namespace demo: api (3 replicas) and front (2 replicas), exposed through ingress-nginx and the ALB. Argo CD keeps the cluster equal to the gitops repository (selfHeal, prune).
+
+VM layer on GCP (subnet 10.20.0.0/20): selfheal-obs 10.20.0.10 (Mimir, Loki, Tempo, Grafana), selfheal-harness 10.20.0.11 (runner, probes, k6), selfheal-ai 10.20.0.12 (diagnosis agent and Discord bot), selfheal-db-primary 10.20.0.20, selfheal-db-replica 10.20.0.21 (streaming replication, slot replica1), legacy MIG selfheal-legacy-mig (instances legacy-xxxx, docker compose legacy-api on 8080) behind the ILB 10.20.0.100:8080.
+
+Ownership: Terraform owns instances, networks and the MIG target size (restore 2 before the end of a session). Git owns everything inside the cluster. Operators change VM services only through the gitops vm/<role>/apply.sh scripts.