feat: AI VM stack (WP-12) and Alloy telemetry on k8s, legacy and DB VMs (WP-15)

WP-12(실험 설계서 6장) vm/ai-env: apply.sh(시크릿 4종을 Secret Manager에서 파일로, 미등록이면
성능 저하 모드로 기동), compose(agent 8080·hitl 8081·rag-db pgvector 0.8.7-pg16), ai.env(Discord
서버 Selfheal Lab 1557424985793101824, approvals 1557425140302872698, escalation 1557425282363691048,
OPENAI_MODEL gpt-6.1-sol 후보), 프롬프트 3종, 러너북 11종(조치별·토폴로지·3층 절차).

WP-15(실험 설계서 4.1·4.3): apps/alloy.yaml(Alloy Helm 1.13.0 DaemonSet, 클러스터링으로 KSM·Argo CD·
cadvisor 스크랩 분산, demo 등 파드 로그 → Loki), apps/kube-state-metrics.yaml(Helm 8.6.0).
legacy-api compose에 Alloy 사이드카(up{job="legacy-api"}, 컨테이너 로그). DB VM은
vm/postgres-common/alloy.sh로 apt 설치 + prometheus.exporter.postgres(pg_replication_lag_seconds)
+ custom query pg_is_wal_replay_paused + PostgreSQL 로그.

vm/harness/apply.sh: 준비 대기 루프의 curl에 || true(러너 재기동 순간 curl 56으로 중단되던 문제,
2026-10-08 harness-sync 실측).

vm/legacy-api/apply.sh

@@ -2,7 +2,7 @@
 # legacy-api VM (GCP MIG instance). Idempotent: run at first boot by the startup script and again by `make vm-sync`.
 # 1) DB password from Secret Manager via the VM service account (metadata token; no key files, design 6.11)
 # 2) docker compose up with the pinned image tag
-# Alloy (metrics/logs shipping) is added in WP-15.
+# 3) Alloy sidecar (WP-15): metrics + logs to the observability VM; INSTANCE = this VM's name for the instance label
 set -euo pipefail
 cd "$(dirname "$0")"
 PROJECT="$(curl -sf -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/project/project-id)"
@@ -16,6 +16,7 @@ ENV_FILE=/opt/selfheal/legacy-api.env
 umask 077
 {
   printf 'DB_DSN=postgres://app:%s@10.20.0.20:5432/app?sslmode=require\n' "${DB_PASSWORD}"
+  printf 'INSTANCE=%s\n' "$(hostname)"
 } > "$ENV_FILE"
 unset DB_PASSWORD
 docker compose --env-file "$ENV_FILE" -f compose.yaml up -d --pull always --remove-orphans

vm/legacy-api/compose.yaml

@@ -1,4 +1,6 @@
 # legacy-api on the MIG instance (experiment design 3.2). Image tag is pinned to the apps repo commit.
+# alloy (WP-15, design 4.1): scrapes legacy-api /metrics (up{job="legacy-api"} feeds LegacyUnhealthy) into Mimir and
+# ships the container logs to Loki. Runs as root for the docker socket (log discovery).
 services:
   legacy-api:
     image: asia-northeast1-docker.pkg.dev/cms-d-414903/selfheal/legacy-api:7186ef0
@@ -14,3 +16,17 @@ services:
       interval: 10s
       timeout: 3s
       retries: 3
+  alloy:
+    image: grafana/alloy:v1.20.1
+    container_name: alloy
+    restart: always
+    user: "0"
+    command: ["run", "--server.http.listen-addr=127.0.0.1:12345", "--storage.path=/var/lib/alloy/data", "/etc/alloy/config.alloy"]
+    environment:
+      INSTANCE: "${INSTANCE}"
+    volumes:
+      - ./config.alloy:/etc/alloy/config.alloy:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - alloy-data:/var/lib/alloy/data
+volumes:
+  alloy-data: {}

vm/legacy-api/config.alloy

@@ -0,0 +1,33 @@
+// Alloy on a legacy MIG instance (design 4.1 / 4.3): legacy-api metrics -> Mimir (job="legacy-api", instance=<vm name>),
+// container logs -> Loki (job="legacy-api"). Single tenant: no X-Scope-OrgID.
+prometheus.scrape "legacy_api" {
+  job_name        = "legacy-api"
+  scrape_interval = "15s"
+  targets         = [{ "__address__" = "legacy-api:8080", "instance" = sys.env("INSTANCE") }]
+  forward_to      = [prometheus.remote_write.mimir.receiver]
+}
+
+prometheus.remote_write "mimir" {
+  endpoint { url = "http://10.20.0.10:9009/api/v1/push" }
+}
+
+discovery.docker "local" {
+  host = "unix:///var/run/docker.sock"
+}
+
+discovery.relabel "containers" {
+  targets = discovery.docker.local.targets
+  rule { source_labels = ["__meta_docker_container_name"], regex = "/(.*)", target_label = "container" }
+  rule { target_label = "job", replacement = "legacy-api" }
+  rule { target_label = "instance", replacement = sys.env("INSTANCE") }
+}
+
+loki.source.docker "containers" {
+  host       = "unix:///var/run/docker.sock"
+  targets    = discovery.relabel.containers.output
+  forward_to = [loki.write.loki.receiver]
+}
+
+loki.write "loki" {
+  endpoint { url = "http://10.20.0.10:3100/loki/api/v1/push" }
+}