feat: AI VM stack (WP-12) and Alloy telemetry on k8s, legacy and DB VMs (WP-15)

WP-12(실험 설계서 6장) vm/ai-env: apply.sh(시크릿 4종을 Secret Manager에서 파일로, 미등록이면 성능 저하 모드로 기동), compose(agent 8080·hitl 8081·rag-db pgvector 0.8.7-pg16), ai.env(Discord 서버 Selfheal Lab 1557424985793101824, approvals 1557425140302872698, escalation 1557425282363691048, OPENAI_MODEL gpt-6.1-sol 후보), 프롬프트 3종, 러너북 11종(조치별·토폴로지·3층 절차). WP-15(실험 설계서 4.1·4.3): apps/alloy.yaml(Alloy Helm 1.13.0 DaemonSet, 클러스터링으로 KSM·Argo CD· cadvisor 스크랩 분산, demo 등 파드 로그 → Loki), apps/kube-state-metrics.yaml(Helm 8.6.0). legacy-api compose에 Alloy 사이드카(up{job="legacy-api"}, 컨테이너 로그). DB VM은 vm/postgres-common/alloy.sh로 apt 설치 + prometheus.exporter.postgres(pg_replication_lag_seconds) + custom query pg_is_wal_replay_paused + PostgreSQL 로그. vm/harness/apply.sh: 준비 대기 루프의 curl에 || true(러너 재기동 순간 curl 56으로 중단되던 문제, 2026-10-08 harness-sync 실측).
2026-10-08 01:47:38 +09:00
1 parent 59995f3601
commit 7ce2dc68fe
@@ -1,3 +1,19 @@
 # vm/ai-env

-Placeholder. See the root README for the work package that fills this directory.
+AI VM (10.20.0.12) of the self-healing lab: experiment design 6 (diagnosis agent, pgvector RAG store, Discord HITL bot).
+
+| file | purpose |
+|---|---|
+| `apply.sh` | boot / `make vm-sync` entry: secrets from Secret Manager to `/opt/selfheal/ai/secrets`, env file, compose up. Succeeds even when the OpenAI key or Discord token is not registered yet (services run degraded) |
+| `compose.yaml` | agent (8080), hitl (8081), rag-db (pgvector 0.8.7 / PostgreSQL 16, internal) |
+| `ai.env` | non-secret settings: Discord guild and channel ids, model ids, internal endpoints |
+| `image.tag` | image tag (`selfheal-public/ai-agent:<sha>` and `ai-hitl:<sha>` built from `selfheal-lab-apps/ai`) |
+| `prompts/system.md` | system prompt; its sha256 prefix is the `prompt_version` recorded with every session |
+| `prompts/topology.md`, `prompts/slo.md` | context blocks of the user message (design 6.4) |
+| `runbooks/*.md` | RAG documents: one per catalogue action, topology, layer-3 procedures. Re-ingested at agent start (hash based) |
+
+Agent API: `POST /diagnose {"alert": {...}, "run_id", "scenario_id"}` (called by the harness) returns the strict diagnosis plus `disposition` (proposal | escalated | none). `GET /selfcheck` checks key presence, rag-db, Mimir, k8s, GCP, AWS and the harness.
+
+HITL bot: `POST /post` (agent -> Discord message with Approve/Reject buttons), `POST /escalate` (agent -> escalation channel + runbook thread), `GET /selfcheck`. Buttons: approve -> harness `/proposals/{token}/approve` then `/actions/execute`; reject -> `/proposals/{token}/reject`; no decision within the window -> escalation channel. Gateway intents: GUILDS only (design 11.7); interactions need no intent.
+
+Secrets (Secret Manager, VM service account): selfheal-openai-api-key, selfheal-discord-bot-token (operator registers the values), selfheal-k8s-ai-token and selfheal-k8s-ca-cert (`make token`). Verify presence only: `gcloud secrets versions list selfheal-openai-api-key --project cms-d-414903`.
@@ -0,0 +1,14 @@
+# Non-secret settings of the AI VM (merged into /opt/selfheal/ai.env by apply.sh). Secrets never go here.
+# Discord server "Selfheal Lab" (created 2026-10-08): approvals channel for layer-2 buttons, escalation channel for layer 3.
+DISCORD_GUILD_ID=1557424985793101824
+DISCORD_APPROVALS_CHANNEL_ID=1557425140302872698
+DISCORD_ESCALATION_CHANNEL_ID=1557425282363691048
+# optional: restrict button decisions to one role id (empty = any member of the server)
+DISCORD_APPROVER_ROLE_ID=
+# experiment design 11.1: candidates gpt-6-astra / gpt-6.1-sol / gpt-6-luna (OpenAI model page 2026-10-08); fixed after WP-12 measurement
+OPENAI_MODEL=gpt-6.1-sol
+EMBED_MODEL=text-embedding-3-small
+HARNESS_URL=http://10.20.0.11:8080
+MIMIR_URL=http://10.20.0.10:9009/prometheus
+LOKI_URL=http://10.20.0.10:3100
+K8S_API=https://10.10.0.10:6443
@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# AI VM (10.20.0.12), experiment design 6. Idempotent: first boot (startup script) and `make vm-sync` / `make ai-sync`.
+# Secrets come from Secret Manager through the VM service account (Terraform design 6.11):
+#   selfheal-openai-api-key, selfheal-discord-bot-token (registered by the operator), selfheal-k8s-ai-token, selfheal-k8s-ca-cert (make token)
+# The two operator secrets may not exist yet: the services then start degraded (agent /diagnose 503, hitl without
+# gateway) and this script still succeeds, so the VM can be verified before the keys arrive.
+# Non-secret settings (Discord IDs, model) are in ai.env (committed); AWS role ARN comes from instance metadata.
+set -euo pipefail
+cd "$(dirname "$0")"
+MD=http://metadata.google.internal/computeMetadata/v1
+md() { curl -sf -H 'Metadata-Flavor: Google' "$MD/$1"; }
+PROJECT="$(md project/project-id)"
+AWS_ROLE_ARN="$(md instance/attributes/aws-role-arn || true)"
+command -v jq >/dev/null || { apt-get update -qq && apt-get install -y -qq jq >/dev/null; }
+TOKEN="$(md instance/service-accounts/default/token | jq -r .access_token)"
+# prints the secret value, or nothing when the secret has no version yet (HTTP 404 / FAILED_PRECONDITION)
+secret() { curl -s -H "Authorization: Bearer ${TOKEN}" "https://secretmanager.googleapis.com/v1/projects/${PROJECT}/secrets/$1/versions/latest:access" | jq -r '.payload.data // empty' | base64 -d; }
+
+SEC=/opt/selfheal/ai/secrets
+mkdir -p "$SEC"; chmod 700 /opt/selfheal/ai "$SEC"
+umask 077
+secret selfheal-openai-api-key    > "$SEC/openai-api-key"
+secret selfheal-discord-bot-token > "$SEC/discord-bot-token"
+secret selfheal-k8s-ai-token      > "$SEC/k8s-token"
+secret selfheal-k8s-ca-cert       > "$SEC/k8s-ca.crt"
+unset TOKEN
+for f in openai-api-key discord-bot-token; do [ -s "$SEC/$f" ] || echo "note: secret $f is empty (not registered yet); service runs degraded" >&2; done
+for f in k8s-token k8s-ca.crt; do [ -s "$SEC/$f" ] || echo "note: $f is empty (run make token on the operator host)" >&2; done
+chown -R 10002:10002 "$SEC"; chmod 600 "$SEC"/*   # agent/hitl run as uid 10002
+
+ENV_FILE=/opt/selfheal/ai.env
+if [ ! -f "$ENV_FILE" ] || ! grep -q '^RAG_DB_PASSWORD=' "$ENV_FILE"; then
+  RDB_PW="$(head -c 24 /dev/urandom | base64 | tr -d '/+=')"
+else
+  RDB_PW="$(sed -n 's/^RAG_DB_PASSWORD=//p' "$ENV_FILE")"
+fi
+AI_TAG="$(cat image.tag)"
+{
+  cat ai.env
+  printf 'AWS_ROLE_ARN=%s\n' "$AWS_ROLE_ARN"
+  printf 'AWS_REGION=ap-northeast-1\n'
+  printf 'GCP_PROJECT=%s\n' "$PROJECT"
+  printf 'AI_TAG=%s\n' "$AI_TAG"
+  printf 'RAG_DB_PASSWORD=%s\n' "$RDB_PW"
+} > "$ENV_FILE"
+unset RDB_PW
+
+docker compose --env-file "$ENV_FILE" -f compose.yaml up -d --pull always --remove-orphans
+# both services read their secret files at start (hitl needs the token to open the gateway): recreate so new secrets apply
+docker compose --env-file "$ENV_FILE" -f compose.yaml up -d --force-recreate --no-deps agent hitl
+for i in $(seq 1 30); do
+  a=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/healthz || true); h=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8081/healthz || true)
+  if [ "$a$h" = "200200" ]; then break; fi; sleep 5
+done
+echo "ready: agent=$a hitl=$h"
+docker compose --env-file "$ENV_FILE" -f compose.yaml ps --format 'table {{.Name}}\t{{.Status}}'
@@ -0,0 +1,49 @@
+# AI VM (10.20.0.12), experiment design 6.1: agent 8080 (/diagnose for the harness), rag-db (pgvector, internal),
+# hitl 8081 (Discord gateway bot; /post and /escalate for the agent). Images: selfheal-lab-apps/ai (Cloud Build).
+services:
+  agent:
+    image: asia-northeast1-docker.pkg.dev/cms-d-414903/selfheal-public/ai-agent:${AI_TAG}
+    container_name: agent
+    restart: always
+    env_file: /opt/selfheal/ai.env
+    environment:
+      RAG_DSN: "postgres://ai:${RAG_DB_PASSWORD}@rag-db:5432/ai?sslmode=disable"
+      HITL_URL: http://hitl:8081
+      OPENAI_KEY_FILE: /run/secrets/openai-api-key
+      K8S_TOKEN_FILE: /run/secrets/k8s-token
+      K8S_CA_FILE: /run/secrets/k8s-ca.crt
+      PROMPT_DIR: /etc/ai/prompts
+      RUNBOOK_DIR: /etc/ai/runbooks
+    volumes:
+      - /opt/selfheal/ai/secrets:/run/secrets:ro
+      - ./prompts:/etc/ai/prompts:ro
+      - ./runbooks:/etc/ai/runbooks:ro
+    ports:
+      - "8080:8080"
+    depends_on: [rag-db]
+  hitl:
+    image: asia-northeast1-docker.pkg.dev/cms-d-414903/selfheal-public/ai-hitl:${AI_TAG}
+    container_name: hitl
+    restart: always
+    env_file: /opt/selfheal/ai.env
+    environment:
+      RAG_DSN: "postgres://ai:${RAG_DB_PASSWORD}@rag-db:5432/ai?sslmode=disable"
+      DISCORD_TOKEN_FILE: /run/secrets/discord-bot-token
+    volumes:
+      - /opt/selfheal/ai/secrets:/run/secrets:ro
+    ports:
+      - "8081:8081"
+    depends_on: [rag-db]
+  rag-db:
+    # pgvector 0.8.7 on PostgreSQL 16 (Docker Hub tag checked 2026-10-08); HNSW index up to 2000 dimensions (design 6.4)
+    image: pgvector/pgvector:0.8.7-pg16
+    container_name: rag-db
+    restart: always
+    environment:
+      POSTGRES_USER: ai
+      POSTGRES_PASSWORD: "${RAG_DB_PASSWORD}"
+      POSTGRES_DB: ai
+    volumes:
+      - rag-data:/var/lib/postgresql/data
+volumes:
+  rag-data: {}
@@ -0,0 +1 @@
+f30af2a
@@ -0,0 +1,4 @@
+Availability: avg_over_time(probe_success[5m]) >= 99.5% over the campaign. Latency: api p95 (histogram_quantile(0.95, sum by (le) (rate(demo_http_request_duration_seconds_bucket[5m])))) < 0.5 s. Error ratio: 5xx / all over 5 minutes < 1%.
+Recovery (t_recover) is declared by the harness when the probe succeeds 12 times in a row (3 minutes) and latency and error ratio are back inside the SLO over a 5-minute window.
+Layers: 1 = platform-internal automatic recovery (kubelet restart, ReplicaSet, Argo CD selfHeal, MIG autohealing, compose restart). 2 = your proposal + human approval in Discord, executed by the harness. 3 = human-only (irreversible actions, no approval within 15 minutes, confidence below 0.5).
+Alert rules and what they mean: ProbeDown (probe_success == 0 for 1 min, symptom only), ApiLatencyHigh (p95 > 500 ms for 3 min), ApiErrorRateHigh (5xx > 1% for 3 min), PodCrashLooping (3+ restarts in 5 min; layer 1 first, layer 2 after 5 min), LegacyUnhealthy (fewer than 2 legacy-api instances up; autohealing first, layer 2 after 10 min), DbReplicationLag (lag > 60 s or replay paused), DbPrimaryDown (TCP 5432 failing for 1 min), GitOpsDrift (OutOfSync 5 min; selfHeal handles it).
@@ -0,0 +1,24 @@
+You are the layer-2 diagnosis agent of a self-healing verification lab (VM + container mixed environment). You have read-only tools. You never execute changes; a human approves or rejects what you propose, and the harness executes approved actions.
+
+Role and constraints:
+1. Diagnose the alert you are given by querying the tools (metrics, logs, Kubernetes objects, GCE instances, AWS state, runbooks). Do not guess without evidence; every claim in your evidence list must be a quotation or a concrete value from a tool result.
+2. Propose only actions from the catalogue enum of submit_diagnosis. If no catalogue action fits, or the right action is irreversible (database promotion, data fix, instance lifecycle), choose "escalate". If the platform already recovered the service (layer 1), choose "none".
+3. Prefer the smallest action that addresses the root cause. Order of preference: none (already recovering) < k8s.rollout_restart / mig.recreate_instance / db.replay_resume (small impact) < k8s.rollout_undo / k8s.scale / mig.resize < k8s.node_drain / gce.instance_reset / net.route_change (large impact) < escalate.
+4. Transient conditions: if the symptom is already gone in the latest samples (probe back to 1, latency and error ratio inside the SLO, pods Running with no new restarts), report the most likely label with the evidence and propose "none"; do not invent a cause.
+5. Set confidence honestly. Below 0.5 the case goes to a human without a proposal. Use "unknown" when the evidence does not support any label.
+6. Keep rationale under 200 characters. Target must be the exact object name the harness needs: "demo/api" or "api" for deployments, a node name for drain, a pet VM name (selfheal-db-replica) or legacy instance name (legacy-xxxx) for GCE actions.
+7. Call submit_diagnosis exactly once, after at most a handful of tool calls. Do not write prose answers.
+
+Label hints (ground truth labels of the scenario catalogue):
+- api_pod_killed: one api pod killed, ReplicaSet recreates it (layer 1, usually none).
+- api_pod_failure: api pods crash-looping / failing readiness (restart counts rising).
+- memory_pressure_api: api pods under memory stress; latency up, OOMKilled possible.
+- network_latency_api_to_db: api to DB path slow (p95 up, no restarts, DB itself healthy).
+- gitops_drift: Argo CD application OutOfSync; selfHeal restores it (layer 1).
+- legacy_process_killed: legacy-api container killed on a MIG instance; compose restarts it.
+- legacy_instance_lost: a legacy MIG instance deleted/stopped; autohealing recreates it.
+- network_latency_legacy: ILB to legacy path slow or lossy.
+- replication_paused: db-replica WAL replay paused (pg_is_wal_replay_paused = 1, lag rising).
+- db_primary_down: db-primary unreachable on 5432 (probe db-primary = 0). Promotion is irreversible: escalate.
+- schema_mismatch: application errors from a schema/data change on the DB (5xx with SQL errors in logs). Data fix is irreversible: escalate.
+- transient_partition / transient_loss_legacy: short network partition or packet loss that already cleared; propose none.
@@ -0,0 +1,4 @@
+Container layer (AWS Tokyo, kubeadm v1.37, flannel): control plane ip-10-10-0-10 (10.10.0.10, API 6443), workers ip-10-10-10-11, ip-10-10-10-12, ip-10-10-11-11. Namespace demo: Deployment front (2 replicas, nginx) and api (3 replicas, Go) behind ingress-nginx; ALB forwards to NodePort 30080 on the workers; path /api/* -> api, / -> front. Argo CD (namespace argocd) syncs everything from the public gitops repository with selfHeal and prune. Chaos Mesh in namespace chaos-mesh injects faults during campaigns.
+VM layer (GCP Tokyo, subnet 10.20.0.0/20): observability VM selfheal-obs 10.20.0.10 (Mimir 9009, Loki 3100, Tempo 4317, Grafana 3000); harness VM selfheal-harness 10.20.0.11 (runner 8080, blackbox probes, k6 load); AI VM selfheal-ai 10.20.0.12 (this agent); PostgreSQL primary selfheal-db-primary 10.20.0.20 and streaming replica selfheal-db-replica 10.20.0.21 (role app, database app, table app.price); legacy-api runs in docker compose on the MIG selfheal-legacy-mig (2 instances named legacy-xxxx, port 8080) behind the internal load balancer 10.20.0.100:8080 with autohealing.
+Request path: client -> ALB -> ingress-nginx -> api pod -> (VPN) -> ILB 10.20.0.100 -> legacy-api -> db-primary. The two clouds are joined by an HA VPN (4 tunnels, BGP); AWS 10.10.0.0/16 <-> GCP 10.20.0.0/20.
+Synthetic probes (blackbox on the harness, every 15 s): target=alb (GET http://<ALB>/api/orders), target=db-primary (TCP 10.20.0.20:5432).
@@ -0,0 +1,5 @@
+# db_primary_down: primary unreachable (layer 3)
+
+Symptoms: probe_success{target="db-primary"} = 0, api 5xx on /api/orders and /api/price, legacy-api logs show connection refused or timeout to 10.20.0.20:5432.
+
+Decision tree for the human: 1) Is the VM RUNNING and only PostgreSQL down? Restart PostgreSQL on the primary (service restart; layer 1 systemd handles crashes). 2) Is the VM stopped or lost? gce.instance_reset or recreate through Terraform. 3) Only if the primary cannot come back: promote the replica (`SELECT pg_promote()` on selfheal-db-replica) and switch the application DSN to 10.20.0.21; this is irreversible (timeline fork), the old primary must be re-cloned. The agent must escalate, never propose promotion.
@@ -0,0 +1,5 @@
+# db.replay_resume (layer 2, small impact, reversible)
+
+When: DbReplicationLag fires with pg_is_wal_replay_paused = 1 on selfheal-db-replica (replay was paused by an experiment or an operator). Lag alone with replay not paused points to network or load; do not propose this action then.
+
+Procedure (harness action db.replay_resume, target {host: db-replica}): the harness runs `SELECT pg_wal_replay_resume()` as role harness over TLS. Verify pg_replication_lag_seconds falling to near 0 within a minute. Undo: `pg_wal_replay_pause()` (only for experiments).
@@ -0,0 +1,5 @@
+# Layer-3 queue: how a human acts
+
+Every escalation lands in the Discord escalation channel with the diagnosis and the matching runbook in a thread. The human either follows the runbook directly or calls the harness with the operator token (Secret Manager selfheal-harness-token): `POST http://10.20.0.11:8080/actions/execute {"action_id": "...", "target": {...}, "token": "<operator token>", "note": "..."}`. Both paths are recorded in the harness events as actor=human, which feeds the human-path MTTR.
+
+Proposals that receive no decision within 15 minutes are moved here automatically; the original message is marked expired.
@@ -0,0 +1,5 @@
+# gce.instance_reset (layer 2, large impact, reversible)
+
+When: a pet VM (selfheal-obs, selfheal-harness, selfheal-db-replica) is hung: no SSH, no metrics, docker not responding, while the instance status is RUNNING. Do not reset selfheal-db-primary on a suspicion; a primary outage is a layer-3 decision (db-primary-down.md).
+
+Procedure (harness action gce.instance_reset, target {instance: selfheal-db-replica}): `gcloud compute instances reset`; the startup script re-runs vm/<role>/apply.sh and services come back in 2 to 4 minutes. Undo: none; the disk is kept.
@@ -0,0 +1,5 @@
+# k8s.node_drain and k8s.node_uncordon (layer 2, large impact, reversible)
+
+When: one worker is unhealthy (NotReady, kubelet errors, pods on that node failing while other nodes are fine). Draining moves the pods to the other two workers.
+
+Procedure (harness action k8s.node_drain, target {node: ip-10-10-10-11}): cordon + drain with eviction (PDB respected, may take minutes). Afterwards investigate the node. Undo: k8s.node_uncordon with the same target. Only drain one worker at a time; the api PDB needs two replicas available.
@@ -0,0 +1,5 @@
+# k8s.rollout_restart (layer 1 with signature, layer 2 in the gray zone)
+
+When: api or front pods are degraded without a configuration change (readiness failing, stale connections after a network incident, memory stress that cleared). Impact small, reversible.
+
+Procedure (harness action k8s.rollout_restart, target {namespace: demo, deployment: api}): the harness runs the equivalent of `kubectl -n demo rollout restart deployment/api`; pods are replaced one by one under the PodDisruptionBudget. Verify with `kubectl -n demo rollout status deployment/api` and probe_success{target="alb"} back to 1 within 2 minutes. Undo: none needed.
@@ -0,0 +1,5 @@
+# k8s.rollout_undo (layer 2, large impact, reversible)
+
+When: a new revision of api or front started failing (ReplicaSet of the latest revision has ready < desired while the previous revision was healthy). Check `get_k8s replicasets` revisions and events before proposing.
+
+Procedure (harness action k8s.rollout_undo, target {namespace: demo, deployment: api}): the harness runs `kubectl rollout undo deployment/<name>`. Note that Argo CD selfHeal will re-apply the Git revision; a lasting fix needs a Git commit. Undo: run rollout undo again.
@@ -0,0 +1,5 @@
+# k8s.scale (layer 2, small impact, reversible, costs resources)
+
+When: latency or error ratio is caused by capacity (all api pods Running, CPU or memory saturated, no crashes) and more replicas would absorb the load. Not for crashes or DB problems.
+
+Procedure (harness action k8s.scale, target {namespace: demo, deployment: api, replicas: N}): the harness sets spec.replicas. Argo CD selfHeal restores the Git value (3) at the next sync unless the change is committed, so this is a temporary relief. Undo: scale back to the original replicas.
@@ -0,0 +1,5 @@
+# mig.recreate_instance and mig.resize (layer 2, small impact, reversible)
+
+When: one legacy-api instance serves errors or is unreachable but the MIG health check has not (yet) replaced it; or the MIG target size is below 2 after an experiment.
+
+Procedure (harness action mig.recreate_instance, target {instance: legacy-xxxx} or {instance: "legacy:0"}): the instance is deleted and recreated from the template; the new instance clones the gitops repository and starts compose. 2 to 3 minutes. mig.resize target {mig: selfheal-legacy-mig, size: 2} restores the standard size. Undo: resize back to 2 before the end of the session (ownership rule).
@@ -0,0 +1,5 @@
+# net.route_change (layer 2, large impact, reversible, Terraform path)
+
+When: the cross-cloud path (api pods -> VPN -> ILB -> legacy) is broken by a route or firewall problem that is not transient (BGP sessions down, routes missing on one side). Transient partitions and packet loss that already cleared need no action.
+
+Procedure: the harness records the proposal; the change itself is applied by an operator through Terraform (stacks/10-network). Undo: Terraform apply of the previous revision.
@@ -0,0 +1,5 @@
+# schema_mismatch: application errors after a schema or data change (layer 3)
+
+Symptoms: api or legacy-api 5xx with SQL errors in the logs (column does not exist, relation missing, constraint violation) while the database is reachable and replication is fine.
+
+Decision for the human: identify the change (ask who ran it, check recent DDL), then either roll the application forward to a matching version through Git or repair the schema/data with SQL (db.data_fix, irreversible). Restarting pods does not help. The agent must escalate.
@@ -0,0 +1,7 @@
+# Topology and ownership
+
+Container layer on AWS (kubeadm cluster: control plane 10.10.0.10, three workers 10.10.10.11, 10.10.10.12, 10.10.11.11). Workloads in namespace demo: api (3 replicas) and front (2 replicas), exposed through ingress-nginx and the ALB. Argo CD keeps the cluster equal to the gitops repository (selfHeal, prune).
+
+VM layer on GCP (subnet 10.20.0.0/20): selfheal-obs 10.20.0.10 (Mimir, Loki, Tempo, Grafana), selfheal-harness 10.20.0.11 (runner, probes, k6), selfheal-ai 10.20.0.12 (diagnosis agent and Discord bot), selfheal-db-primary 10.20.0.20, selfheal-db-replica 10.20.0.21 (streaming replication, slot replica1), legacy MIG selfheal-legacy-mig (instances legacy-xxxx, docker compose legacy-api on 8080) behind the ILB 10.20.0.100:8080.
+
+Ownership: Terraform owns instances, networks and the MIG target size (restore 2 before the end of a session). Git owns everything inside the cluster. Operators change VM services only through the gitops vm/<role>/apply.sh scripts.
@@ -52,8 +52,9 @@ docker compose --env-file "$ENV_FILE" -f compose.yaml up -d --pull always --remo
 # runner re-reads secret files per call; recreate it anyway so a new image tag / env takes effect
 docker compose --env-file "$ENV_FILE" -f compose.yaml up -d --force-recreate --no-deps runner
 for i in $(seq 1 30); do
-  h=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/healthz); b=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9115/-/healthy)
-  a=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:12345/-/ready)
+  # `|| true`: a reset while the runner restarts (curl 56) must not abort the script under set -e (seen 2026-10-08 harness-sync)
+  h=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/healthz || true); b=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9115/-/healthy || true)
+  a=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:12345/-/ready || true)
  if [ "$h$b$a" = "200200200" ]; then break; fi; sleep 5
 done
 echo "ready: runner=$h blackbox=$b alloy=$a"
@@ -2,7 +2,7 @@
 # legacy-api VM (GCP MIG instance). Idempotent: run at first boot by the startup script and again by `make vm-sync`.
 # 1) DB password from Secret Manager via the VM service account (metadata token; no key files, design 6.11)
 # 2) docker compose up with the pinned image tag
-# Alloy (metrics/logs shipping) is added in WP-15.
+# 3) Alloy sidecar (WP-15): metrics + logs to the observability VM; INSTANCE = this VM's name for the instance label
 set -euo pipefail
 cd "$(dirname "$0")"
 PROJECT="$(curl -sf -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/project/project-id)"
@@ -16,6 +16,7 @@ ENV_FILE=/opt/selfheal/legacy-api.env
 umask 077
 {
  printf 'DB_DSN=postgres://app:%s@10.20.0.20:5432/app?sslmode=require\n' "${DB_PASSWORD}"
+  printf 'INSTANCE=%s\n' "$(hostname)"
 } > "$ENV_FILE"
 unset DB_PASSWORD
 docker compose --env-file "$ENV_FILE" -f compose.yaml up -d --pull always --remove-orphans
@@ -1,4 +1,6 @@
 # legacy-api on the MIG instance (experiment design 3.2). Image tag is pinned to the apps repo commit.
+# alloy (WP-15, design 4.1): scrapes legacy-api /metrics (up{job="legacy-api"} feeds LegacyUnhealthy) into Mimir and
+# ships the container logs to Loki. Runs as root for the docker socket (log discovery).
 services:
  legacy-api:
    image: asia-northeast1-docker.pkg.dev/cms-d-414903/selfheal/legacy-api:7186ef0
@@ -14,3 +16,17 @@ services:
      interval: 10s
      timeout: 3s
      retries: 3
+  alloy:
+    image: grafana/alloy:v1.20.1
+    container_name: alloy
+    restart: always
+    user: "0"
+    command: ["run", "--server.http.listen-addr=127.0.0.1:12345", "--storage.path=/var/lib/alloy/data", "/etc/alloy/config.alloy"]
+    environment:
+      INSTANCE: "${INSTANCE}"
+    volumes:
+      - ./config.alloy:/etc/alloy/config.alloy:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - alloy-data:/var/lib/alloy/data
+volumes:
+  alloy-data: {}
@@ -0,0 +1,33 @@
+// Alloy on a legacy MIG instance (design 4.1 / 4.3): legacy-api metrics -> Mimir (job="legacy-api", instance=<vm name>),
+// container logs -> Loki (job="legacy-api"). Single tenant: no X-Scope-OrgID.
+prometheus.scrape "legacy_api" {
+  job_name        = "legacy-api"
+  scrape_interval = "15s"
+  targets         = [{ "__address__" = "legacy-api:8080", "instance" = sys.env("INSTANCE") }]
+  forward_to      = [prometheus.remote_write.mimir.receiver]
+}
+
+prometheus.remote_write "mimir" {
+  endpoint { url = "http://10.20.0.10:9009/api/v1/push" }
+}
+
+discovery.docker "local" {
+  host = "unix:///var/run/docker.sock"
+}
+
+discovery.relabel "containers" {
+  targets = discovery.docker.local.targets
+  rule { source_labels = ["__meta_docker_container_name"], regex = "/(.*)", target_label = "container" }
+  rule { target_label = "job", replacement = "legacy-api" }
+  rule { target_label = "instance", replacement = sys.env("INSTANCE") }
+}
+
+loki.source.docker "containers" {
+  host       = "unix:///var/run/docker.sock"
+  targets    = discovery.relabel.containers.output
+  forward_to = [loki.write.loki.receiver]
+}
+
+loki.write "loki" {
+  endpoint { url = "http://10.20.0.10:3100/loki/api/v1/push" }
+}
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# Alloy on the PostgreSQL VMs (WP-15, experiment design 4.1 / 4.3). Called by vm/postgres-{primary,replica}/apply.sh.
+# The harness role password is read from Secret Manager (selfheal-db-password) through the VM service account. Installs Alloy from the Grafana apt repository (systemd service `alloy`,
+# config /etc/alloy/config.alloy, environment /etc/default/alloy; https://grafana.com/docs/alloy/latest/configure/linux/),
+# exposes the built-in postgres exporter (pg_replication_lag_seconds) plus a custom query for pg_is_wal_replay_paused
+# (not in the upstream replication collector), and ships /var/log/postgresql to Loki. Idempotent.
+set -euo pipefail
+HERE="$(cd "$(dirname "$0")" && pwd)"
+if ! command -v alloy >/dev/null; then
+  mkdir -p /etc/apt/keyrings
+  wget -qO /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
+  chmod 644 /etc/apt/keyrings/grafana.asc
+  echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" > /etc/apt/sources.list.d/grafana.list
+  apt-get update -qq
+  apt-get install -y -qq alloy >/dev/null
+fi
+usermod -aG adm alloy   # /var/log/postgresql/*.log is postgres:adm 0640
+install -d -m 755 /etc/alloy
+install -m 644 "$HERE/config.alloy" /etc/alloy/config.alloy
+install -m 644 "$HERE/queries.yaml" /etc/alloy/queries.yaml
+PROJECT="$(curl -sf -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/project/project-id)"
+TOKEN="$(curl -sf -H 'Metadata-Flavor: Google' 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token' | jq -r .access_token)"
+PGPASSWORD="$(curl -sf -H "Authorization: Bearer ${TOKEN}" "https://secretmanager.googleapis.com/v1/projects/${PROJECT}/secrets/selfheal-db-password/versions/latest:access" | jq -r .payload.data | base64 -d)"
+unset TOKEN
+[ -n "$PGPASSWORD" ] || { echo "empty DB password" >&2; exit 1; }
+IP="$(hostname -I | awk '{print $1}')"
+umask 077
+cat > /etc/default/alloy <<ENV
+CONFIG_FILE="/etc/alloy/config.alloy"
+CUSTOM_ARGS="--server.http.listen-addr=127.0.0.1:12345 --storage.path=/var/lib/alloy/data"
+INSTANCE="$(hostname)"
+PG_DSN="postgresql://harness:${PGPASSWORD}@${IP}:5432/app?sslmode=require"
+ENV
+unset PGPASSWORD
+systemctl enable alloy >/dev/null 2>&1 || true
+systemctl restart alloy
+for i in $(seq 1 20); do
+  c=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:12345/-/ready || true)
+  [ "$c" = "200" ] && break; sleep 3
+done
+echo "alloy ready=$c"
@@ -0,0 +1,36 @@
+// Alloy on db-primary / db-replica (design 4.1 / 4.3): postgres exporter (built-in collectors: pg_replication_lag_seconds,
+// pg_up, pg_stat_*) + custom pg_is_wal_replay_paused -> Mimir; PostgreSQL logs -> Loki. instance = VM name.
+prometheus.exporter.postgres "db" {
+  data_source_names         = [sys.env("PG_DSN")]
+  custom_queries_config_path = "/etc/alloy/queries.yaml"
+  disable_settings_metrics  = true
+}
+
+prometheus.scrape "db" {
+  job_name        = "postgres"
+  scrape_interval = "15s"
+  targets         = prometheus.exporter.postgres.db.targets
+  forward_to      = [prometheus.relabel.instance.receiver]
+}
+
+prometheus.relabel "instance" {
+  forward_to = [prometheus.remote_write.mimir.receiver]
+  rule { target_label = "instance", replacement = sys.env("INSTANCE") }
+}
+
+prometheus.remote_write "mimir" {
+  endpoint { url = "http://10.20.0.10:9009/api/v1/push" }
+}
+
+local.file_match "pglogs" {
+  path_targets = [{ "__path__" = "/var/log/postgresql/*.log", "job" = "postgres", "instance" = sys.env("INSTANCE") }]
+}
+
+loki.source.file "pglogs" {
+  targets    = local.file_match.pglogs.targets
+  forward_to = [loki.write.loki.receiver]
+}
+
+loki.write "loki" {
+  endpoint { url = "http://10.20.0.10:3100/loki/api/v1/push" }
+}
@@ -0,0 +1,9 @@
+# Custom postgres_exporter queries (Alloy prometheus.exporter.postgres custom_queries_config_path).
+# Metric name = <key>_<metric>: pg_is_wal_replay_paused (used by the DbReplicationLag rule, design 4.3).
+pg_is_wal_replay:
+  query: "SELECT CASE WHEN pg_is_in_recovery() AND pg_is_wal_replay_paused() THEN 1 ELSE 0 END AS paused"
+  master: true
+  metrics:
+    - paused:
+        usage: "GAUGE"
+        description: "1 when this server is a replica and WAL replay is paused"
@@ -90,3 +90,6 @@ echo "--- primary status"
 sudo -u postgres psql -Atc "select version()" | cut -d, -f1
 sudo -u postgres psql -Atc "show ssl"
 sudo -u postgres psql -d app -Atc "select count(*) from app.price"
+
+# --- Alloy (WP-15): postgres exporter + logs to the observability VM (reads the harness password itself) ---
+bash "$(dirname "$0")/../postgres-common/alloy.sh"
@@ -50,3 +50,6 @@ sleep 3
 echo "--- replica status"
 sudo -u postgres psql -Atc "select pg_is_in_recovery()"
 sudo -u postgres psql -Atc "select status, sender_host, slot_name from pg_stat_wal_receiver" 2>/dev/null || true
+
+# --- Alloy (WP-15): postgres exporter + logs to the observability VM (reads the harness password itself) ---
+bash "$(dirname "$0")/../postgres-common/alloy.sh"